Single-pixel imaging enables the acquisition of hyperspectral data across thousands of spectral channels at a low cost. After acquisition of the spectrum of scalar products of the scene with several light patterns, the hypercube of the scene is reconstructed by solving an inverse problem. Then, the hypercube can be decomposed into a few material maps and their corresponding spectral signatures. In this paper, we propose and discuss alternatives to the straightforward two-step approach. In particular, we propose a joint approach that recovers the material maps and spectral signatures directly from the measurements. The joint problem can be solved by tailoring existing nonnegative matrix factorization tools to single-pixel imaging. We demonstrate that the proposed joint method provides more accurate estimation of the spectra and material maps in simulations at different noise levels and sparsity levels.