Recent work has shown that existing powerful Bayesian hyperspectral unmixing algorithms can be significantly improved by incorporating the inherent local spatial correlations between pixel class labels via the use of Markov random fields. We here propose a new Bayesian approach to joint hyperspectral unmixing and image classification such that the previous assumption of stochastic abundance vectors is relaxed to a formulation whereby a common abundance vector is assumed for pixels in each class. This allows us to avoid stochastic reparameterizations and, instead, we propose a symmetric Dirichlet distribution model with adjustable parameters for the common abundance vector of each class. Inference over the proposed model is achieved via a hybrid Gibbs sampler, and in particular, simulated annealing is introduced for the label estimation in order to avoid the local- trap problem. Experiments on a synthetic image and a popular, publicly available real data set indicate the proposed model is faster than and outperforms the existing approach quantitatively and qualitatively. Moreover, for appropriate choices of the Dirichlet parameter, it is shown that the proposed approach has the capability to induce sparsity in the inferred abundance vectors. It is demonstrated that this offers increased robustness in cases where the preprocessing endmember extraction algorithms overestimate the number of active endmembers present in a given scene.