Features computed as statistics (e.g. histograms) of local filter responses have been reported as the most powerful descriptors for texture classification and segmentation. The selection of the filter banks remains however a crucial issue, as well as determining a relevant combination of these descriptors. To cope with selection and fusion issues, we propose a novel approach relying on the definition of the texture-based similarity measure as a weighted sum of the Kullback–Leibler measures between empirical feature statistics. Within a supervised framework, the weighting factors are estimated according to the maximization of a margin-based criterion. This weighting scheme can also be considered as a filter selection method: texture filter response distributions are ranked according to the associated weighting factors so that the problem of selecting a subset of filters reduces to picking the first features only. An application of this similarity measure to texture recognition is reported. We also investigate its use for texture segmentation within a Bayesian Markov Random Field (MRF)-based framework. Experiments carried out on Brodatz textures and sonar images show that the proposed weighting method improves the classification and the segmentation rates while relying on a parsimonious texture representation