This paper presents a novel approach for visual object classification. Based on Gestalt theory, we propose to extract features from coarse regions carrying visually significant information such as line segments and/or color and to include neighborhood information in them. We also introduce a new classification method based on the polynomial modeling of feature distribution which avoids some drawbacks of a popular approach, namely “bag of keypoints”. Moreover we show that by separating features extracted from different sources in different “channels”, which are then combined using a late fusion strategy, we can limit the impact of feature dimensionality and actually improve classification accuracy. Using this classifier, experiments reveal that our features lead to better results than the popular SIFT descriptors, but also that they can be combined with SIFT features to reinforce performance, suggesting that our features managed to extract information which is complementary to the one of SIFT features.