In this paper, we present an approach for the segmentation of people silhouettes in images. Since in real-world images estimating pixel probabilities to belong to people or background is difficult, we propose to optimally combine several ones. A local window classifier based on SVMs with Histograms of Oriented Gradients features estimates probabilities from pixels' appearance. A shape template prior is also computed over a set of training images. From these two probability maps, color distributions relying on color histograms and Gaussian Mixture Models are estimated and the associated probability maps are derived. All these probability maps are optimally combined into a single one with weighting coefficients determined by a genetic algorithm. This final probability map is used within a graph-cut to extract accurately the silhouette. Experimental results are provided on both the INRIA Static Person Dataset and BOSS European project and show the benefit of the approach.