In this paper, we address the question of process and model fairness. We propose FixOut, a human-centered and model-agnostic framework, that uses any explanation method (based on feature importance) to assess model's reliance on sensitive features. Given a pre-trained classifier, FixOut first checks whether it relies on user defined sensitive features. If it does, then FixOut employs feature dropout to produce a pool of simplified classifiers that are then aggregated into an ensemble classifier. We present empirical results using different models on several real-world datasets, that show a consistent improvement in terms of widely used fairness metrics, decreased reliance on sensitive features, and without compromising the classifier's accuracy.