We are interested in the question of stability in the field of shape optimization, with focus on the strategy using second order shape derivative. More precisely, we identify structural hypotheses on the hessian of the considered shape function, so that critical stable domains (i.e. such that the first order derivative vanishes and the second order one is positive) are local minima for smooth perturbations; as we are in an infinite dimensional framework, and that in most applications there is a norm-discrepancy phenomenon, this type of result require a lot of work. We show that these hypotheses are satisfied by classical functionals, involving the perimeter, the Dirichlet energy or the first Laplace-Dirichlet eigenvalue. We also explain how we can easily deal with constraints and/or invariance of the functionals. As an application, we retrieve or improve previous results from the existing literature, and provide new local stability results. We finally test the sharpness of our results by showing that the local minimality is in general not valid for non-smooth perturbations.