We consider a linear Schrödinger equation, on a bounded interval, with bilinear control. Beauchard and Laurent proved that, under an appropriate non degeneracy assumption, this system is controllable, locally around the ground state, in arbitrary time. Coron proved that a positive minimal time is required for this controllability, on a particular degenerate example. In this article, we propose a general context for the local controllability to hold in large time, but not in small time. The existence of a positive minimal time is closely related to the behaviour of the second order term, in the power series expansion of the solution.