Atom interferometers are very sensitive to accelerations and rotations. This property, which has some very interesting applications, induces a deleterious phase noise due to the seismic noise of the laboratory and this phase noise is sufficiently large to reduce the fringe visibility in many experiments. We develop a model calculation of this phase noise in the case of Mach-Zehnder atom interferometers and we apply this model to our thermal lithium interferometer. We are able to explain the observed phase noise which has been detected through the rapid dependence of the fringe visibility with the diffraction order. We think that the dynamical model developed in the present paper should be very useful to reduce the vibration induced phase noise in atom interferometers, making many new experiments feasible.