The high inertial sensitivity of atom interferometers has been used to build accelerometers and gyrometers but this sensitivity makes these interferometers very sensitive to the laboratory seismic noise. This seismic noise induces a phase noise which is large enough to reduce the fringe visibility in many cases. We develop here a model calculation of this phase noise in the case of Mach-Zehnder atom interferometers and we apply this model to our thermal lithium interferometer. We are thus able to explain the observed dependence of the fringe visibility with the diffraction order. The dynamical model developed in the present paper should be very useful to further reduce this phase noise in atom interferometers and this reduction should open the way to improved interferometers.