This works extends the recent study on the dielectric permittivity of crystals within the Hartree model [E. Cances and M. Lewin, Arch. Rational Mech. Anal., 197 (2010) 139--177] to the time-dependent setting. In particular, we prove the existence and uniqueness of the nonlinear Hartree dynamics, also called the random phase approximation in the physics literature, in a suitable functional space allowing to describe a local defect embedded in a perfect crystal. We also give a rigorous mathematical definition of the microscopic frequency-dependent polarization matrix, and derive the macroscopic Maxwell-Gauss equation for insulating and semiconducting crystals, from a first order approximation of the nonlinear Hartree model, by means of homogenization arguments.