We present a microscopic description of neutron and proton elastic and inelastic scattering off Pb206,207,208 and Bi209 together with new medium-energy (7.5–15.5 MeV) neutron elastic and inelastic scattering measurements for various low-energy levels. The Jeukenne, Lejeune, Mahaux (JLM) semimicroscopic folding model provides the relevant optical and transition potentials used to calculate the elastic and inelastic cross sections. Rearrangement corrections that account for proton and neutron density variations during the transition are considered. The nuclear diagonal and transition densities are calculated from a (quasiparticle) random phase approximation [(Q)RPA] structure model implemented with the Gogny D1S interaction. Calculated differential elastic and inelastic cross sections are mostly in very good agreement with available data. Predicted nuclear structure properties as well as scattering calculations and measurements for the yrast quadrupole excitation in Pb206 are carefully discussed. The rearrangement corrections to inelastic scattering form factors are shown to reduce medium energy nucleon inelastic cross section by up to 55%. The magnitude of these corrections changes depending on the probe, incident energy, multipolarity, and nature of the transitions considered. The combined analysis of nuclear structure properties and those for nucleon scattering cross sections measured previously over a range of incident energies provides stringent tests of present microscopic models and helps distinguish those scattering data which most likely call for revision.