We investigate how the microstructure of a colloidal polycrystal influences its linear viscoelasticity. We use thermosensitive copolymer micelles that arrange in water in a cubic crystalline lattice, yielding a colloidal polycrystal. The polycrystal is doped with a small amount of nanoparticles, of size comparable to that of the micelles, which behave as impurities and thus partially segregate in the grain boundaries. We show that the shear elastic modulus only depends on the packing of the micelles and does not vary neither with the presence of nanoparticles nor with the crystal microstructure. By contrast, we find that the loss modulus is strongly affected by the presence of nanoparticles. A comparison between rheology data and small-angle neutron scattering data suggests that the loss modulus is dictated by the total amount of nanoparticles in the grain boundaries, which in turn depends on the sample microstructure.