This paper is the second part of the study. In Part~I, self-similar solutions of a weighted fast diffusion equation (WFD) were related to optimal functions in a family of subcritical Caffarelli-Kohn-Nirenberg inequalities (CKN) applied to radially symmetric functions. For these inequalities, the linear instability (symmetry breaking) of the optimal radial solutions relies on the spectral properties of the linearized evolution operator. Symmetry breaking in (CKN) was also related to large-time asymptotics of (WFD), at formal level. A first purpose of Part~II is to give a rigorous justification of this point, that is, to determine the asymptotic rates of convergence of the solutions to (WFD) in the symmetry range of (CKN) as well as in the symmetry breaking range, and even in regimes beyond the supercritical exponent in (CKN). Global rates of convergence with respect to a free energy (or entropy) functional are also investigated, as well as uniform convergence to self-similar solutions in the strong sense of the relative error. Differences with large-time asymptotics of fast diffusion equations without weights will be emphasized.