Non-magnetic particles in suspension in a ferrofluid act as magnetic holes when an external magnetic field is exerted: They acquire an effective dipolar moment opposing the surrounding one, which induces dipolar magnetic interactions. For a large enough imposed field and particle size, the induced interactions dominate the thermal forces, dipolar chains form. At equilibrium, these chains fluctuate under the effect of brownian noise. When the imposed magnetic field is suddenly decreased, these chains roughen dynamically. We study the time and size scaling of these fluctuations and roughening, and the relationships between the equilibrium and out of equilibrium behavior. We compare the experimental data both to brownian dynamics simulations, and to a simple theory of semi-flexible polymer chains, a generalization of the Rouse model. The scaling behavior of the experiments agree with the predictions of both theory and simulations over 5 orders of magnitudes. The roughening follows three successive regimes: The Root Mean Square Width of the chain initially evolves as $W\sim t^{1/2}$, then it enters a subdiffusive regime where $W\sim t^{1/4}$ and eventually it saturates to a level $W\sim N$, where $N$ is the number of particles in the chain. The exact prefactors as a function of the applied field, particle diameter and temperature are also derived analytically. We also show that this phenomenon can be described equivalently as a non-Markovian diffusion process for a particle in an environment with memory effects. Within this framework, our system is shown to confirm the predictions of theories for anomalous diffusion in systems with memory.