Magnetotactic bacteria are swimming microorganisms able to follow magnetic field lines with the help of an organelle called the magnetosome that is made of biomineralized magnetic crystals assembled in a chain. In combination with this ability, these bacteria perform active oxygen sensing to reach the oxic-anoxic transition zone, which is often located in the upper part of the sediment. From a physicist's perspective, magnetotactic bacteria can be seen at the interface between bacterial active matter and magnetic colloids, which gives them unique properties at both the individual and collective levels. In crowded media and/or when they are submitted to external flows, their motion can be efficiently driven by magnetic fields, which leads to surprising effects. In this Colloquium, the different features of magnetotactic bacteria at are reviewed at every scale, from single cell to collective motion, from simple to complex environments, and by emphasizing the differences from other bacterial species or passive magnetic colloids. The Colloquium concludes with a discussion of perspectives on using magnetotactic bacteria in active magnetorheology.