Based on the theory of macroscopic quantum electrodynamics, we generalize the expression of the Casimir force for nonreciprocal media. The essential ingredient of this result is the Green’s tensor between two nonreciprocal semi-infinite slabs, including a reflexion matrix with four coefficients that mixes optical polarizations. This Green’s tensor does not obey Lorentz’s reciprocity and thus violates time-reversal symmetry. The general result for the Casimir force is analyzed in the retarded and nonretarded limits, concentrating on the influences arising from reflections with or without change of polarization. In a second step, we apply our general result to a photonic topological insulator whose nonreciprocity stems from an anisotropic permittivity tensor, namely InSb. We show that there is a regime for the distance between the slabs where the magnitude of the Casimir force is tunable by an external magnetic field. Furthermore, the strength of this tuning depends on the orientation of the magnetic field with respect to the slab surfaces.