We theoretically study the Casimir-Polder force on an atom in an arbitrary initial state in a rather general electromagnetic environment wherein the materials may have a nonreciprocal bianisotropic dispersive response. It is shown that under the Markov approximation the force has resonant and nonresonant contributions. We obtain explicit expressions for the optical force both in terms of the system Green function and of the electromagnetic modes. We apply the theory to the particular case wherein a two-level system interacts with a topological gyrotropic material, showing that the nonreciprocity enables exotic light-matter interactions and the opportunity to sculpt and tune the Casimir-Polder forces on the nanoscale. With a quasistatic approximation, we obtain a simple analytical expression for the optical force and unveil the crucial role of surface plasmons in fluctuation-induced forces. Finally, we derive the Green function for a gyrotropic material half-space in terms of a Sommerfeld integral.