We investigate the quantum optical torque on an atom interacting with an inhomogeneous electromagnetic environment described by the most general linear constitutive relations. The atom is modeled as a two-level system prepared in an arbitrary initial energy state. Using the Heisenberg equation of motion (HEM) and under the Markov approximation, we show that the optical torque has a resonant and nonresonant part, associated, respectively, with a spontaneous-emission process and Casimir-type interactions with the quantum vacuum, which can both be written explicitly in terms of the system Green function. Our formulation is valid for any three-dimensional inhomogeneous, dissipative, dispersive, nonreciprocal, and bianisotropic structure. We apply this general theory to a scenario in which the atom interacts with a material characterized by strong nonreciprocity and modal unidirectionality. In this case, the main decay channel of the atom energy is represented by the unidirectional surface waves launched at the nonreciprocal material-vacuum interface. To provide relevant physical insight into the role of these unidirectional surface waves in the emergence of nontrivial optical torque, we derive closed-form expressions for the induced torque under the quasistatic approximation. Finally, we investigate the equilibrium states of the atom polarization, along which the atom spontaneously tends to align due to the action of the torque. Our theoretical predictions may be experimentally tested with cold Rydberg atoms and superconducting qubits near a nonreciprocal material. We believe that our general theory may find broad application in the context of nanomechanical and biomechanical systems.