The dynamics of a binary system moving in the background of a black hole is affected by tidal forces. In this work, for the Kerr black hole, we derive the electric and magnetic tidal moments at quadrupole order, where the latter are computed for the first time in full generality. We make use of these moments in the scenario of a hierarchical triple system made of a Kerr black hole and an extreme-mass ratio binary system consisting of a Schwarzschild black hole and a test particle. We study how the secular dynamics of the test particle in the binary system is distorted by the presence of tidal forces from a much larger Kerr black hole. Our treatment includes strong gravitational effects beyond the post-Newtonian approximation both for the binary system and for the tidal forces since the binary system is allowed to be close to the event horizon of the Kerr black hole. We compute the shifts in the physical quantities for the secular dynamics of the test particle and show that they are gauge-invariant. In particular, we apply our formalism to the innermost stable circular orbit for the test particle and to the case of the photon sphere. Our results are relevant for the astrophysical situation in which the binary system is in the vicinity of a supermassive black hole.