Most of the mitigation techniques against access-driven cache side-channel attacks (CSCAs) are not very effective. This is mainly because most mitigation techniques usually protect against any given specific vulnerability of the system and do not take a system-wide approach. Moreover, they either completely remove or greatly reduce the performance benefits. Therefore, to find a security vs performance trade-off, we argue in favor of need-based protection in this paper, which will allow the operating system to apply mitigation only after successful detection of CSCAs. Thus, detection can serve as a first line of defense against such attacks. In this work, we propose a novel OS-level runtime detection-based mitigation mechanism, called the Kingsguard, against CSCAs in general-purpose operating systems. The proposed mechanism enhances the security and privacy capabilities of Linux as a proof of concept, and it can be widely used in commodity systems without any hardware modifications. We provide experimental validation by mitigating three state-of-the-art CSCAs on two different cryptosystems running under Linux. We have also provided results by analyzing the effect of the combination of multiple attacks running concurrently under variable system noise. Our results show that the Kingsguard can detect and mitigate known CSCAs with an accuracy of more than 99% and 95%, respectively.