The rapid technological advancements in wireless communications, ubiquitous sensing and mobile networking have paved the way for the emergence of the Internet of Things (IoT) era, where “anything” can be connected “anywhere” at “anytime”. However, the flourish of IoT still faces various security and privacy preserving challenges that need to be addressed. In such pervasive and heterogeneous environment where the context conditions dynamically and frequently change, efficient and context-aware mechanisms are required to meet the users' changing needs. Therefore, it seems crucial to design an adaptive access control scheme in order to remotely control smart things while considering the dynamic context changes. In this paper, we propose a Context-Aware Attribute-Based Access Control (CAABAC) approach that incorporates the contextual information with the Ciphertext-Policy Attribute-based Encryption (CP-ABE) to ensure data security and provide an adaptive contextual privacy. From a security perspective, the proposed scheme satisfies the security requirements such as confidentiality, context-aware privacy, and resilience against key escrow problem. Performance analysis proves the efficiency and the effectiveness of the proposed scheme compared to benchmark schemes in terms of storage, communication and computational cost.