Nowadays, large amounts of data are generated by a wide range of data streams. This data contains valuable knowledge to support decision-making processes in a variety of application domains, such as smart cities, social media analysis, and the Internet of Things. To address the constantly changing data, stream reasoning has emerged as a crucial method for performing logical reasoning on streams of data. This paper focuses on the limitations of existing stream reasoning systems based on Answer Set Programming (ASP). It proposes a novel solution based on Deep Reinforcement Learning (DRL) to handle continuous data streams efficiently. Existing ASP-based stream reasoning systems face challenges in managing their internal state and effectively handling data streams over extended periods. To overcome these limitations, we propose an approach that integrates cache management techniques with DRL and ideas from heuristics developed for the Conflict-Driven Constraint Learning (CDCL) algorithm. DRL is chosen for its ability to outperform traditional Reinforcement Learning (RL) algorithms in managing complex tasks and generalizing learned policies to unseen situations. The research contributions include a DRLbased framework that efficiently manages learned constraints in ASP-based stream reasoning engines. The proposed approach is compared to recent work in the literature, and the results demonstrate its benefits in terms of average cache utilization. The findings showcase the potential of the proposed DRL-based solution for improving the performance and widespread implementation of stream reasoning with ASP engines. By effectively handling the dynamic and continuous nature of data streams.