Heterogeneous Cloud Radio Access Network (H-CRAN) is considered as a cost-efficient network solution to meet 5G data traffic requirements. In this paper, we consider the problem of beamforming and user clustering (user-to-Remote Radio Head (RRH) association) in the downlink of a H-CRAN where users have heterogeneous mobility profiles. Given the rapidly time-varying nature of the mobile wireless environment, it is challenging to offer an optimal beamforming and user association performance during a long-term allocation process without incurring large Channel State Information (CSI) and signaling overheads. For that purpose, we proposed in Ha et al. (2019) an Adaptive Beamforming and User Clustering (ABUC) algorithm which resolves the joint beamforming and user clustering problem when considering CSI cost and imperfectness under user mobility assumptions. In this paper, we design a deep reinforcement-learning framework which enables the proposed ABUC algorithm to select on-the-fly its best feedback scheduling parameters, namely the period and type of CSI feedback, given each user mobility profile. The proposed ABUC-DQL approach can overcome the scalability limitation of the Q-learning approach (Ha et al., 2019) and better handle the problem when formulated using a POMDP (Partially Observable Markov Decision Process) model. The simulation results show that the convergence time is mainly impacted by the number of users in the network, and the online-learning ability of the framework can quickly adapt to the changes of users mobility.