The optimization of channel assignment in cellular networks is a very complex optimization problem and it becomes more difficult when the network handles different classes of traffic. The objective is that channel utility be maximized so as to maximize service in a stochastic caller environment. We address the dynamic channel assignment (DCA) combined with call admission control (CAC) problem in a multimedia cellular network that handles several classes of traffic with different resource requirements. The problem is naturally formulated as a semi-Markov decision process (SMDP) problem and we use an approach based on reinforcement learning (RL) [neuro-dynamic programming (NDP)] method to solving it. We show that the policy obtained using our Q-DCA algorithm provides a good solution and is able to earn significantly higher revenues than classical solutions. A broad set of experiments illustrates the robustness of our policy that improves the quality of service (QoS) and reduces call-blocking probabilities for handoff calls in spite of variations in the traffic conditions.