The optimization of channel assignment in cellular networks is a very complex optimization problem and it becomes more difficult when the network handles different classes of traffic. The objective is that channel utility be maximized so as to maximize service in a stochastic caller environment. We address in this paper, the dynamic channel assignment (DCA) combined with call admission control (CAC) problem in a multimedia cellular network that handles several classes of traffic with different resource requirements. The problem is naturally formulated as a Semi-Markov Decision Process (SMDP) problem and we use an approach based on reinforcement learning (RL) [neuro-dynamic programming (NDP)] method to solving it. An additional solution to construct a call admission control (CAC) policy in an FCA (Fixed Channel Assignment) system through the same approach (RL) is proposed in this paper. We show that the policies obtained using our algorithms QCAC-FCA and Q-DCA provide a good solution and are able to earn significantly higher revenues than classical solutions. A broad set of experiments illustrates the robustness of our policies that improve the Quality of Service (QoS) and reduce call-blocking probabilities for handoff calls in spite of variations in the traffic conditions.