Abstract Facing the exponential demand for massive connectivity and the scarcity of available resources, next-generation wireless networks have to meet very challenging performance targets. Particularly, the operators have to cope with the continuous prosperity of the Internet of things (IoT) along with the ever-increasing deployment of machine-type devices (MTDs). In this regard, due to its compelling benefits, non-orthogonal multiple access (NOMA) has sparked a significant interest as a sophisticated technology to address the above-mentioned challenges. In this paper, we consider a hybrid NOMA scenario, wherein the MTDs are divided into different groups, each of which is allocated an orthogonal resource block (RB) so that the members of each group share a given RB to simultaneously transmit their signals. Firstly, we model the densely deployed network using a mean field game (MFG) framework while taking into consideration the effect of the collective behavior of devices. Then, in order to reduce the complexity of the proposed technique, we apply the multi-armed bandit (MAB) framework to jointly address the resource allocation and the power control problem. Thereafter, we derive two distributed decision-making algorithms that enable the users to autonomously regulate their transmit power levels and self-organize into coalitions based on brief feedback received from the base station (BS). Simulation results are given to underline the equilibrium properties of the proposed resource allocation algorithms and to reveal the robustness of the proposed learning process.