This paper studies the linear and nonlinear stages of the fan instability, considering electromagnetic waves of the whistler frequency range interacting resonantly with energetic electron fluxes in magnetized plasmas. The main attention is paid to determine the wave-particle interaction processes that can lead to the excitation of intense electromagnetic waves by nonequilibrium particle distributions involving suprathermal tails, and to explain under what conditions and through what mechanisms they can occur, develop, and saturate. This paper presents and discusses two main processes: (i) the linear fan instability and (ii) the nonlinear process of dynamical resonance merging, which can significantly amplify the energy carried by linearly destabilized waves after they saturate due to particle trapping. This study consists of (i) determining analytically and numerically, for parameters typical of space and laboratory plasmas, the linear growth rates of whistlers excited by suprathermal particle fluxes through the fan instability, as well as the corresponding thresholds and the physical conditions at which the instability can appear, (ii) building a theoretical self-consistent 3D model and a related numerical code for describing the nonlinear evolution of the wave-particle system, and (iii) performing numerical simulations to reveal and characterize the nonlinear amplification process at work, its conditions of development, and its consequences, notably in terms of electromagnetic wave radiation. The simulations show that when the waves have reached sufficient energy levels owing to the linear fan instability, they saturate by trapping particles and due to the complex dynamics of these particles in the electromagnetic fields, the resonant velocities' domains of the waves overlap and merge, meanwhile a strong increase of the wave energy occurs.