Because of technological constraints, transducers are usually not ideal. In musical and audio applications, this is the case of electrodynamic loudspeakers used eg. in guitar amplifiers. Thus, to build realistic numerical simulations of such systems, it is important to pay close attention to their non ideality. These systems include several nonlinearities, mainly due to mechanical suspensions, magnetic properties and temperature variations. At the same time, it is not straightforward to model such refinements while preserving basic physical properties such as causality, stability, passivity. In this paper, we introduce a new modeling of loudspeaker which includes fractional order dynamics and nonlinearities, such that the power balance is guaranteed. Since the mechanical-acoustic coupling is well described in the literature, we focus on the functioning of loudspeaker in the electrical, magnetical and mechanical domains, applying a standard acoustical load on the diaphragm. The approach is based on Port- Hamiltonian Systems theory. This formalism naturally preserves the energetic behavior of elementary components and the power balance, including the nonlinear case. In conjunction, we describe the nonlinearities due to the lossy coil in terms of fractional derivative. This permit to well approximate the nonlinear electrical impedance using a reduced number of parameters, still preserving passivity of continuous-time models despite the approximation. By transcribing this property in the digital domain, we guarantee the stability of the simulations. Thermodynamic effects are neglected in this preliminary work, but can directly be incorporated in the Port-Hamiltonian model.