The purpose of this paper is to propose an energy based multi-scale fatigue approach which handles multidimensional time varying loading histories. Our fundamental thought is to assume that the energy dissipated at small scales governs fatigue at failure in a nonlinear additive way. We follow the Dang Van paradigm at macro scale. The structure is elastic at the macroscopic scale. At each material point, there is a stochastic distribution of weak points which will undergo strong plastic yielding, which contribute to energy dissipation without affecting the overall macroscopic stress. A kinematic hardening under the assumption of associative plasticity is also considered. Instead of using the number of cycles, we use the concept of multi-scale damage accumulation during the considered load history. A concise non-linear damage accumulation law is also proposed in our model. Fatigue will then be determined from the plastic shakedown cycle and from a phenomenological fatigue law linking lifetime and accumulated mesoscopic plastic dissipation.