We consider an Yb-doped double-clad fiber laser in a unidirectional ring cavity containing a polarizer placed between two half-wave plates. Depending on the orientation of the phase plates, the laser operates in continuous, Q-switch, mode-lock or unstable self-pulsing regime. An experimental study of the stability of the mode locking regime is realized versus the orientation of the half-wave plates. A model for the stability of self-mode-locking and cw operation is developed starting from two coupled nonlinear Schrödinger equations in a gain medium. The model is reduced to a master equation in which the coefficients are explicitly dependent on the orientation angles of the phase plates. Analytical solutions are given together with their stability versus the angles.