This paper is concerned with the dynamic instability of an internally damped rotating composite shaft. A homogenized finite element beam model, which takes into account internal damping, is introduced and then used to evaluate natural frequencies and instability thresholds. The influence of laminate parameters: stacking sequences, fiber orientation, transversal shear effect on natural frequencies and instability thresholds of the shaft are studied. The results are compared to those obtained by using equivalent modulus beam theory (EMBT), modified EMBT and layerwise beam theory (LBT), which are used in the literature. This parametric study shows that shaft instability thresholds can be very sensitive to laminate parameters.