This paper is concerned with the dynamic instability of an internally damped rotating composite shaft. A three-dimensional homogenized finite element beam model is developed and then used to evaluate natural frequencies and instability thresholds of rotating structures. The results, obtained by the proposed method which takes into account internal damping, transversal shear effects and coupling effects, are compared to those obtained by using Equivalent Modulus Beam Theory (EMBT) and by a Simplified Homogenized Beam theory (SHBT) found in literature. Then the influence of coupling effects on the instability thresholds is investigated.