We analyze, both theoretically and experimentally, the phase noise of the radio frequency (RF) beatnote generated by optical mixing of two orthogonally polarized modes in an optically pumped dual-frequency Vertical External Cavity Surface Emitting Laser (VECSEL). The characteristics of the RF phase noise within the frequency range of 10 kHz - 50 MHz are investigated for three different nonlinear coupling strengths between the two lasing modes. In the theoretical model, we consider two different physical mechanisms responsible for the RF phase noise. In the low frequency domain (typically below 500 kHz), the dominant contribution to the RF phase noise is shown to come from the thermal fluctuations of the semicondutor active medium induced by pump intensity fluctuations. However, in the higher frequency domain (typically above 500 kHz), the main source of RF phase noise is shown to be the pump intensity fluctuations which are transfered to the intensity noises of the two lasing modes and then to the phase noise via the large Henry factor of the semiconductor gain medium. For this latter mechanism, the nonlinear coupling strength between the two lasing modes is shown to play an important role in the value of the RF phase noise. All experimental results are shown to be in good agreement with theory.