This paper proposes a novel technique of incorporating factor-analysis-based inter-session variability (ISV) modelling in speaker verification systems that employ continuous progressive speaker model adaptation. Continuous model adaptation involves the use of all encountered trials in the adaptation process through the assignment of confidence measures. The proposed approach incorporates these confidence measures in the general statistics used in the ISV modelling process. Progressive SVM-based classification was integrated into the system through the utilisation of GMM mean supervectors. The proposed system demonstrated a gain of 50% over baseline results when trialled on the NIST 2005 SRE corpus. Adaptative score normalisation techniques were found to be beneficial to both GMM and SVM configurations alleviating the detrimental effects of score shift in progressive systems.