Primary productivity in the Southern Ocean plays a key role in global biogeochemical cycles. While much focus has been placed on phytoplankton production seasonality, non-seasonal fluctuations exceed the amplitude of the seasonal cycle across large swaths of the Antarctic Circumpolar Current. This non-seasonal variability comprises a broad range of timescales from sub-seasonal (<3 months) to multi-annual (>1 year), all of which can project onto the annual mean value. However, year-to-year variations of surface chlorophyll (SChl), a proxy for phytoplankton biomass, are typically attributed to ocean circulation changes associated with the Southern Annular Mode (SAM), which implicitly assumes that sub-seasonal variability averages to near-zero over long timescales. Here, we test this assumption by applying a timeseries decomposition method to satellite-derived SChl in order to separate the low-frequency and high-frequency contributions to the non-seasonal variability. We find that throughout most of the Southern Ocean, year-to-year SChl variations are dominated by the sub-seasonal component, which is not strongly correlated with the SAM. The multi-annual component, while correlated with the SAM, only accounts for about 10% of the total SChl variance. This suggests that changes in annual mean SChl are related to intermittent forcing at small scales, rather than low-frequency climate variability, and thus do not remain correlated over large regions.