The influence of the physical activity environment in the home and at work on cardiorespiratory fitness (CRF) and objectively-measured physical activity has not been extensively studied. We recruited 147 women with a (mean ± SD) age of 54 ± 7 years and without evidence of chronic disease. The physical activity environment was assessed by self-report (Assessing Levels of PHysical Activity or ALPHA questionnaire), CRF using a submaximal step test, usual physical activity using combined heart rate and accelerometry, as well as by a validated questionnaire (Recent Physical Activity Questionnaire). Summary scores of the home environment and the work environment derived from the ALPHA questionnaire were positively correlated with CRF after adjustment for age (r = 0.18, p = 0.03 and r = 0.28, p < 0.01, respectively). Women owning a bicycle or having a garden (which may prompt physical activity) had higher CRF; those with a bicycle at home also had a higher physical activity energy expenditure. Similarly, women who had access to fitness equipment at work had higher CRF. In conclusion, these results provide new insights into potential environmental influences on physical capacity and physical activity that could inform the design of physical activity promotion strategies.