Afforestation plays an important role in soil protection and ecological restoration. The objective of this study is to understand the effect of afforestation on soil carbon and soil microbial communities in the Loess Plateau of China. We measured two chemically-separated carbon fractions (i.e., humic acid, HA, and fulvic acid, FA) and soil microbial communities within shrublands (18-year-old Caragana korshinskii Kom (shrubland I) and 28-year-old Caragana korshinskii Kom (shrubland II)) and cropland. The size and structure of the soil microbial community was measured by phospholipid fatty acid (PLFA) analysis. The analysis of C-fractions indicated that at a depth of 0–20 cm, FA-C concentration in shrubland I and shrubland II were 1.7 times that of cropland, while HA-C had similar values across all three sites. Total PLFAs, G+ (Gram positive) bacterial, G- (Gram negative) bacterial, and actinobacterial PLFAs were highest in shrubland II, followed by shrubland I and finally cropland. Fungal PLFAs were significantly higher in shrubland II compared to the other sites. Additionally, we found a high degree of synergy between main microbial groups (apart from fungi) with FA-C. We concluded that planting C. korshinskii in abandoned cropland could alter the size and structure of soil microbial community, with these changes being closely related to carbon sequestration and humus formation.