Mining and smelting activities have brought potentially serious heavy metal(loid)s pollution to their surrounding locale. However, studies on microbial metabolic activities, community structure, and adaptation in soils proximal to non-ferrous metal mining and smelting areas are still lacking. Here the effects of biotic and abiotic characteristics of soil taken from sites surrounding inactive and active non-ferrous metal mine smelting facilities on microbial enzyme activity, microcalorimetry, and high-throughput sequencing of 16S rRNA gene barcoding were studied. Data indicated that the soils were heavily polluted by toxic metal(loid)s, of which As and Cd were the main contaminants. Microbial acid phosphatase activity and microcalorimetric total heat value were sensitive metabolic indicators in the studied areas. Actinobacteriota had the highest relative abundance, followed by Proteobacteria, Chloroflexi, and Acidobacteria. Microbial metabolic activity, bacterial community structure and phenotype varied between inactive and active sites (p < 0.05). Such analyses indicated that electrical conductivity, total As, Cu, and Mn contents, and bioavailable As, Cu, Cd, and Mn concentrations were key factors determining microbial activities, bacterial community structure, and phenotypes. Knowledge of microbial adaptation to heavy metal stressors is important for better understanding the aerial transfer of fugitive heavy metal(loid)s (and possibly microbes) and for designing future strategies for improved soil bioremediation.