The competition between food and feed challenges the use of alternative resources such as fibrous feedstuffs in diets for pigs. Adding fat to high fiber diets appears as a relevant strategy to improve dietary energy value and feed efficiency, but this changes the nutrients and energy source compared to a standard low fat diet. This study aimed to elucidate the transcriptional mechanisms involved in variations of adiposity when pigs were fed a high fat high fiber diet. Growing barrows (Large White, n = 48) divergently selected for feed efficiency were offered during 10 weeks either a low fat, low fiber diet (LF) or a high fat, high fiber diet (HF) where oils and wheat straw were used to partially substitute cereals (n = 24 per diet). At 132 days of age, HF pigs displayed lower (P < 0.001) proportions of perirenal (PRAT; -16%) and subcutaneous (SCAT; -28%) fat tissues than LF pigs. Analyses using a porcine microarray showed that diet had pronounced effects on adipose tissue transcriptomes. The number of differentially-expressed genes (DEG) was greater in PRAT than in SCAT, with 1,251 and 825 unique genes being up-regulated and 2,440 and 1,279 unique genes being down-regulated by the HF diet in PRAT and SCAT, respectively (cutoffs for p corrected BH < 0.10, raw p < 0.01 and fold-changes between conditions > 1.1 or < 0.9). A multiple factor analysis revealed large similarities between the two adipose tissues in response to diets. Notably, different genes related to protein catabolism (33 DEG), protein transport (33 DEG), apoptosis (31 DEG), phosphate metabolic process including ATP synthesis (34 DEG), response to stress (26 DEG) and glucose metabolism (13 DEG) were commonly down-regulated in HF pigs. Conversely, IGF1R participating to the negative regulation of apoptotic process was up-regulated by the HF diet in the two adipose tissues. Correlation modules also stressed the up-regulation by the HF diet of genes related to immunity and defense response specifically in PRAT. Causality graph analysis highlighted MLXIPL, SREBF1, peroxisome proliferator-activated receptors (PPARG, PPARD) and their heterodimer partner RXRA as candidate upstream regulators of these processes. qPCR analyses confirmed dietary-related differences in expression levels of these regulators in the two adipose tissues. Altogether, high fiber intake in growing pigs was associated with lower body fatness, which was related to lower glucose metabolism in adipose tissues; its effect on immune factors in the perirenal fat deserves further studies.