This research gauges the ability of deep reinforcement learning (DRL) techniques to assist the control of conjugate heat transfer systems. It uses a novel, "degenerate" version of the proximal policy optimization (PPO) algorithm to train a neural network in optimizing said system only once per learning episode, and an in-house stabilized finite elements environment combining variational multiscale (VMS) modeling of the governing equations, immerse volume method, and multi-component anisotropic mesh adaptation to compute the numerical reward fed to the neural network. Several test cases of natural and forced convection are used as testbed for developing the methodology, that proves successful to alleviate the heat transfer enhancement related to the onset of convection in a two-dimensional, differentially heated square cavity, and to improve the homogeneity of temperature across the surface of two and three-dimensional hot workpieces under impingement cooling. Beyond adding value to the shallow literature on this subject, these findings establish the potential of single-step PPO for reliable black-box optimization of computational fluid dynamics (CFD) systems, and pave the way for future progress in the optimal control of conjugate heat transfer using this new class of methods.