Gold films do not adhere well on glass substrates, so plasmonics experiments typically use a thin adhesion layer of titanium or chromium to ensure a proper adhesion between the gold film and the glass substrate. While the absorption of light into gold structures is largely used to generate heat and control the temperature at the nanoscale, the influence of the adhesion layer on this process is largely overlooked. Here, we quantify the role of the adhesion layer in determining the local temperature increase around a single nanohole illuminated by a focused infrared laser. Despite their nanometer thickness, adhesion layers can absorb a greater fraction of the incoming infrared light than the 100 nm thick gold layer leading to a significant increase of the local temperature. Different experimental designs are explored, offering new ways to promote or avoid the temperature increase inside nanoapertures. This knowledge further expands the plasmonic toolbox for temperature-controlled experiments including single molecule sensing, nanopore translocation, polymerization, or nano-optical trapping.