Euclidean time windows in the integral representation of the hadronic vacuum polarization contribution to the muon $g-2$ serve to test the consistency of lattice calculations and may help in tracing the origins of a potential tension between lattice and data-driven evaluations. In this paper, we present results for the intermediate time window observable computed using O($a$) improved Wilson fermions at six values of the lattice spacings below 0.1 fm and pion masses down to the physical value. Using two different sets of improvement coefficients in the definitions of the local and conserved vector currents, we perform a detailed scaling study which results in a fully controlled extrapolation to the continuum limit without any additional treatment of the data, except for the inclusion of finite-volume corrections. To determine the latter, we use a combination of the method of Hansen and Patella and the Meyer-Lellouch-Lüscher procedure employing the Gounaris-Sakurai parameterization for the pion form factor. We correct our results for isospin-breaking effects via the perturbative expansion of QCD+QED around the isosymmetric theory. Our result at the physical point is $a_\mu^{\mathrm{win}}=(237.30\pm0.79_{\rm stat}\pm1.22_{\rm syst})\times10^{-10}$, where the systematic error includes an estimate of the uncertainty due to the quenched charm quark in our calculation. Our result displays a tension of 3.8$\sigma$ with a recent evaluation of $a_\mu^{\mathrm{win}}$ based on the data-driven method.