We present a numerical solver for the incompressible Navier-Stokes equations that combines fourth-order-accurate discrete approximations and an adaptive tree grid (i.e. h-refinement). The scheme employs a novel compact-upwind advection scheme and a 4th-order accurate projection algorithm whereby the numerical solution exactly satisfies the incompressibility constraint. Further, we introduce a new refinement indicator that is tailored to this solver. We show tests and examples to illustrate the consistency, convergence rate and the application for the adaptive solver. The combination of the solver scheme and the proposed grid adaptation algorithm result in fourth-order convergence rates whilst only tuning a single grid-refinement parameter. The speed performance is benchmarked against a well-established second-order accurate adaptive solver alternative. We conclude that the present 4th order solver is an efficient design for problems with strong localization in the spatial-temporal domain and where a high degree of convergence for the solution statistics is desired.