$\tau$ Sco, a well-studied magnetic B-type star in the Upper Sco association, has a number of surprising characteristics. It rotates very slowly and shows nitrogen excess. Its surface magnetic field is much more complex than a purely dipolar configuration which is unusual for a magnetic massive star. We employ the CMFGEN radiative transfer code to determine the fundamental parameters and surface CNO and helium abundances. Then, we employ MESA and GENEC stellar evolution models accounting for the effects of surface magnetic fields. To reconcile $\tau$ Sco's properties with single-star models, an increase is necessary in the efficiency of rotational mixing by a factor of 3 to 10 and in the efficiency of magnetic braking by a factor of 10. The spin down could be explained by assuming a magnetic field decay scenario. However, the simultaneous chemical enrichment challenges the single-star scenario. Previous works indeed suggested a stellar merger origin for $\tau$ Sco. However, the merger scenario also faces similar challenges as our magnetic single-star models to explain $\tau$ Sco's simultaneous slow rotation and nitrogen excess. In conclusion, the single-star channel seems less likely and versatile to explain these discrepancies, while the merger scenario and other potential binary-evolution channels still require further assessment as to whether they may self-consistently explain the observables of $\tau$ Sco.