We extend Tropp's analysis of Orthogonal Matching Pursuit (OMP) using the Exact Recovery Condition (ERC) to a first exact recovery analysis of Orthogonal Least Squares (OLS). We show that when ERC is met, OLS is guaranteed to exactly recover the unknown support. Moreover, we provide a closer look at the analysis of both OMP and OLS when ERC is not fulfilled. We show that there exist dictionaries for which some subsets are never recovered with OMP. This phenomenon, which also appears with $\ell_1$ minimization, does not occur for OLS. Finally, numerical experiments based on our theoretical analysis show that none of the considered algorithms is uniformly better than the other.