Ghys and Sergiescu proved in the $80$s that Thompson's groups $F$ and $T$ admit actions by $C^{\infty}$ diffeomorphisms of the interval. They proved that the standard actions of these groups are topologically conjugate to a group of $C^\infty$ diffeomorphisms. Monod defined a family of groups of piecewise projective homeomorphisms, and Lodha-Moore defined finitely presentable groups of piecewise projective homeomorphisms. These groups are of particular interest because they are nonamenable and contain no free subgroup. In contrast to the result of Ghys-Sergiescu, we prove that the groups of Monod and Lodha-Moore are not topologically conjugate to a group of $C^1$ diffeomorphisms. Furthermore, we show that the group of Lodha-Moore has no nonabelian $C^1$ action on the interval. We also show that Monod's groups $H(A)$, in the case where $\mathsf{PSL}(2,A)$ contains a rational homothety $x\mapsto \tfrac{p}{q}x$, do not admit a $C^1$ action on the interval. The obstruction comes from the existence of hyperbolic fixed points for $C^1$ actions. With slightly different techniques, we also show that some groups of piecewise affine homeomorphisms of the interval are not smoothable.