In 2000, Darmon described a program to study the generalized Fermat equation using modularity of abelian varieties of $\mathrm{GL}_2$-type over totally real fields. The original approach was based on hard open conjectures, which have made it difficult to apply in practice. In this paper, building on the progress surrounding the modular method from the last two decades, we analyze and expand the current limits of this program by developing all the necessary ingredients to use Frey abelian varieties for new Diophantine applications. In particular, we deal with all but the fifth and last step in the modular method for Fermat equations of signature $(r,r,p)$ in almost full generality. As an application, for all integers $n \geq 2$, we give a resolution of the generalized Fermat equation $x^{11} + y^{11} = z^n$ for solutions $(a,b,c)$ such that $a + b$ satisfies certain $2$- or $11$-adic conditions. Moreover, the tools developed can be viewed as an advance in addressing a difficulty not treated in Darmon's original program: even assuming `big image' conjectures about residual Galois representations, one still needs to find a method to eliminate Hilbert newforms at the Serre level which do not have complex multiplication. In fact, we are able to reduce the problem of solving $x^5 + y^5 = z^p$ to Darmon's `big image conjecture', thus completing a line of ideas suggested in his original program, and notably only needing the Cartan case of his conjecture.