We present a rest-UV selected sample of 32 lensed galaxies at $z\sim 2$ observed with joint Keck/LRIS rest-UV and Keck/MOSFIRE rest-optical spectra behind the clusters Abell 1689, MACS J0717, and MACS J1149. The sample pushes towards the faintest UV luminosities observed ($-19 \le {\rm M_{\rm UV}} \le -17$) at this redshift. The fraction of dwarf galaxies identified as Ly$\alpha$ emitters ($\rm EW \ge 20\ \overset{\lower.5em\circ}{\mathrm{A}}$) is ${\rm X_{\rm LAE}}=25^{+15}_{-10}\%$. We use the Balmer lines and UV continuum to estimate the intrinsic EW allowing us to distinguish the effects of the ionizing spectra and Ly$\alpha$ escape fraction on the observed EW distribution. Fainter galaxies ($\rm M_{\rm UV} > -19$) show larger intrinsic EWs and escape fractions than brighter galaxies. Only galaxies with intrinsic EWs greater than 40$\ \overset{\lower.5em\circ}{\mathrm{A}}$ have escape fractions larger than 0.05. We find an anti-correlation between the escape fraction and $\rm A_V$ as well as UV spectral slope. The volumetric escape fraction of our sample is $f_{\rm esc}^{\rm Ly\alpha} = 4.59^{+2.0}_{-1.4}\%$ in agreement with measurements found elsewhere in the literature. About half of the total integrated Ly$\alpha$ luminosity density comes from galaxies with ${\rm EW}_{\rm obs}>20\ \overset{\lower.5em\circ}{\mathrm{A}}$.