By the Parisi-Sourlas conjecture, the critical point of a theory with random field (RF) disorder is described by a supersymmeric (SUSY) conformal field theory (CFT), related to a $d-2$ dimensional CFT without SUSY. Numerical studies indicate that this is true for the RF ${\varphi }^3$ model but not for the RF ${\varphi }^4$ model in $d<5$ dimensions. Here we argue that the SUSY fixed point is not reached because of new relevant SUSY-breaking interactions. We use a perturbative renormalization group in a judiciously chosen field basis, allowing systematic exploration of the space of interactions. Our computations agree with the numerical results for both cubic and quartic potential.