Abstract We investigate the properties of the glass phase of a recently introduced spin glass model of soft spins subjected to an anharmonic quartic local potential, which serves as a model of low temperature molecular or soft glasses. We solve the model using mean field theory and show that, at low temperatures, it is described by full replica symmetry breaking. As a consequence, at zero temperature the glass phase is marginally stable. We show that in this case, marginal stability comes from a combination of both soft linear excitations—appearing in a gapless spectrum of the Hessian of linear excitations—and pseudogapped non-linear excitations—corresponding to nearly degenerate two level systems. Therefore, this model is a natural candidate to describe what happens in soft glasses, where quasi localized soft modes in the density of states appear together with non-linear modes triggering avalanches and conjectured to be essential to describe the universal low temperature anomalies of glasses.