Public speaking is a near universally anxiety-inducing social situation. Applications recreating social interactions with autonomous agents in virtual reality have been proposed as tools to alleviate public speaking anxiety, recreating exposure therapy methods with virtual audiences. To be efficient, such applications rely on the precise induction of controlled amounts of social stress. We reviewed the literature that studied the effect of virtual audience behaviors on participants' stress levels, and found contradictory results that we attempt to explain in this article. We examine those studies and propose that social presence is one of the important factors mediating the effect of audience behavior on stress levels, and through social presence we can explain the different effects of audience behavior on stress observed in past studies. We conducted a study to test this theory, and expected that high social presence would lead to larger difference in stress induced by positive and negative audiences, as opposed to lower social presence which would attenuate the effect of audience behavior. We compared two display mediums, VR headsets to induce high social presence and wall projection to induce low social presence. We find an interaction effect of social presence and audience behavior on subjectively reported stress levels, but not on physiological measures.