The study of polarization has gained increasing attraction in the past decades. Since observing both opinions and interactions is challenging, epistemic programs such as agent-based models have been proposed as a means to assessing the systemic consequences of social psychology mechanisms. Most results in agent-based models for opinion dynamics have focused on individual opinion constructs and pairwise interactions, with a few works treating group effects as constraints. Meanwhile, a tradition in social sciences has been putting emphasis on how group configuration affects individual behavior. In this work, we introduce a new model for accounting for both pairwise interactions in which actors observe and update opinions, and individual perception of the evolving configuration of groups that make up the population in which they are embedded. Through experiments, we show that different treatment given to pairwise interactions, depending on whether they occur between in-groups or out-groups, has quantifiable impacts in the resulting polarization of a population. In particular, the tolerance toward out-group opinions is shown to have a strong impact on the resulting polarization. Our model produces and accounts for polarized states resulting from group consolidation and fragmentation.