Mating system theory based on economics of resource defense has been applied to describe social system diversity across taxa. Such models are generally successful but fail to account for stable mating systems across different environments or shifts in mating system without a change in ecological conditions. We propose an alternative approach to resource defense theory based on frequencydependent competition among genetically determined alternative behavioral strategies characterizing many social systems (polygyny, monogamy, sneak). We modeled payoffs for competition, neighborhood choice, and paternal care to determine evolutionary transitions among mating systems. Our model predicts four stable outcomes driven by the balance between cooperative and agonistic behaviors: promiscuity (two or three strategies), polygyny, and monogamy. Phylogenetic analysis of 288 rodent species supports assumptions of our model and is consistent with patterns of evolutionarily stable states and mating system transitions. Support for model assumptions include that monogamy and polygyny evolve from promiscuity and that paternal care and monogamy are coadapted in rodents. As predicted by our model, monogamy and polygyny occur in sister taxa among rodents more often than by chance. Transitions to monogamy also favor higher speciation rates in subsequent lineages, relative to polygynous sister lineages. Taken together, our results suggest that genetically based neighborhood choice behavior and paternal care can drive transitions in mating system evolution. While our genic mating system theory could complement resource-based theory, it can explain mating system transitions regardless of resource distribution and provides alternative explanations, such as evolutionary inertia, when resource ecology and mating systems do not match.