A major goal of molecular evolutionary biology is to identify loci or regions of the genome under selection versus those evolving in a neutral manner. Correct identification allows accurate inference of the evolutionary process and thus comprehension of historical and contemporary processes driving phenotypic change and adaptation. A fundamental difficulty lies in distinguishing sites targeted by selection from both sites linked to these targets and sites fully independent of selection. These three categories of sites necessitate attention in light of the debate over the relative importance of selection versus neutrality and the neutral theory. Modern genomic insights have proved that complex processes such as linkage, demography, and biased gene conversion complicate our understanding of the role of neutral versus selective processes in evolution. In this perspective, we first highlight the importance of the genomic and (a)biotic context of new mutations to identify the targets of natural selection. We then present mechanisms that may constrain the evolution of genomes and bias the inference of selection. We discuss these mechanisms within the two critical levels that they occur: the population level and the molecular level. We highlight that they should be taken into account to correctly distinguish sites across the genome subject to selective or non-selective forces and stress that a major current field-wide goal is to quantify the absolute importance of these mechanisms.