When incompletely isolated taxa coexist in a patchy environment (e.g. mosaic hybrid zones, host-race complexes), patterns of variation may differ between selected traits/genes and neutral markers. While the genetic structure of selected traits/loci tends to coincide with habitat variables (producing Genetic-Environment Association or GEA), genetic differentiation at neutral loci unlinked to any selected locus rather depends on geographic connectivity at a large scale (e.g. Isolation-By-Distance or IBD), although these loci often display GEA at a small scale. This discrepancy has been repeatedly taken as evidence for parallel primary divergence driven by local adaptation. We argue that this interpretation needs to be addressed more thoroughly by considering the alternative hypothesis that speciation was initiated in allopatry and secondary introgression has subsequently erased the signal of past differentiation at neutral loci. We present a model of neutral introgression after secondary contact in a mosaic hybrid zone, which describes how GEAs dissipate with time and how neutral variation self-organizes according to the environmental and geographic structures. We show that although neutral loci can be affected by environmental selection, they are often more affected by history and connectivity: the neutral structure retains the initial geographic separation more than it correlates with the environment during the colonization and introgression phases, and then converges to a migration-drift balance, the most frequent outcome of which is GEA at a local scale but IBD at a large scale. This is the exact pattern usually attributed to parallel ecological speciation. Introgression is heterogeneous in space and depends on the landscape structure (e.g. it is faster in small patches, which are more impacted by immigration). Furthermore, there is no directionality in the association and it is possible to observe reversed GEAs between distant regions. We argue that the history of differentiation should ideally be reconstructed with selected loci or neutral loci linked to them, not neutral ones, and review some case studies for which the hypothesis of a long coexistence of co-adapted genetic backgrounds might have been refuted too hastily [Current Zoology 59 (1): 72-86, 2013].