The Goldilocks zone between inbreeding and outbreeding

It's closer than you think

The Little Girl Next Door, 1912, poster for an American short drama.

Abnormal embryos are most often produced when the prospective parents are siblings, followed by first cousins and then second cousins. This is well known. But did you know that the risk then progressively rises as couples become less related to each other? The sweet spot for having healthy children seems to be marriage between third or fourth cousins.

Subspecies can breed with each other, but species cannot. So it is said. The reality is less clear-cut. “Subspecies” and “species” are arbitrary points along a continuum of decreasing genetic compatibility. As we move along that continuum, we observe more and more problems when two populations try to interbreed. But how much trouble must they have before we classify them as two species? There is no right answer. Many populations have been redefined from “subspecies” to “species” and back again.

Yes, even subspecies can have trouble breeding with each other (Presgraves, 2010). When two subspecies of the house mouse are crossed (Mus musculus domesticus and Mus musculus musculus), the offspring suffer not only from reduced fertility but also from reduced immune function and higher loads of intestinal parasites (Turner et al., 2011). The reduction in fertility seems to have multiple causes: "The type and severity of fertility defects observed depends on the geographic origin of the strains and also varies among individuals within regions. This variability suggests that multiple genetic incompatibilities contribute to hybrid male sterility" (Turner et al., 2011). Moreover, one incompatibility can interact with another to produce an adverse effect greater than the sum of the two. This is often why an embryo will fail to develop beyond a certain point, since embryonic development requires coordination among many genes. Such failures, if frequent enough, will make interbreeding unlikely.

Incompatibilities are more likely to occur in complex traits that rely on many genes working together. The larger and more complex the gene system, the higher the risk of something going wrong when two populations interbreed. That “something” will involve only a few genes in any one gene system, since the vast majority are essentially the same in the two populations—which belong, after all, to the same species. But those few genes will have a disproportionate impact, like a monkey wrench thrown into a threshing machine. Or like a new gene mutation in a developing embryo—the main difference being that new mutations are much less frequent in any one embryo.

In short, genes are team players. As the geneticist Ernst Mayr wrote:

Hybridization between species leads almost invariably to unbalance through deleterious combinations of genes. ... in [the fruit fly] Drosophila even the hybridization of races may lead to destruction of well-integrated gene combinations. (Mayr, 1970, p. 399)

The risk of incompatibility depends not only on the complexity of the trait but also on the degree of divergence between the two interbreeding populations. The more their genomes have diverged from each other, the higher the risk of something going wrong.

Such divergence is due partly to the passage of time and partly to differences in natural selection:

• Passage of time. The longer the two populations are apart, the more their genomes will diverge through genetic drift and other stochastic processes.

• Differences in natural selection. The more the two populations differ in their environment, the more their genomes will diverge in significant ways and with significant consequences. Unlike genetic drift, this divergence is not a linear function of time. It is faster when the two populations have just separated from each other—usually because one of them has split off from the other and entered a very different environment that requires new adaptations. Later, the genomic divergence will slow down, as rapid evolution gives way to slower finetuning and occasional evolutionary breakthroughs.

This point is made by a review article on outbreeding depression in various plants and animals. Outbreeding will more likely reduce fertility if the two populations have 1) been separated for more than 500 years and 2) been adapting to substantially different environments for more than 20 generations. The second cause seems to be the primary cause of outbreeding depression (Frankham et al., 2011).

What about humans? In the late 1960s, the geneticist Ernst Mayr reviewed the literature and concluded that no one had yet shown that outbreeding reduces human biological fitness. He also pointed to a lack of controlled studies on mortality and fertility in the first and second generations. Finally, he added: “nor is there much evidence on nonphysical traits” (Mayr, 1970, pp. 399-400).

There have since been two controlled studies on outbreeding and human fertility in the first and second generations.

The Icelandic study

A research team looked at the fertility of 160,811 Icelandic couples born between 1800 and 1964, using a database that covers 95% of all Icelanders born since 1700. Fertility varied with the relatedness of the parents. Specifically, marriages between third or fourth cousins produced the most children, who likewise had higher fertility:

... the reproductive success of the couples, as reflected by the number of their children who reproduced, followed an n-shaped curve from the relatively low reproductive success of couples related at the level of second cousins or closer, to the maximum for couples related at the level of third and fourth cousins, after which there is a steady decrease in reproductive success with diminishing kinship between spouses. A similar picture emerges when the number of grandchildren per couple is examined (Helgason et al., 2008)

The cause seemed to be parental relatedness:

Our results, drawn from all known couples of the Icelandic population born between 1800 and 1965, show a significant positive association between kinship and fertility, with the greatest reproductive success observed for couples related at the level of third and fourth cousins. Owing to the relative socioeconomic homogeneity of Icelanders, and the observation of highly significant differences in the fertility of couples separated by very fine intervals of kinship, we conclude that this association is likely to have a biological basis. (Helgason et al., 2008)

Was the relationship between fertility and kinship due to social changes over time? In Iceland, mean fertility decreased from 1800 to 1965, while outbreeding increased. To control for both trends, the authors divided the data into 25-year intervals. Again, “the same general relationship between kinship and reproductive outcome was observed within each 25-year subinterval” (Helgason et al., 2008).

What about differences in educational status? Perhaps Icelanders who marry beyond their fourth cousins are more likely to be those who have gone to university and met a future spouse from the other side of the country. They would have fewer children, but only because they had married later in life.

That potential confound was investigated in Denmark by another research team, who examined how fertility varies with the distance between the mother’s home parish and her husband’s.

The Danish study

The researchers looked at the fertility of 22,298 Danish women born in 1954 and still living in their country in 1969. They were followed up to the end of 1999. The data came from the Danish Central Personal Register, which covers almost all of Denmark’s residents.

The researchers compared each woman’s fertility with the distance between her home parish and her husband's. Fertility peaked at around 75 km. The relationship between fertility and distance was not explained by education, family income, urbanicity, or mother's age at first birth. Again, the cause seemed to be parental relatedness. As the distance lengthened between the mother’s parish and her husband’s, a steady rise in fertility gave way to a steady fall (Labouriau & Amorim, 2008).

Mean number of children as a function of geographic distance between Danish marriage partners (Labouriau & Amorim, 2008).

The Icelandic and Danish studies have their respective advantages and disadvantages. The Icelandic study used a larger sample size over a longer period. The family data also came from “one of the most socioeconomically and culturally homogeneous societies in the world, with little variation in family size, use of contraceptives, and marriage practices" (Helgason et al., 2008).

The Danish study was likewise done within a relatively homogeneous society, at least during the period in question, while controlling for a larger number of possible socioeconomic confounds. But parental relatedness was not directly measured.

Outbreeding and sperm quality

Outbreeding can reduce fertility in many ways. Incompatibilities may arise during embryonic development and lead to miscarriage, or they may arise in fetal germ cells and reduce the fertility of the next generation. For epidemiologist Michael Joffe, this is the explanation for a century-long decline in sperm quality, and a corresponding rise in the rate of testicular cancer (Joffe, 2009).

Joffe rejects the usual explanation: the rising level of estrogenic compounds in the environment, like dioxin, DDT, PCBs, PBBs, phthalates, and others. This putative cause fails to explain why the decline in sperm quality varies so much between different geographic regions, often within the same country. Why, for instance, has it been steep in Paris and nonexistent in Toulouse? Why is it nonexistent in domestic animals that are no less exposed to estrogenic compounds? And why did the decline in sperm quality begin before the commercial production of most estrogenic compounds?

Similar doubts have been raised by Deonandan and Jaleel (2012), who point out that the decline in sperm quality seems to be confined to sperm donors from Western countries. No decline is observable in a meta-study of sperm donors from non-Western countries who “likely … were not rural, agrarian workers, but urban individuals experiencing the same environmental and occupational exposures as men from wealthier countries.”

Joffe attributes the male fertility decline to an increase in outbreeding in both the previous generation and the current one, and a consequent increase in “D&D”— duplications and deletions of genetic material during meiosis:

On the face of it, this contradicts the earlier observation that hybrid vigour would tend to increase fertility in unrelated individuals. Yet these two ideas may be compatible: both extremes — genetic similarity (inbreeding) and genetic distance (D&D accumulation) — could decrease fertility, so that an intermediate degree of relatedness would be associated with the highest degree of fertility. (Joffe, 2009, p. 303)

Outbreeding and certain psychological traits

Ernst Mayr said he knew of little evidence that outbreeding has adverse effects on mind and behavior. Yet there was already some evidence when he wrote those words. In 1926, the Harvard zoologist Charles Davenport went to Jamaica to study this question with his assistant Morris Steggerda. The two researchers administered anthropometric, physiological, and psychological tests to 300 adults and some 1,200 children of Black, White, and Biracial (“Brown”) origin. All three groups (Whites, Blacks, Biracials) were selected from the same economic level:

... it was decided that all three groups should belong to the prevailing agricultural class and that the Whites of the governing class and the white merchants of Kingston should be excluded. A difficulty arises in this, that just those Whites who are satisfied to work as agriculturalists in the midst of the island are hardly as representative of the more ambitious and intellectually endowed Whites as the agricultural Blacks are of the run of the Black population. (Davenport, 1928, p. 9)

In general, the tests placed the Biracial participants midway between the Black and White ones. But a higher proportion of the Biracial participants failed certain psychological tests, particularly when asked to make comparisons or imagine objects in two or three dimensions:

Failure rates of Black, White, and Biracial participants for tests of comparison or 3D imagination (Davenport, 1928)

Davenport concluded:

One gains the general impression that, though on the average the Browns did not do so badly, there was among them a greater number of persons than in either Blacks or Whites who were muddled and wuzzle-headed. The Blacks may have low intelligence, but they generally can use what they have in fairly effective fashion; but among the Browns there appear to be an extra 5 per cent who seem not to be able to utilize their native endowment. (Davenport, 1928, p. 69)

Since the negative impacts of outbreeding were evident on only some of the psychological tests, one could accuse Davenport of cherry-picking. On the other hand, there was little evidence of positive impacts:

The question arises: are there any traits in which, on the average, the adult Browns are superior to the Whites? We might, theoretically, expect such, yet when we have tested all of the probably genetically distinct traits between Blacks and Whites, we find only one in which the mean of the adult Browns is clearly higher than that of either parental stock.

On the other hand, if we examine the means for children of 10 to 13 or 13 to 16 years there are a few in which the brown children grade higher than either the black or the white children. ... The fact that youthful Browns sometimes score higher than youthful Blacks or Whites suggests the conclusion that brown children develop in some mental capacities precociously; and then fall behind in development. (Davenport, 1928, pp. 69-70)

If we exclude the possibility of incompatibilities during development, we are left with two alternative explanations:

• The Biracial participants had suffered some kind of deprivation that the Black and White ones had not.

• The researchers had oversampled the population of Biracial participants with certain mental issues.

Neither explanation seems likely. The Biracial participants were selected from the same economic class as the other two groups. Nor is it clear why Biracial individuals with mental issues would be oversampled. 

Conclusion

The creation of human life is a complex process that requires coordination among many genes. It is thus vulnerable to malfunctions at various stages of development, particularly at the embryonic stage. The result is usually a spontaneous abortion, most often without the prospective mother even knowing it. Such malfunctions are difficult to observe directly. For the most part, the only observable sign is a reduction in fertility, i.e., the ability to conceive a child.

Fertility is the canary in the coal mine. It is an early warning of developmental problems that may otherwise go unnoticed. Among other things, it can show us the optimal degree of parental relatedness for having healthy children.

How related, then, should prospective parents be to produce the most viable embryos? Apparently, they should be third or fourth cousins. Marriages closer in or farther out produce children who are, on average, less biologically viable.

Inbreeding and outbreeding can cause embryonic development to fail in different ways:

• Inbreeding. Parents are more likely to share the same defective allele at various gene loci. Recessive disorders may thus develop in the embryo, leading to higher rates of miscarriage. This reduction of fertility is called inbreeding depression.

• Outbreeding. Parents are more likely to combine genetic variants that have never co-adapted to each other. Incompatibilities may thus develop in the embryo, leading to higher rates of miscarriage. This reduction of fertility is called outbreeding depression.

Outbreeding has been on the rise in the West. Its effects might seem beneficial, as indicated by the upward trend of height and the downward trend of menarche. Both trends, however, have come to a halt: “In Northern Europe, adult height has largely stabilised, and the age of menarche has also settled at around 13 years, while weight continues to increase due to obesity” (Cole, 2003). The rise in IQ during the twentieth century, known as the Flynn effect, has sometimes been attributed to outbreeding, although that claim has been disputed (Flynn, 2007, pp. 101-102; Woodley, 2011). In any case, the Flynn effect too is coming to a halt throughout the West (Flynn, 2007, p. 143).

The above Icelandic and Danish studies, together with the rise of male infertility throughout the West, suggest that outbreeding has become more problematic than inbreeding. Yet most people seem to think otherwise. When I did a Google search for "inbreeding is bad," I got 15,200 hits. "Outbreeding is bad" got me only 7.

How bad can outbreeding be for humans? Davenport examined perhaps the maximum degree of outbreeding possible within our species, and only a minority of individuals from the cross-bred population showed more dysfunction than expected. The proportion varied from one psychological test to another, from a low of about 5% to a high of 30%. That proportion should decline over successive generations within an initially cross-bred population, as natural selection removes less functional individuals through illness, reduced fertility, and inability to find a mate.

Davenport interpreted his findings cautiously, noting the difficulty in estimating the size of the negative effect of outbreeding. He also analyzed his data at a time when statistical analysis was less advanced than it is today, with no tests of significance for race, age, or sex. His self-criticism was taken up by his critics, and the current prevailing view is that his study has been discredited, if not by methodological issues, then … “because.”

Postscript

When I first published this article in Aporia Magazine, it attracted much criticism, particularly from Razib Khan (Khan, 2024). I urge you to read his criticisms, as well as my reply (Frost, 2024). I’m not convinced by his arguments, which rely heavily on unnamed sources and unpublished studies that are impossible to verify. Moreover, these studies would have little value because their methodology is much less precise than those of the Icelandic and Danish studies.

On a final note, I have never claimed that the increase in outbreeding explains the decline in fertility rates throughout the world. Clearly, that decline has other causes.

References

Cole, T.J. (2003). The secular trend in human physical growth: a biological view. Economics & Human Biology, 1(2), 161-168. https://doi.org/10.1016/S1570-677X(02)00033-3

Davenport, C.B. & Steggerda, M. (1928). Race Crossing in Jamaica. Washington: Carnegie Institution, Publication no. 395. http://www.velesova-sloboda.info/archiv/pdf/davenport-race-crossing-in-jamaica.pdf   

Deonandan, R., & Jaleel, M. (2012). Global decline in semen quality: ignoring the developing world introduces selection bias. International Journal of General Medicine, 5, 303-306. https://doi.org/10.2147/ijgm.s30673

Flynn, J.R. (2007). What is Intelligence? Beyond the Flynn Effect. Cambridge University Press.

Frankham, R., Ballou, J.D., Eldridge, M.D., Lacy, R.C., Ralls, K., Dudash, M.R., & Fenster, C.B. (2011). Predicting the probability of outbreeding depression. Conservation Biology, 25(3), 465-475. https://doi.org/10.1111/j.1523-1739.2011.01662.x

Frost, P. (2024). Does fertility start to decrease beyond marriages with fourth cousins? Peter Frost’s Newsletter, August 13.

Helgason, A., Pálsson, S., Guðbjartsson, D.F., Kristjánsson, þ., & Stefánsson, K. (2008). An association between the kinship and fertility of human couples. Science, 319(5864), 813-816. https://doi.org/10.1126/science.1150232

Joffe, M. (2010). What has happened to human fertility? Human Reproduction, 25(2), 295-307. https://doi.org/10.1093/humrep/dep390  

Khan, R. (2024). The Genetic Time Machine: Neanderthals, genomics and hybridization priors. Aporia Magazine, January 16.

Labouriau, R., & Amorim, A. (2008). Comment on "An Association Between the Kinship and Fertility of Human Couples." Science, 322(5908), 1634. https://doi.org/10.1126/science.1161907  

Mayr, E. (1970). Populations, Species, and Evolution. Cambridge (Mass.): Belknap Press.

Presgraves, D.C. (2010). The molecular evolutionary basis of species formation. Nature Reviews Genetics, 11,175-180. https://doi.org/10.1038/nrg2718

Teasdale, T.W., & Owen, D.R. (2005). A long-term rise and recent decline in intelligence test performance: The Flynn Effect in reverse. Personality and Individual Differences, 39(4), 837-843. https://doi.org/10.1016/j.paid.2005.01.029

Turner, L.M., Schwahn, D.J., & Harr, B. (2011). Reduced male fertility is common but highly variable in form and severity in a natural house mouse hybrid zone. Evolution, 66(2), 443-458. https://doi.org/10.1111/j.1558-5646.2011.01445.x  

Woodley, M.A. (2011). Heterosis doesn't cause the Flynn effect: A critical examination of Mingroni (2007). Psychological Review, 118(4), 689-693. http://dx.doi.org/10.1037/a0024759