Abstract
Tracking the origins of new species and delimiting taxa across space and time present well-trodden sources of controversy for palaeoanthropology. Although biological diversity comes with frustratingly elusive boundaries, the task of describing and understanding diversity remains no less crucial, and palaeotaxonomy no more dispensable. This is epitomized by recent developments in discussions on our species’ origins and the extent to which Middle Pleistocene hominin forms represent distinct lineages. While it is tempting to think that progress in such debates is only hampered by the paucity of fossil and genomic data, we argue that problems also lie with unrealistic assumptions in theory. In particular, we examine ongoing discussions on whether Homo sapiens and Neanderthals deserve distinct species status as a means to advocate for the necessity of reframing speciation in palaeoanthropology in a more biologically plausible way. We argue that available palaeontological evidence is best interpreted under a framework that sees speciation as an evolutionary process that starts in space, thereby involving a geographical dimension, and progresses in time, thereby involving a diachronic dimension, with an incremental accumulation of relevant characters at different phases of the process. We begin by discussing evidence about species-level differentiation of H. sapiens and Neanderthals and analyse major sources of taxonomic disagreement, before illustrating the potential of this perspective in making progress on the earliest stages of H. sapiens speciation within Africa.
Introduction
Questions surrounding the origins and evolution of our species Homo sapiens have long been on the agenda of palaeoanthropological research. Yet, this topic has hardly become any less relevant today than it was half a century ago. While it can then be tempting to deem recurrent questions as a sign of stagnation and repetition within a discipline (see discussions in Delisle 2012), reiterating the same questions against novel data and analytical tools can drive the development of more nuanced models and finer-grained versions of the original puzzle. Debates on the origins and evolution of H. sapiens and its relationship with other coeval hominin forms—most notably, the Neanderthals—instantiate this pattern most glaringly (Galway-Witham and Stringer 2018, Meneganzin 2022) and will be at the core of this paper.
It is through this iterative questioning that we have reached a broad agreement that we are a species with a recent, (mostly) African origin (‘RAO’, Stringer 2022) and shared the planet with at least five other hominin species, and with some of which we interbred. Besides the Neanderthals, these include the Denisovans in East Eurasia and Island South East Asia, late Homo erectus, the diminutive species Homo floresiensis and Homo luzonensis in Island South East Asia, and, looking at some of the earliest stages of our speciation in Africa (~300 kya), Homo naledi, Homo heidelbergensis, and/or Homo rhodesiensis (see Galway-Witham et al. 2019 for a review). As a consequence, the scope of the debate moving within the RAO framework has become more wide-ranging and ambitious. Regarding early H. sapiens history, we now want to get a better grip on plausible scenarios of divergence and migration across the African continent (Bergström et al. 2021, Meneganzin et al. 2022, Ragsdale et al. 2023). Further, outside Africa, we want to focus on the dynamics of interaction of different hominin lineages in Eurasia to model their biological and cultural outcomes (Stringer and Crété 2022, Vallini et al. 2022). Palaeotaxonomy and models on species origins should aid in the description and analysis of hominin diversity and macroevolutionary patterns, supporting rather than distracting from relevant biological questions (Meneganzin and Bernardi 2023). So, what prevents us from gaining even more nuance within this picture?
This paper suggests that while the relative scarcity of fossil and genomic data is partly to blame (Bergström et al. 2021), some fundamental hindrances lie with theory. Disagreements about hominin taxonomy and models of H. sapiens origins in Africa appear to come down to assumptions about how the speciation process works and how to track this in the palaeontological and molecular records. Here we suggest that to make further progress on these problems, discussions on how speciation is modelled should be foregrounded. Indeed, speciation and taxonomic practice in palaeoanthropology have long entertained a complex dialogue with evolutionary theory (Tattersall 2012), with the record often being interpreted against biologically unrealistic expectations. While we are not alone in advocating a more critical reflection on the speciation process and its impact on hominin taxonomy (e.g. van Holstein and Foley 2022, Martin et al. 2024), here we discuss a framework for speciation to guide the integration of different evidential strands and the interpretation of morphology in H. sapiens origins.
This paper is structured as follows. Section 1 will provide a theoretical discussion of speciation in the fossil record, emphasizing the importance of considering speciation stages in a chronologically explicit context. Section 2 will address species delimitation in light of this view, drawing on discussions on H. sapiens and Neanderthal species status as an illustrative case of recurrent points of contention. Section 3 will discuss arguments favouring a lumping move and highlights sources of taxonomic disagreement. Section 4 will outline the implications of a spatially explicit and temporally extended framework of speciation for the initial phases of H. sapiens speciation. Section 5 concludes.
1. Framing speciation
In this paper, we maintain that the appreciation of the complexity of speciation as a process should inform species delimitation practice in palaeoanthropology and, crucially, the framing of early H. sapiens history within Africa.
The first claim requires some preliminary caveats. It is evident that in many contexts of enquiry (within and outside palaeoanthropology) taxonomic choices often need to be made in the absence of information on how speciation occurred or unfolded through time. While species and speciation are inherently co-dependent concepts, debates continue over the possibility of studying one without addressing or defining the other (cf. Zachos 2016: pp. 3–4, Doolittle 2018). We agree that patterns of differentiation at specific time horizons can be recognized without knowledge about how speciation occurred. However, we also believe that data on the evolutionary trajectory leading to that horizon, when available, do provide important contextual value for interpreting the observed pattern. Section 2 will argue that this is indeed crucial for framing the relationship of H. sapiens and Neanderthals (and other sister lineages) and for adopting biologically meaningful taxonomies. In fact, the suggestion that a focus on speciation should take the lead in informing species delimitation is not new to the debate, as species are typically the product of speciation processes (cf. Coyne and Orr 2004). This shift in emphasis has been defended even in cases where speciation does not generate discrete species (e.g. Novick and Doolittle 2021). While our palaeoanthropological focus cannot give justice to the depth of the broader theoretical debate (for a useful overview of current issues see Wilkins et al. 2022), it will foreground concerns specific to the study of palaeobiological diversity and taxonomy.
Here, looking at the context of H. sapiens origins and evolution, we argue that focusing on understanding the speciation process is of particular value for moving current debates toward more productive directions. In particular, we focus on two aspects where palaeoanthropology often falls short of capturing the complexity of speciation. Broadly, these challenges can be summarized as follows: speciation is a process that (i) starts in space and (ii) progresses in time through different stages of speciation, with the degree of differentiation observed at a specific stage being a function of divergence time. We will proceed in reverse order, starting with (ii) and related issues of species delimitation, and addressing (i) later.
1.1. Stages of speciation
Speciation is a special kind of lineage-splitting evolutionary process, in which one ancestral species generates one or more descendant species. Fundamentally, what speciation comprises is the study of divergence—the accumulation of different traits resulting from a combination of mutation, natural selection, phenotypic plasticity, or drift. However, divergence, albeit necessary, is not sufficient for tracking speciation, as divergence can also take place at a lower (populational) level. What further criteria are needed to legitimize the species status of separately evolving lineages is precisely the problem of species delimitation—the imposition of a dichotomous categorization on a dynamic process (Zachos 2016).
What has always been clear in the study of speciation is that the process does not resolve with the initial divergence impetus but progresses over time, with populations showing varying degrees of divergence along the so-called ‘speciation continuum’. The idea that speciation should be addressed by studying the stages that populations pass in the process of becoming new species traces back to Alfred Russel Wallace (1865, see also discussions in Lowry 2012, Kulmuni et al. 2020). This basic but fundamental intuition has been famously illustrated by Mayr’s textbook model of the stages of allopatric speciation (Mayr 1942)—a sequence, to be read top-down, showing the initial detachment of a daughter population from the peripheral range of an ancestral metapopulation by means of a geographical barrier, and the progressive accumulation of divergence, finally causing reproductive isolation to evolve.
More recently, Allmon and Sampson (2016), building on Allmon (1992), have proposed a four-stage framework for studying animal speciation in the fossil record, which we will draw on here (see the schema at the bottom of Fig. 1A). This is defined by the formation of (i) population isolates by means of mechanisms dramatically reducing gene flow, (ii) their persistence in isolation (i.e. neither going extinct nor merging with the parental population via interbreeding), (iii) their differentiation granting them separate evolutionary status, and (iv) finally the persistence and population expansion of newly formed species for long enough to play different evolutionary roles (cf. Allmon and Sampson 2016: 125–6). Each stage may be influenced by a variety of factors, some of which can indeed be identified via palaeontological and geological evidence—from environmental disturbance and dispersal ability affecting the isolate formation stage, to the study of adaptation and the production of phenotypic differences between populations tracking differentiation, to species abundance in the latest stage (see our discussion in Sections 2 and 4).
Homo sapiens and Neanderthal divergence along the speciation continuum. A, a selection of fossils, together with their lineage attributions and major events. Grey dots indicate fossils suggested to belong to the H. sapiens lineage, but their attribution is still currently being debated. Redrawn after information reviewed in Bergström et al. (2021). Below, Allmon and Sampson’s four-stage framework for studying animal speciation mapping the ‘speciation continuum’ (redrawn after Nosil et al. 2009). The isolate persistence and differentiation phase are drawn in parallel as these can occur together and the latter requires the former, though the reverse is not true. B, location of a selection of H. sapiens and Neanderthal fossils from the past 500 kyr.
Crucially, the differentiation phase (and also the persistence one, as these can occur together, see Fig. 1A) can be thought of as marked by many other time horizons or ‘thresholds’ in which the diverging populations acquire different diagnostic properties—morphological, genetic, ecological, etc.—relevant for distinct operational species criteria (De Queiroz 1998, 2007). More specifically, these thresholds refer to populations becoming, for instance, morphologically distinguishable groups, forming identifiable genotypic clusters and establishing nonoverlapping adaptive zones. The crossing of these diagnostic thresholds need not occur in a strict order or by a strict tempo: this framework is compatible with multiple patterns of divergence, with some properties being acquired more ‘quickly’ or ‘slowly’ on a palaeontological scale. Moreover, schematic, lineage-based representations indicating thresholds being crossed at specific points in time should not be interpreted as divergence halting at those points (see Meneganzin and Bernardi 2023). Divergence progresses even when the relative threshold can be said to have been crossed. This ‘grey area’ is notoriously the one where tracking speciation is most tricky, as it clusters most disagreement in the evolutionary continuum from an (ideal) fully interconnected metapopulation to purportedly ‘good species’.
This emphasis on stages and phases and their practical utility has been received with scepticism by those acknowledging an unavoidable degree of subjectivity in establishing when a phase has been completed or otherwise along a continuum (Stankovski and Ravinet 2021). However, instead of shifting the search for a yes-or-no watershed to the level of stages, it is much more realistic to assume that morphological distinctiveness, reproductive isolation, and other criteria used for species delimitation vary in a continuous fashion. That is, instead of an unproductive way of multiplying yardsticks, we consider this framework to have heuristic value in asking how far speciation has progressed and whether it has approached at a stage where reversal has become increasingly unlikely or difficult. So the speciation process neither is one-way—it can reverse, especially in the earliest stages—nor is it unidimensional, but rather multivariate (Dieckmann et al. 2004, Bolnick et al. 2023). This highlights the limitations of focusing the analysis on one single threshold or axis of differentiation (like full reproductive isolation, as we shall see).
It is often claimed that establishing when a point of no return has been reached—that is, when the split has become permanent—can only be done in retrospect (Kornet 1993, see Maddison and Whitton 2023 on retrospective units). Interestingly, palaeontology and palaeoanthropology, for the very nature of the broad temporal scales adopted and the focus on the past, can provide contexts where some benefit of hindsight is offered on divergence not being reversed, even when favourable circumstances are present. Further, when data about divergence times are available (e.g. molecular estimates, as for recent hominin evolution) or are inferable as preceding first appearance data (FADs)—keeping in mind that molecular and fossil data capture different aspects of the speciation process—the degree of differentiation observed at a specific temporal horizon tx can be interpreted within the context of divergence time from t0.
That speciation is a time-extended process unfolding through various stages might therefore not sound particularly controversial to an evolutionary biologist. However, recent calls in palaeoanthropology for the importance of integrating chronological information in the study of species diversity (Martin et al. 2024) and taxonomic debates (van Holstein and Foley 2022, Meneganzin and Bernardi 2023) indicate that discussions on speciation and species have yet to appreciably diverge from long-standing typological tendencies within the discipline. While past human variation (primarily morphological for deeper evolutionary times, but also genetic for more recent ones) remains a crucial window for classification and evolution, the temporal and geographical patterning of such variation—of which individual fossils and genomes serve as data points—still holds rather neglected explanatory potential. Regarding taxonomy, what is particularly interesting to palaeoanthropology is not that debates over classification are still (unsurprisingly) rife but on what terms these are conducted. In what follows, we analyse more explicitly species delimitation in light of the above-discussed framework on speciation.
2. Species delimitation
It is generally held that some progress in the practice of taxonomy and the ‘species problem’ has been achieved by dividing the conceptual labour, which implies hierarchically distinguishing primary or nonoperational species concepts from secondary or operational concepts that provide evidence to track a primary one (Mayden 1997, De Queiroz 1998, 2007). From our previous discussion and the apparent growing consensus about the conceptual framework in use in scientific research (Conix 2022), and particularly in palaeontological contexts (cf. Allmon and Yacobucci 2016), species are theoretically seen as separately evolving metapopulation lineages, in line with Simpson’s and Wiley’s Evolutionary Species Concept (Simpson 1951, 1961, Wiley 1978) and De Queiroz’s General Lineage or Unified Species Concept (De Queiroz 1998, 2007). While this does not per se solve the delimitation problem, it can have heuristic value in framing the evidence for tracking speciation’s progression and laying the ground for justifying taxonomic choices. Let us consider now current discussions on the species-level delimitation of H. sapiens and Neanderthals.
2.1. Homo sapiens and Neanderthal lineages: isolate formation and persistence
The species status of Neanderthals with respect to H. sapiens has historically been subject to iterative reassessment, tracking the pendulum of scientific consensus on contrasting models of our species origins (Stringer 2014, 2022). As such, the taxonomic status of Neanderthals has oscillated between distinct species status (Homo neanderthalensis) to subspecies (or population) of H. sapiens (Homo sapiens neanderthalensis) (Harvati 2003). The current debate is arguably driven by input from molecular and archaeological data, with increasing evidence of interbreeding between the two lineages (see Liu et al. 2021 for a review) and behavioural complexity motivating suggestions to drop specific taxonomies in favour of an expanded H. sapiens concept. To see whether this can be justified and where disagreement stems from, let us analyse this issue from the perspective of the stage-based framework delineated above—that is, let us consider whether Allmon and Sampson’s (2016) framework is recognizable in H. sapiens and Neanderthal divergence along the speciation continuum.
First, can we identify an isolate formation stage from an ancestral metapopulation? For populations to have the potential to become distinct evolutionary lineages and species, they must undergo a phase of separation from the parental species. While different dynamics can form the basis for a substantial reduction (or cessation) of gene flow, this is commonly initiated by means of geographical separation in animals (Hernández-Hernández et al. 2021). A combination of spatial separation and divergent selection would place a population (an ‘isolate’) on a distinct evolutionary trajectory.
Currently, debates persist on the identification of the last common ancestor (LCA) shared by Neanderthals, H. sapiens, and Denisovans (Bergström et al. 2021), with some scholars suggesting the polymorphic species H. heidelbergensis represented in Africa and Eurasia (Di Vincenzo and Manzi 2023), others the species Homo antecessor, known from the site of the Sierra de Atapuerca, in Spain, on morphological and palaeoproteomic grounds (Welker et al. 2020). As such, the modalities and dynamics of the initial isolate formation from the parental species at the basis of H. sapiens and Neanderthal divergence are an active area of research, much of which hinges on the interpretation of elusive Middle Pleistocene fossils (Hautavoine et al. 2024). This is carried over into debates on the specifics of the African origins of our species (we will turn to these in Section 4). However, a few facts hold firm. There is no dispute over the fact that H. sapiens and Neanderthals bud off from the same parental species and spent a significant degree of their evolutionary histories separate from each other (Galway-Witham et al. 2019, Bergström et al. 2021). Current molecular estimates based on nuclear DNA (nDNA) place the divergence between the Neanderthal and H. sapiens lineage between 500 and 700 kya (Prüfer et al. 2014), with suggestions of earlier dates based on dental evolutionary rates (pre-800 kya, Gómez-Robles 2019) and phenomic analyses (around 1 Mya; Ni et al. 2021, Feng et al. 2024). As regards the degree of respective isolation, fossil and molecular evidence indicate that Neanderthals evolved in Eurasia for at least 400 kyr (as indicated by the Spanish site of Sima de los Huesos; Arnold et al. 2014, Arsuaga et al. 2014) while our species evolved in Africa over a timeframe of at least ~300 kyr (Hublin et al. 2017, Bergström et al. 2021, Meneganzin et al. 2022). As such, sceptics should concede that H. sapiens and Neanderthals have at least reached the stage of ‘incipient species’ (i) by means of formation of population isolates (Allmon and Sampson 2016).
However, not all incipient species go on to become an established species. It has long been suggested that daughter populations form much more frequently than novel species (Mayr 1963, Allmon and Sampson 2016). In the context of H. sapiens and Neanderthals, it has become increasingly clear that such isolation was far from perfect from the earliest stages. This should not be surprising, as perfect allopatry is rarely realized, even more so when one adopts a process-based perspective on speciation (Endler 1977). Analyses of Neanderthal mitochondrial DNA (mtDNA) and Y chromosome suggest the possibility of early H. sapiens dispersals in Eurasia, tracked by signals of gene flow from an African source, potentially leading to a complete replacement of the initial Neanderthal mtDNA and Y chromosome (probably due to the low effective population size of Neanderthals; Petr et al. 2020, Posth et al. 2017; see Fig. 1A). Early H. sapiens excursions into Eurasia have also been suggested based on fossil material (notably, from the Greek cave of Apidima; Harvati et al. 2019, but also from sites in the Levant, such as Misliya, Skhul, and Qafzeh; Hershkovitz et al. 2018, Grün and Stringer 2023). The question is whether the dynamics of these encounters precluded speciation progression towards isolate persistence (ii) and further differentiation (iii). To see this, we must consider the different fates that newly formed isolated populations might encounter (Mayr 1963, Allmon and Sampson 2016).
Newly formed isolated populations might incur a range of possible outcomes, from scenarios of disappearance of the incipient species, either via merging with the parent population or with daughter populations or by extinction, to scenarios of persistence, either persistence in isolation without achieving reproductive isolation, or persistence accompanied by differentiation to become neospecies (Allmon and Sampson 2016: p. 131). The evolutionary history of the H. sapiens and Neanderthal lineages arguably falls within a scenario of persistence. After several hundreds of thousands of years persisting across Eurasia, Neanderthals disappear at around 40 kya (albeit probably with different timing in different regions, not indicative of an abrupt event; Higham et al. 2014, Currie and Meneganzin 2022). Apart from earlier dispersals mentioned above, which putatively involved zones of contact at the boundary of H. sapiens and Neanderthal respective geographical ranges, our species’ history has mostly unfolded on the African continent. A well-known major range expansion event—the so-called Out-of-Africa—is estimated to have occurred around 60 kya (Soares et al 2012, Pagani et al. 2016, Mallick et al. 2016; see Fig. 1A), plausibly involving different waves of expansion of H. sapiens into Eurasia before stably colonizing it at around 45 kya (Vallini et al. 2024, see also Iasi et al. 2024). Therefore, multiple strands of evidence available so far point at speciation having also reached the second stage along the continuum (ii).
2.2. Homo sapiens and Neanderthal lineages: differentiation and species persistence
Moving forward, we ask now: did lineage persistence lead to appreciable differentiation between the two lineages to grant them separate evolutionary status? (iii) The different evolutionary ‘tendencies’ and ‘fates’ (Wiley 1978) encountered by the two lineages can by themselves be leveraged as an evidential window into the evolutionary distinctiveness reached by H. sapiens and Neanderthals by the time of the extinction of the latter. However, a more complete argument is offered by the abundant evidence of significant differentiation between the two (reviewed in Meneganzin and Bernardi 2023). This can be captured by tracing the acquisition of diagnostic features relevant to different operational species concepts (see Fig. 2)—chiefly, morphological, genetic, and ecological ones. This helps track speciation progression along different axes of variation. Let us briefly see these in turn.
In the ‘grey area’ of speciation, lineages diverge along multiple axes of variation, crossing different diagnostic ‘thresholds’ (as identified by various secondary species concepts) in parallel or in sequence. The thresholds are indicated with various dashed lines, to signal these should not be interpreted as all-or-nothing yardsticks. Divergence progression and threshold crossing need not occur in a strict order or by a strict tempo.
From a morphological perspective, it should be noted that demarcating H. sapiens and Neanderthals has never been a serious matter of contention. Some have deemed the Neanderthals as ‘the most clearly demarcated extinct hominin groups’ (Tattersall and Schwartz 2006). Compared to H. sapiens, Neanderthals are distinguished by a set of cranial, dental, and postcranial features, although debates persist on which should be considered synapomorphic, autapomorphic, or symplesiomorphic. Reference traits include an anteroposteriorly elongated cranial vault, a protruding midface, distinctive middle and inner ear bone shapes, a strong double-arched browridge, large incisors, no (or incipient) chin (but see Meneganzin et al. 2024), a wide and deep ribcage, and a wide and flared pelvis (for a detailed review, see White et al. 2014). While an exhaustive morphological assessment is beyond the scope of this paper, what is relevant from the perspective on speciation proposed here is that morphological differences seem to be recognizable at virtually all stages of the speciation process—from the earlier known stages represented by the Sima de los Huesos specimens (Arsuaga et al. 2014) to the later conditions of sympatry with H. sapiens in the Levant (Rak 1993, Hovers et al. 1995). While the full suite of Neanderthal derived features did not appear as a package (e.g. the Sima de los Huesos specimens do not display the occipital bun or ‘chignon’ typical of ‘classic Neanderthals’), the fact that morphological differences between the two lineages are evolutionarily deep and persist even after well-known phases of interbreeding (which could have led to lineage fusion or homogenization) is indicative of far-reaching distinct adaptive and physiological trajectories (see also Pomeroy 2023). Further, as estimates of intraspecific variation in fossil samples are typically informed by variation among extant primate taxa (but see Wood and Smith 2022 for critical considerations), it is relevant to note that dropping species-level demarcation between H. sapiens and Neanderthals would produce a taxon with an anomalous range of variation when compared to other primates. As an example, we can consider the tiny auditory ossicles of the middle ear, which form early in fetal development and are fully ossified at birth. An analysis based on a large sample of Neanderthal auditory ossicles, covering a wide geographical and temporal range, found striking morphological differences from H. sapiens, with average shape distance between the two exceeding in some cases that between H. sapiens and Pan troglodytes (as regards the malleus and the stapes, Stoessel et al. 2016). As such, we believe that differentiation between the two lineages is readily manifested along the axis of morphological variation, with speciation having progressed beyond the threshold of producing two well-diagnosable morphological (or ‘phylogenetic’) species.
From an ecological perspective, the prolonged occupation by H. sapiens and Neanderthal populations of different geographical niches until the former’s Out-of-Africa range expansion already speaks to different ecological and environmental dynamics underlying the early evolution of the two lineages. This is tied to discussions of niche-specific metabolic and anatomical adaptations. While simple correlations between the glacial environments of western Eurasia and the Neanderthal stocky physique, short limbs, high body mass, and large noses are increasingly challenged (reviewed in Pomeroy 2023), links between phenotypic traits and niche-contextual diet, metabolism, and physiology remain plausible within more integrated frameworks. Neanderthal diet appears today more varied than previously appreciated, including seafood, plants, and carbohydrates (Power et al. 2018, Hardy et al. 2022). However, the weight of evidence still seems to point to a diet composed of a high proportion of animal products (Jaouen et al. 2019), especially in environments with scarce plant food availability, with implications for the Neanderthal build, and possibly also ribcage shape and pelvis (Ben-Dor et al. 2016), as well as physiology. Crucially, differences between H. sapiens and Neanderthal resource exploitation strategies seem to be indicated by multiple strands of evidence. These are tied to distinctive energetic and locomotor features, as suggested by the Neanderthal occupation of woodland environments, where power-locomotion (rather than endurance) is well-suited (Stewart et al. 2019). Differential territory exploration is also evidenced in the Levant, with Upper Palaeolithic groups seemingly exploring wider settlement ranges with varied terrain typologies, whereas Neanderthals showed a preference for rugged terrain within smaller areas of their sites (Henry et al. 2017). The differential responses to climate-driven niche deterioration provide a further line of evidence. Shifts in the Neanderthal diets in response to changing palaeoecological conditions have been registered towards the end of Marine Isotope Stage 3 (El Zaatari et al. 2016). Progressive loss of habitat quality and connectivity among Neanderthal populations has been traced throughout the ~8000 years preceding their physical extinction in Europe (Melchionna et al. 2018). The contraction of suitable habitats probably began even earlier, from about 70 kya, with ecological niche dynamics resulting from the occupation of habitual territories characterized by novel environmental conditions (Banks et al. 2021). As such, we believe that available evidence indicates Neanderthals and H. sapiens had differentiated to the point of occupying and exploring ‘minimally different’ adaptive zones (sensuVan Valen 1976), which of course does not exclude the possibility of competition in known areas of overlap.
From a genetic perspective, analytical and methodological efforts to sequence high-quality Neanderthal genomes over the past 15 years, along with studies of present-day human diversity, have provided unique insights into patterns of divergence, interaction, and accumulation of lineage-specific molecular events since the split from the last common ancestor (Meyer et al. 2012, Prüfer et al. 2014, 2017, Mafessoni et al. 2020). Over (at least) 500–700 thousand years of evolutionary history, while sharing ~99.7% of their DNA, the Neanderthal and H. sapiens lineages have accumulated an array of genetic differences that ensure their genetic distinguishability (Green et al. 2010, Prüfer et al. 2014). Efforts to identify lineage-specific genetic variants are producing catalogues with increased resolution, with genetic variants emerging among Neanderthals linked to metabolism, sensory organs, gestation, and immune system (Kuhlwilm and Boeckx 2019, Zeberg et al. 2024). However, these do not (nor can) suggest direct associations with complex behavioural and cognitive traits (or ‘behavioural modernity’, Meneganzin and Currie 2022), as complex phenotypes require investigation within multicausal and coevolutionary frameworks. It is therefore possible to make the argument that, despite limited contacts, the Neanderthal and H. sapiens lineages formed ‘genotypic clusters’ (sensuMallet 2020), with their respective metapopulations sometimes overlapping in space and time without complete fusion or becoming genetically or phenotypically indistinguishable.
At the same time, evidence of gene flow between the two lineages is now well-established (reviewed in Bergström et al. 2021, Stringer and Crété 2022), indicating that reproductive isolation, another key criterion for species delimitation, was not entirely achieved between them. This has notably left the ~2% signature of Neanderthal ancestry in most non-Africans, with East Asians exhibiting ~20% more Neanderthal ancestry when compared to West Eurasians. The relative consistency of this signature in present-day humans suggests that the bulk of Neanderthal contribution happened during the Out-of-Africa range expansion, and possibly as late as 47 kya (Zeberg et al. 2024). However, how selection has shaped the pattern of Neanderthal ancestry across the human genome through time is a relevant line of evidence for tracking speciation progression and establishing the evolutionary distinctiveness between the two lineages. Several studies have identified signals of purifying selection against Neanderthal ancestry (Harris and Nielsen 2016, Juric et al. 2016), with strong depletion around functional regions of the genome, on the X chromosome, and in brain- and testis-expressed genes (Sankararaman et al. 2014, Vernot et al. 2016). This can be due to a variety of reasons, from Dobzhansky–Muller incompatibilities, to Neanderthal alleles being maladaptive in the recipient species’ ecology, or the higher genetic load carried by Neanderthal populations. While the average deleteriousness of Neanderthal ancestry is acknowledged, there are indications of some sequences being beneficial, thereby serving as plausible candidates for adaptive introgression (reviewed in Racimo et al. 2015). Interestingly, recent analyses suggest a relatively rapid selection acting on Neanderthal variants, already taking place in early Out-of-Africa individuals (Iasi et al. 2024, see also Harris and Nielsen 2016). If all of this is correct, the action of selection as inferable from genetic data, together with morphological and ecological evidence mentioned above, seems to point to speciation having advanced beyond the threshold of producing appreciably differentiated evolutionary and adaptive trajectories (iii).
Consider now the final stage in Allmon and Sampson’s framework (iv)—that is, the persistence and population expansion of newly formed species for long enough to play different evolutionary roles. We note that in the original framework, this is presented as a stage following the establishment of reproductive isolation (cf. Allmon and Sampson 2016: p. 135). While this was not completed in the case of H. sapiens and Neanderthals, their visibility in the fossil record implies a story of persistence, accompanied by the accumulation of more adaptations, with further population expansion experienced by our ancestors on their way to becoming a global species. The Neanderthal disappearance from the fossil record at ~40 kya only happened at later stages of a speciation process which, we believe, grants them separate evolutionary status and taxonomic identity—Homo neanderthalensis.
3. Diagnosing disagreement
Having delineated H. sapiens and Neanderthal species-level distinctiveness against the backdrop of a speciation process moving along different axes of variation and thus crossing different taxonomically relevant ‘thresholds’, let us consider and analyse more closely two arguments that would favour a lumping move. For simplicity, we will call the first the ‘argument from interbreeding’ and the second the ‘argument from behavioural complexity’.
The first stems from evidence of genetic exchanges between the two lineages, as mentioned above, which also now draw on the genetic evidence of the early colonization of Eurasia, with different H. sapiens individuals showing various degrees of Neanderthal ancestry, albeit with different contributions to subsequent populations (Stringer and Crété 2022, Meneganzin and Bernardi 2023, Vallini et al. 2024). In light of the evidence of gene flow between H. sapiens and Neanderthals, Pääbo notably suggested that we talk of one hominin ‘metapopulation’, with ‘populations linked by limited, but intermittent or sometimes perhaps even persistent, gene flow’, thus avoiding the question of species designation (Pääbo 2015). Other suggestions explicitly invoke a strict application of Mayr’s Biological Species Concept (BSC, Mayr 1942) to claim that not only the Neanderthals but also the Denisovans should be included in the same taxon as H. sapiens (e.g. see Hawks in Gibbons 2011). Therefore, it could be tempting to interpret these suggestions as underwriting a textbook conflict arising between morphology-based and molecular-based taxonomies. The latter, generally inapplicable in the fossil record, would now demand evidential prioritization over inferences from morphology.
While this diagnosis may seem prima facie plausible, we think this debate is rather grounded in biologically unrealistic and static models of speciation still at play in palaeoanthropology, in particular as regards the lack of appreciation of the temporal dimension of speciation. There is no genuine biological conflict arising from evidence of gene flow in the presence of strong morphological distinguishability, if evaluated within the context of the speciation continuum. Reproductive isolation, where this can be empirically detected, can notoriously remain incomplete even in advanced stages of the speciation process, taking up to millions of years in mammals (Coyne and Orr 2004). It is under this rationale that Mayr himself had argued that species-level should be considered reached ‘when the process of speciation has become irreversible, even if some of the (component) isolating mechanisms have not yet reached perfection’ (Mayr 1963: p. 26). While a degree of reversibility is technically possible until the evolution of full reproductive isolation, we take this to mean that it is legitimate to consider speciation nearly completed (or pragmatically completed) in the vicinity of the second end of the continuum, even in the face of evidence of limited introgression escaping isolating mechanisms. This clearly bears implications beyond what strictly concerns human evolution: if evidence of introgression is sufficient to deny species status, species limits would need to be dramatically revised in many other taxa (see Meneganzin and Bernardi 2023, section 5, for a more detailed discussion). What is crucial in the palaeoanthropological context is that different types of data—whether morphological, genetic, or behavioural (see also the discussion below)—be interpreted as time-contextual samples of the time-extended process of speciation. The taxonomic question should then include a temporal component: when did two hominin forms become reasonably separate species? The benefit of hindsight provided by our perspective of lone survivors should admit different answers as we move our focus from the later to the earliest stages of the speciation process.
The second argument—the argument from ‘behavioural complexity’—foregrounds evidence of increasing cultural and behavioural sophistication now being attributed to Neanderthals. These include various forms of personal ornamentation, possible cave ‘art’, complex underground construction, cordage, medicinal use of plants, support for disabled individuals, and numerical cognition (reviewed in Nowell 2023). While we welcome current updates to the Neanderthal behavioural repertoire and consider that the behavioural gap has certainly narrowed over the past years (see d’Errico and Stringer 2011, Galway-Witham et al. 2019, Meneganzin and Currie 2022, Meneganzin and Killin 2024), we also want to raise caution against transforming the Neanderthals into a stockier version of ourselves, particularly while the strength of the evidence for some these behavioural traits is still under discussion. As concerns the taxonomic argument, taking behavioural and cultural traits at face value as proxies of species boundaries can be misleading. We think there are two main reasons for this. First, these traits are highly dynamic entities, following channels of inheritance that often depart from biological ones (Kronfeldner 2021). Not only are the transmission patterns of cultural traits not confined within taxonomic affiliations—they can travel across them—but patterns of cultural change within human evolution more broadly do not neatly map onto biological ones, as demonstrated by visualizing technological change and lithic ‘modes’ against a hominin fossil timeline (see Galway-Witham et al. 2019). Second, and crucially for the main message of this paper, the argument from behavioural complexity again seems to ignore a diachronic, evolutionary contextualization. It should not be surprising for species with an evolutionarily recent common ancestor to share behavioural and cultural practices, some of which have a deeper evolutionary history than previously appreciated (see Meneganzin and Killin 2024 for discussions on aesthetic capacity). However, it is important to contextualize these within their respective evolutionary trajectories, which show differential patterning and exploration (and/or preference) towards specific cultural practices. Lumping the Neanderthals within an expanded H. sapiens taxon in light of the evidence of behavioural complexity would keep this unjustifiably hostage to our evolutionary history alone, at the same time stripping the Neanderthals of their evolutionary identity.
The discussion presented above carries implications beyond what strictly concerns H. sapiens and Neanderthals. Taxonomic disagreement is frequently cited as integral to palaeoanthropological practice, with debates often extending beyond the mere availability of new fossil data, persisting instead over the degree of species richness in long-sampled portions of the hominin tree. Drawing from the conceptual labour division between primary/nonoperational and secondary/operational approaches to species, Zachos (2022) helpfully distinguishes two hearts of the taxonomic enterprise. The first is a descriptive and quantitative analysis of biodiversity in space and time (along its morphological, genetic, ecological, etc. dimensions); the second concerns the translation of these results in species status, unavoidably implying an executive decision. Although taxonomic practice is often operationalized in terms of hypothesis generation and hypothesis testing (e.g. Martin et al. 2024 for a recent example), this arguably pertains only to the first problem—that is individuating separately evolving metapopulation lineages via available evidential tools. However, the act of classifying these as species along the problematic continuum requires argument. We follow Thiele et al. (2021) and Zachos (2022) in suggesting the map-making analogy for this second problem, arguing, in the palaeoanthropological context, that the territory we should aim at mapping should be the degree of advancement of the speciation process. As we saw in the previous section, when looking in hindsight at some cases of sister lineage divergence, more of the territory would arguably be lost via a lumping move rather than by granting distinct species status.
4. Homo sapiens origins
The perspective discussed above bears important implications for the developing debate on the earliest stages of H. sapiens origins within the African continent. While several proposals are now emphasizing the complex dynamics at the roots of our lineage, these are rarely discussed against the backdrop of speciation theory. Yet, assumptions on how H. sapiens speciation has probably occurred are certainly at play and we believe an explicit framing would make available evidence less prone to multiple, underdetermined interpretations. In this section, we delineate open issues and directions for future research.
Proposed models of Middle Pleistocene evolution in Africa available range across a spectrum, from a simple single-origin model—where all derived traits of H. sapiens emerge from a single, localized ancestral population—to the pan-African model, which posits a polycentric origin of H. sapiens autapomorphies within a highly structured meta-population distributed across Africa and connected by gene flow (for reviews, see Henn et al. 2018, Scerri et al. 2018, Bergström et al. 2021, Meneganzin et al. 2022, Ragsdale et al. 2023). It has become increasingly clear that a simple single-origin model excluding multiple ancestral contributions prior to H. sapiens differentiation is theoretically and empirically simplistic, being especially at odds with demographic models best explaining patterns of present-day genetic diversity. However, the scattered African fossil record from the 400–200 kya time range (see Fig. 1B), combined with these models, does not yet possess the resolution to discriminate among different scenarios underlying the observed timescale and patterns of shared genetic ancestry (Bergström et al. 2021).
A recent study has concluded that the earliest genetic divergence among contemporary human populations dates to 110–135 kya and that despite the challenge of discriminating between demographic configurations prior to that temporal threshold, a higher likelihood can be attributed to a scenario in which the population ancestral to all present-day humans results from the merger of two stem populations which experienced both gene flow between them and an extended period of drift (Ragsdale et al. 2023). Rather than one-sidedly supporting long-term population structure (or African ‘multiregionalism’), this seems to be best interpreted as a mixed model, combining elements of multiple ancestral populational contributions (a tenet of the pan-African scenario) with more recent times of population divergence (as postulated by models of geographical expansion from a more localized source) (cf. Ragsdale et al. 2023: p. 760). This scenario resonates with earlier proposals from phenotype-based phylogenetic modelling and theoretical analysis emphasizing a complex and prolonged evolution of H. sapiens, including a phase of phenotypic differentiation among demes of late Middle Pleistocene hominins, followed by a dynamic phase of fragmentation, mixing, and coalescence into a more derived hominin form at more local scales (Mounier and Mirazón Lahr 2019, Meneganzin et al. 2022, see Fig. 3). The time depth of current human population structure is, however, prone to continuous re-evaluation. Approaches aimed at reconstructing current human genetic ancestry can yield different estimates due to distinct modelling assumptions (see Cousins et al. 2024 for a merger event at ~300 kya of two populations ancestral to present-day humans).
Possible scenarios in Homo sapiens origins. From a structured African metapopulation, in which local populations carried various combinations of ancestral (green) and derived (yellow) characters, available models point to a geographically widespread (‘pan-African’) or a more regionally circumscribed (allopatric) speciation. Key in discriminating among these models is to gain nuance on the ecological, geographical, and climatic factors modulating probabilities of population distribution, connectivity (indicated by dashed lines), and vicariance (with barriers indicated by solid lines).
While it is indisputable that many crucial details within this broad picture are still missing, we think that the interpretation of available records (be they genetic or morphological) should be more explicitly oriented at explaining how the earliest stages in Allmon and Sampson’s framework unfolded. Specifically, more concerted efforts are needed to clarify the dynamics leading to the formation of (arguably imperfect) isolates and their subsequent persistence among late Middle Pleistocene hominins. This implies addressing the palaeontologically visible mechanisms governing these stages, including evidence of environmental disturbances modulating vicariance and conditions for dispersal ability—differently put, the evolutionary geography underlying species formation (Lahr and Foley 1998, Lahr 2016, Foley 2018).
It is one thing to claim that proposed models match available patchy data (which often allow for multiple interpretations); it is another to put these in an explicit geographical and ecological context that explains how speciation unfolded. In particular, working within a plausible theoretical framework of speciation implies bringing the spatial dimension, which lies at the core of this evolutionary process, to the forefront of current investigations. Climatic and palaeoecological models offer a crucial template for framing demographic discontinuities and speciation dynamics (Blome et al. 2012, Maslin et al. 2014). Climate variability in the Pleistocene is well established, and available records are gaining increasing resolution in palaeoenvironmental regional contexts, with the potential to shed light on patterns of local adaptation, drift, and differential survival of hominin lineages (see Foerster et al. 2022). Simulations of climate-induced habitat changes provide further insights. While mapping climate events to speciation events is extremely complex, it is however possible to analyse whether aspects of the fossil, genomic, and archaeological record track inferred habitable regions as the climate shifted over time. A recent example is offered by the work of Timmermann et al. (2022), in which the combination of archaeological and fossil datasets with a simulation of the Earth’s astronomically driven climate history has allowed inferences into the habitat suitability of distinct hominin groups over time and to identify possible linkages between regional shifts and evolutionary diversification. Understanding the patchwork of habitable areas in the past, along with their degree of connectivity, is particularly important for gaining insights into the dynamics shaping population patterns and isolate formation that underpin the early stages of the speciation process.
In the context of multiple hominin lineages coexisting during the late Middle Pleistocene, this research is inextricably tied to taxonomically interpreting hominin diversity against a scenario of speciation that is geographically explicit and incorporates a temporal dimension of diversification along various axes of variation. The morphological variation exhibited by the late Middle Pleistocene fossil record, with often difficult-to-interpret mosaic combinations of features eluding taxonomic consensus (DiVincenzo and Manzi 2023, Feng et al. 2024, Hautavoine et al. 2024), can be reconfigured as a source of evidence, rather than a stumbling block, for tracking evolutionary processes shaping diversity in time and space.
Without an appropriate theoretical framework able to do justice to the complexity of the speciation process in palaeoanthropology, the integration of available strands of evidence and the revision of available models in light of incoming information and methodological approaches can only become more challenging rather than increasingly more informative.
5. Summary and conclusion
In this paper, we have argued that emphasizing the study of speciation within its temporal and geographical contexts is crucial for better framing and potentially reducing taxonomic conflicts in the palaeoanthropological investigation of our species’ origins and Middle Pleistocene diversity. These conflicts are often rooted not only in limited evidence but also in insufficient theoretical foundations. We began by outlining a framework for studying speciation in palaeoanthropology, drawing from broader palaeontological discussions (Allmon 1992, Allmon and Yacobucci 2016), and applied this framework to the issue of species delimitation. By examining the species status of Neanderthals and H. sapiens, we tracked speciation progression through various stages, as inferred from different available evidential strands. We identified a major source of taxonomic controversy in the lack of evolutionary and chronological contextualization of observed patterns of variation. Finally, we proposed future research directions focusing on the earliest stages of H. sapiens speciation, emphasizing the need to address the mechanisms of early species formation and their underlying evolutionary geography.
The origins of H. sapiens and our interactions with contemporary hominin forms are subjects of ongoing debate and reconfiguration. In this dynamic field, where the interpretation of both past and novel evidence is often contentious, explicit theoretical frameworks are essential to avoid investigative deadlocks and productively move research forward.
Acknowledgements
The authors would also like to thank the editors, Roger Butlin and Julia Day, as well as two anonymous reviewers, for valuable comments and feedback on drafts of this work.
Funding
A.M. gratefully acknowledges funding from the Research Foundation – Flanders (FWO), Grant No. G070122N. C.S.’s research is supported by the Calleva Foundation and the Human Origins Research Fund.
CONFLICT OF INTEREST
The authors declare no conflict of interest.
