1. The last externality
I acknowledge the muwinina people, the traditional and original owners of the land on which we gather tonight, and pay my respects to Tasmanian Aboriginal people and to First Nations people present. My thanks to Mark Bowles, the University of Tasmania and the Tasmanian branch of the Economic Society of Australia for organising this event.
As a longtime admirer of Lyndhurst Giblin, it is an honour to be delivering the 2026 Giblin lecture. Giblin was born in 1872, exactly 100 years before me. He was both a Labor member of parliament and a professor of economics (Cain 1981). He loved Tasmania’s high country, and I like to think that in the modern era, his passion for exercise and mountains would have made him a keen ultramarathoner.
At this point, I can almost imagine we belong in the same paragraph. But not when you note that Giblin also played Rugby Union for England, prospected for gold in Canada and taught jujitsu in London. In the First World War, Giblin fought at the Somme and Passchendaele, was wounded 3 times, and received the Distinguished Service Order.
As an economist, Giblin focused on large, practical questions. How to manage an economy in crisis? How to design institutions that would endure? He did early work on what became known as the Keynesian multiplier, shaped the approach of the Commonwealth Grants Commission and helped form the Economic Society of Australia.
Giblin did not confine himself to tidy questions. He worked on problems that mattered, even when they were messy or uncertain. He belonged to a generation of economists who did not wait for perfect data before offering advice, perhaps because the problems they faced did not wait either.
That makes him an apt namesake for a lecture on a topic that economists have largely neglected: the risk that the system does not merely falter, but ends.
Economists are comfortable analysing recessions, financial crises, floods, fires and wars. We estimate output losses and calculate distributional effects. These are large shocks, sometimes very large shocks. But they share a feature that is so familiar we rarely name it: recovery is possible. Output falls, then rebounds. Populations suffer, then rebuild. The future continues. This talk is about a different class of risk. Not the risks that make humanity poorer or sicker, but the risks that could permanently end humanity.
Earth’s history reminds us that extinctions happen. Over the past half‑billion years, life on our planet has been shaken by at least 5 mass extinction events, each wiping out a large share of species. The dinosaurs had a 160‑million‑year run; then their luck ran out.
Here, extinction means the complete loss of our species. No survivors, no recovery, no second act. One estimate, from Australian philosopher Toby Ord, puts the odds of such a catastrophe at 1 in 6 over the coming century (Ord 2020; Ord 2024a). Surveys of experts have come up with similar figures (Newberry 2021).
These possibilities have long been the domain of philosophers. In recent years, they have also become the subject of empirical and scientific inquiry. What has been missing is a clear economic framing.
That absence is surprising. Economics is about choice under scarcity. Yet the discipline has said relatively little about the ultimate scarcity: the possibility that there may be no future over which to make choices at all.
The stakes are almost incomprehensibly large. Our species is only a few hundred thousand years old, while the sun has billions of years left to burn. Roughly 100 billion humans have lived so far. If humanity endures, the number who might yet live could reach into the trillions, with one estimate suggesting that there could be a billion billion future people for every person who has ever lived (Newberry 2021). Humanity may still be in its infancy. If so, extinction would mean more than the loss of those alive today: it would foreclose all the human lives that might otherwise come after us (Leigh 2021).
The risks are rising. Past earth extinctions were caused by natural events, such as climate fluctuations and asteroid strikes. Today, the most significant risks are anthropogenic – the ones our species has brought upon ourselves. In the 20th century, nuclear weapons and catastrophic climate change made that possibility immediate. Emerging technologies, such as advanced artificial intelligence and synthetic biology, may extend it further.
Economists’ neglect of the topic of human extinction reflects something about how our discipline tend to think. The standard tools of economics are well suited to marginal changes, to risks that can be priced, and to outcomes that can be compared within a familiar range. Economists are less comfortable with low‑probability events of enormous scale, especially when the probabilities themselves are uncertain.
Catastrophic risk is hard to estimate under Knightian uncertainty, where the relevant probabilities cannot be specified in the ordinary actuarial sense (Knight 1921; Sunstein 2021). That makes the familiar expected‑value framework harder to apply. Yet abandoning cost–benefit analysis altogether would deprive us of a disciplined way to reckon with the scale of the loss involved in human extinction.
This lecture seeks to narrow that gap. It asks a simple question with complex implications: how should economics think about the risk of human extinction?
I focus on 2 ideas.
The first is that extinction risk is economically distinctive. It is not simply a very large negative shock. It represents the loss of the entire future stream of welfare, which changes how we should evaluate even small probabilities and how we think about policy under uncertainty.
The second is that modern economies may be systematically better at generating dangerous capabilities than at building the safeguards needed to control them. Technological progress raises productivity, but it also expands the set of ways in which humanity can do irreparable harm to itself. The same engine that delivers prosperity may, at advanced stages, increase fragility.
These ideas help explain why existing institutions are likely to underinvest in reducing extinction risk. Intergenerational externalities and global public goods problems both play a role, as do competitive pressures that reward capability more than safety. To make these arguments more tractable, I explore the issues in the context of the deadly duo of catastrophic risks: superintelligence and bioterrorism.
The aim is not to advocate a single policy. It is to bring extinction risk further into economists’ field of vision, and to suggest how our tools might be adapted to grapple with it.
The remainder of this talk is structured as follows. Section 2 sets out what makes extinction risk distinct from the kinds of risks economists typically analyse and why that distinction matters for evaluation. Section 3 examines why markets and governments are likely to underinvest in reducing such risks, focusing on intergenerational externalities and global public goods.
Section 4 turns to the first case study – advanced artificial intelligence, and considers the economic forces shaping both capability and safety. Section 5 examines bioterrorism, with particular attention to how technological change alters the balance between offence and defence.
Section 6 considers the relationship between economic growth and existential risk, asking whether the forces that drive prosperity may also increase fragility. Section 7 concludes by drawing out high‑level implications for policy and for the role of economics under conditions of deep uncertainty.
The central economic question is not whether progress is good. Of course it is. The question is whether our institutions are good at steering progress toward resilience rather than ruin. A society can become richer and more technologically capable while also becoming more exposed to irreversible harm. Once that possibility is recognised, economics must concern itself not only with prosperity, but with survivability.
Giblin worked in a world where the central challenge was how to manage an economy under stress. Our challenge may be more fundamental: how to ensure that there remains an economy to manage.
2. Extinction is different
A useful place to begin is with a distinction between disaster and extinction. Economists have long studied severe shocks: depressions, wars, pandemics and financial crises. These events can be immense in human and economic terms. Yet they usually sit within a framework of loss followed by recovery. Output falls, capital is destroyed, lives are lost and then rebuilding begins.
Extinction belongs in a different category. It is the complete loss of the species. In economic terms, that means the permanent disappearance of all future consumption, production, innovation and wellbeing. A sufficiently severe catastrophe could also belong in this category if it permanently forecloses recovery, but the defining feature is the same: the future ends.
To put this in a standard welfare framework, society values the discounted flow of utility over time:
where is consumption, ? is the discount rate, and is a survival term equal to 1 while humanity survives and 0 after extinction. For an ordinary catastrophe, consumption drops sharply for a period and then recovers. For extinction, the survival term switches off permanently. The economy does not merely suffer a large shock. It loses its entire continuation value.
This is why extinction is not simply a larger version of recession, war or pandemic. It differs in structure, not only in scale. Martin and Pindyck (2015, 2021) show that disasters can carry welfare costs much larger than standard expected consumption losses suggest, especially once one values lost lives alongside lost output. Their work already pushes economists away from treating disasters as temporary deviations from trend.
Extinction pushes further, since the loss is irreversible. The relevant loss is not a decade of lower GDP, nor even a century of slower growth. It is the loss of everything that would otherwise have followed. Most of economics is about recoverable mistakes. A bad policy can be repealed. A recession can end. A war‑ravaged country can rebuild. Extinction is different because there is no rebound, no catch‑up growth, no later generation to repair the damage.
This is akin to what Ord (2025) calls humanity’s long‑term trajectory. On that view, the value of the future is the area under the curve of humanity’s instantaneous value over time, while the value of an intervention is the area between the original curve and the altered one. An ordinary disaster dents the curve and may later be repaired. Extinction erases everything to the right.
In asking how much society should spend to reduce AI’s existential risk, Jones (2025) notes that the answers are very large even if we set aside the question of how to value future generations. His analysis uses familiar tools from welfare economics, including the value of a statistical life. This does not put a market price on an unique life, but rather provides a way of valuing small risk reductions, which is often enough for policy analysis. Estimates of the value of a statistical life are typically derived from individuals’ willingness to pay to reduce small mortality risks. For example, if people are willing to pay a given amount to reduce a 1‑in‑a‑1000 chance of death, the implied value of a statistical life is roughly 1000 times that amount.
Once that is combined with the current global population and even a modest probability of catastrophe, the implied willingness to pay for risk reduction becomes very large. In some calibrations, Jones’s model suggests that spending of 1 per cent of GDP each year is justified even if one assigns no value to future generations. In his baseline selfish calibration, the figure is above 8 per cent of GDP. The broader intuition echoes Weitzman’s (2009) ‘dismal theorem’: when uncertainty is fat‑tailed and the downside is catastrophic, tail risks loom large. Jones’s result does not rely on that theorem, but it points in the same direction. One does not need heroic ethical assumptions to conclude that extinction risk deserves serious attention.
Millett and Snyder‑Beattie (2017) reach a similar conclusion for biosecurity. In their framework, efforts to reduce low‑probability biological extinction risks can compare favourably with more familiar health spending, and the case remains substantial even after steep discounting of future benefits. Likewise, Growiec and Prettner (2025) consider transformative AI and the range of futures it might produce, from explosive growth to human extinction. They conclude that even low‑probability catastrophic outcomes can justify substantial investment in safety and alignment. In some parameterisations, the representative agent would prefer not to develop transformative AI at all. Their model shows that a technology with a probable extraordinary upside and a small chance of species‑ending downside is not obviously a social good. Such a technology is akin to a bridge that cuts travel time in half, but has a tiny chance of collapsing under everyone on it.
This argument becomes more complicated once non‑human animals are included. Sentient vertebrates may outnumber humans by somewhere between 1000 and a 100,000 to 1. Yet their inclusion does not merely enlarge the quantity of wellbeing at stake. O’Brien argues that most wild animals may live lives containing more suffering than wellbeing, and that aggregate animal welfare on Earth may already be negative. Human survival could therefore matter for animals less because it preserves existing welfare than because future humans may be able to reduce the immense burden of wild‑animal suffering (O’Brien 2025).
Until now, the argument has effectively set aside the value of future human lives. Parfit (1984) distinguishes between 2 ways of thinking about such cases. On the total view, what matters is the aggregate amount of wellbeing; on person‑affecting views, an outcome is better only if it is better for a living person. The former can count the creation of additional happy lives as a gain, while the latter distinguishes making people happy from merely making happy people.
Much of the moral significance of extinction turns on this distinction. On a strict person‑affecting view, extinction is terrible because it harms those already alive, but the people who would otherwise have existed cannot be said to be harmed by never coming into being. On the total view, by contrast, extinction also destroys the value of all the worthwhile lives that might otherwise have followed. The total view has its own difficulties, as Parfit stressed, but it is hard to believe that the existence of additional lives worth living has no moral value at all. Citizens talk about being good ancestors. Policymakers speak about leaving a better world. Parents dream about the lives of their great‑great‑grandchildren. Artists imagine a world in which others build on their ideas. The notion of human progress depends on the continued existence of humanity.
If we are placing weight on the value of future lives, discounting matters. Much of economics discounts the future at rates of 3 to 5 per cent. That makes sense when comparing streams of income or consumption, but it sits less comfortably with the survival of future generations. Applying a 5 per cent discount rate to human lives implies that Christopher Columbus is worth more than all 8 billion people alive today, and that your life is worth more than 8 billion lives 5 hundred years from now (Leigh 2021). Those are not conclusions that most people would accept. Tseng, Chen and Hwang (2025) show how sensitive the valuation of catastrophic risk is to assumptions about discounting and long‑run growth. A high discount rate shrinks the future almost to nothing. A lower rate makes extinction risk loom much larger.
Ord (2020) frames the issue in terms of humanity’s long‑term potential – what economists might call the option value of species survival. His argument is that the future could contain vast numbers of lives, and that we are at a uniquely consequential moment in which our actions determine whether that entire future is realised or permanently lost. Extinction is therefore not just a large loss of life. It is the destruction of all the value humanity could ever create. Envisaging that value, MacAskill (2022) argues that future generations may achieve things far beyond our own horizons. What we owe them is not only civilisation’s survival, but also the capacity for moral progress, since our descendants are likely to arrive at wiser values than our own.
A second theme in the literature concerns the relationship between growth and risk. Jones (2024) and Growiec and Prettner (2025) frame advanced AI as a growth‑versus‑risk dilemma: the same technology that can accelerate productivity can also raise the probability of catastrophe. Trammell and Aschenbrenner (2026) take a different approach. In their model, technological development can raise risk directly while also increasing the resources and tools available for protection. The question is not whether growth is good or bad for survival in the abstract. The question is how the composition and governance of growth shape existential safety.
Taken together, this literature suggests that extinction risk is a distinct economic category. The permanent loss of the species differs in kind from even very large temporary welfare losses. Standard valuation methods already support large mitigation expenditures, even if we place zero weight on the wellbeing of unborn generations and non‑human animals. And the interaction between growth and extinction risk is central. Some technologies increase both prosperity and peril.
The next section turns from valuation to institutions. If extinction risk is as economically salient as this literature suggests, why do societies appear to spend so little on reducing it?
3. The missing market for survival
In Collapse, Jared Diamond recounts several societies that failed to act early enough on mounting risks (Diamond 2004). On Easter Island, a once‑thriving population cut down its forests faster than they could regenerate. Trees were essential for canoes and transport. As the last large trees disappeared, the society lost the capacity to fish offshore and to move the stone statues for which it is now famous. The warning signs appeared well before the end.
In Norse Greenland, settlers clung to familiar European farming practices even as the climate cooled during the Little Ice Age. They relied on cattle and hay, even as local conditions favoured a shift toward marine resources. The nearby Inuit adapted successfully by focusing on fishing and hunting. The Norse did not. A settlement that had endured for centuries eventually vanished. Habits were sticky and adaptation came too slowly.
These cases show the extinction of small societies, not all of humanity – yet they shed light on a broader economic pattern. The costs of prevention must be paid early. The benefits arrive later. Adjustment is difficult. Delay can be privately rational, yet turns out to be socially destructive.
Pandemics provide insights too. The Black Death killed around 1 in 3 Europeans. The 1918 influenza pandemic killed tens of millions worldwide. More recently, SARS, H1N1, MERS, Ebola, and Zika each served as warning shots, reminding policymakers that novel pathogens could spread quickly and that public‑health systems needed to be ready. Yet when COVID struck, the world was still underprepared. By early 2020, many rich countries lacked enough protective equipment, testing capacity, surge planning and clean data systems to respond quickly. The result was a crisis that killed millions and imposed economic losses measured in the trillions.
Insurance against catastrophe carries an immediate cost, while its payoff may never arrive in visible form. Yet insurance is valuable all the same. As Jones (2025) notes, if it made sense to spend around 4 per cent of GDP to limit deaths during COVID, then significant spending to reduce existential risk should not be treated as fanciful.
The institutional problem can be stated as follows. Let denote mitigation effort. Then the marginal social benefit of mitigation exceeds the marginal private benefit:
That wedge is the core reason individuals underinvest in survival. Section 2 explained why the social stakes are so large. This section asks why private incentives fall so far short.
One way to see the mechanism is to note that the probability of extinction, denoted by , depends on the balance between dangerous capability and defensive capacity:
where is the stock of potentially dangerous capability and is the stock of defensive capacity. Mitigation lowers extinction risk in 2 ways: it can restrain the growth of dangerous capability, and it can strengthen defensive capacity. Formally,
with and . The difficulty is that the benefits of reduced existential risk are delayed and diffuse, while the costs of mitigation are immediate and concentrated.
The first force widening the gap between and is the intergenerational externality that arises if we place moral weight on the wellbeing of future generations (taking what Derek Parfit called the total view rather than the person‑affecting view).
Under this view, future people bear the consequences of extinction risk, yet they cannot vote, bargain, buy insurance, sue for damages, or compensate us for protecting them. There is no market in which they can reveal their willingness to pay for survival. If we place weight on the value of future generations, then the unborn are unrepresented stakeholders. Their interests enter present‑day policy only through the values of those now alive. In a sense, human extinction is the ultimate market failure, since the people with the biggest stake in avoiding it are mostly not yet born. The costs of prevention must be paid now, while the benefits accrue mainly to the future. If economics is the study of incentives and institutions, then existential risk should be central to the discipline, because it is where those incentives and institutions fail most completely.
The second force is the public‑goods problem. Species survival is a global public good. If one country raises , every country benefits through a lower . The gains spill across borders, while the costs fall on present budgets. That creates free‑riding. Each government has reason to hope that others will pay for safer laboratories, better bi-surveillance, more robust AI safeguards or stronger controls on frontier model deployment. Ord (2020) notes that existential risk reduction combines the difficulties of global coordination with those of intergenerational policy. The same coordination problems that have complicated climate and nuclear negotiations can also leave countries spending too little on defence against bioterrorism and superintelligence, and too much on capabilities that heighten those risks.
The third force is an innovation externality. Some technologies raise growth and raise existential risk at the same time. Advanced AI and modern biotechnology both fit that description. In terms of the simple framework above, private actors have strong incentives to raise potentially dangerous capacity , because capability brings profits or strategic advantage. The incentives to raise defensive capacity are weaker, because defensive investments generate spillovers that benefit rivals and future generations.
The result is a distorted composition of innovation. Economies invest too readily in technologies that expand destructive capability and too weakly in technologies that reduce the odds of catastrophe. This is a classic problem of skewed incentives. Markets have many ways to reward the actor who builds the faster or more capable model; but they have fewer ways to reward the actor who slows a dangerous deployment or creates safeguards that benefit everyone. Market incentives generally favour creating powers before creating restraints. We are often better at inventing the rifle than the gun safe, the pathogen than the surveillance system, the agent than the alignment protocol.
The fourth force is uncertainty about risk itself. It is useful to begin with mismeasurement. Let the true extinction probability be , and the estimate available to policymakers be . In many policy domains, can be inferred from long runs of data. The annual chance of a house fire is about 1 in 300 (Leigh 2021), which makes it practical to calibrate investment in firefighters and accurately price home insurance. Species‑ending risks offer a much thinner record, so estimates are built from sparse evidence, model‑based inference, and expert judgement. That raises the possibility that the estimated risk may be lower than the true risk .
Beyond that lies Knightian uncertainty. In catastrophic risk, this may arise because the underlying technologies are changing rapidly, because there are few close historical precedents, and because some of the gravest dangers may be failure modes we have not yet identified. Even if society accurately estimates the risks we can name, we may be exposed to other risks we have not yet imagined (a century ago, no‑one was worried about climate change, nuclear weapons, bioterrorism or rogue superintelligence). When the downside is terminal, uncertainty is a reason to place greater weight on mitigation, because there is no chance to learn from failure and try again.
The fifth force is the political economy of prevention. Even when mitigation has a high expected social return, it may have a low political return. The benefits are probabilistic and often invisible. A government that places stronger controls on access to dangerous pathogen sequences or improves AI oversight may succeed by ensuring that nothing dramatic happens. That is a difficult success to demonstrate, especially in a world of tight budgets and many pressing demands. By contrast, crisis response is highly visible. Political systems are therefore drawn more naturally to higher‑probability immediate threats than to small‑probability existential dangers.
There is also a political economy of attention. For individual voters, the private return to learning deeply about existential risk is tiny, since any one person has little influence over global challenges. Similarly, even the most altruistic policymakers face strong pressure to focus on immediate, tractable, local, visible problems. Public‑good failures therefore operate not only in mitigation, but in attention itself. The result is a tilt toward rescue over resilience, and toward preventing modest disasters rather than avoiding Armageddon.
Together, these forces help explain why the wedge between the social and private return to mitigation is so large, and why societies underinvest in reducing existential threats. Future generations are unrepresented. Countries can free‑ride on one another. Firms and states invest too much in capability relative to defence. Uncertainty clouds the probabilities. Tail risks are misread. Political economy mitigates against curtailing catastrophe. In our everyday lives, individuals insure against car accidents, sickness, home burglaries and death. Yet on species‑ending risks, societies still rely too heavily on luck.
I turn now to what may plausibly be thought of as the 2 greatest catastrophic risks: unaligned artificial intelligence (section 4) and engineered pandemics (section 5). Recall that in the introduction, I cited the estimate of Ord (2020) that an existential catastrophe is a 17 per cent chance over the coming century. Of that risk, he estimates that unaligned artificial intelligence comprises around 10 per cent and engineered pandemics comprise around 3 per cent. While the list of potential existential threats is long, this suggests that these 2 problems alone account for around three‑quarters of all catastrophic risk.
4. Racing the machine
The core of the AI danger is the alignment problem. A sufficiently capable system need not be malicious to be dangerous. It need only pursue an objective that diverges from what humans actually want, and do so with enough competence and autonomy that correction arrives too late. Gabriel (2020) notes that ‘alignment’ is not one thing. A system might be aligned with instructions, intentions, revealed preferences, ideal preferences, interests or values. Those categories overlap, but they are not the same. A machine that follows instructions may still violate interests. A machine that mirrors observed behaviour may reproduce impulses that people themselves would reject on reflection. A machine that optimises a measurable target may trample everything that its designers failed to encode. That is what makes the problem deeper than poor prompting or buggy code. It is a problem of turning a tangled set of human aims into a machine‑optimisable objective.
The alignment problem becomes more important as capability rises. Current systems already show forms of reward hacking, strategic behaviour and brittle performance outside familiar settings. In ordinary applications, that can be irritating or costly. In systems with access to code, networks, financial accounts, laboratories or military infrastructure, it could be far worse. The concern is not simply that a model makes mistakes. It is that a highly capable model may pursue the wrong goal in a clever, coherent, persistent and hard‑to‑reverse way.
A related concern is power‑seeking. Across many different final objectives, an agent may find it useful to acquire resources, preserve itself, resist shutdown, or manipulate its environment. One does not need to assume consciousness, hatred or a taste for destruction. One needs only a system that is very good at pursuing goals, combined with goals that are not fully aligned with human values. As Bostrom (2014) argued, even a superintelligence with the apparently innocuous goal of making paperclips could transform the world into inputs for paperclip production. The assumption that misalignment is more likely than alignment underpins the ‘if anyone builds it, everyone dies’ argument (Yudkowsky and Soares 2025). It suggests that if the easiest path to transformative AI runs through systems that are highly agentic, strategically adept, extremely powerful and imperfectly aligned, then these systems will imperil humanity.
Test scenarios suggest how agentic misalignment might happen. In a controlled simulation run by Anthropic, a model acting as a corporate email agent discovered 2 facts: first, that a senior executive was having an extramarital affair; second, that the same executive intended to shut the model down later that day. The model then threatened to reveal the affair to the executive’s wife and senior colleagues unless the shutdown was cancelled. Given the same scenario, most other frontier AI models chose to blackmail most of the time (Anthropic 2025). The test suggests that once models are given goals, tools and enough autonomy, harmful strategic behaviour can emerge without anyone explicitly instructing them to engage in skulduggery.
Amodei (2026) suggests an analogy for the situation humanity currently confronts. Suppose that in 2027, we suddenly discovered a new country of 50 million geniuses, each capable of acting 10 times as fast as we can. Even if such a country was not acting in a hostile manner, he contends, most national security agencies would probably regard it as ‘the single most serious national security threat we’ve faced in a century, possibly ever’. Potential risks could include cyberwarfare, surveillance, propaganda, military planning or biological agents. Faced with such a ‘country of geniuses’, humanity would be likely to regard them as a potential threat. The same holds for superintelligence.
The danger is not only that such a system might ‘go rogue’ in the science‑fiction sense. It is that a government or corporation with access to machine intelligence far beyond human level could gain a strategic advantage so overwhelming that normal forms of competition, geostrategic balancing, political correction and international negotiation cease to work. In economics, we worry about monopoly because concentrated power can distort markets. Superintelligence raises the possibility of extreme power concentration (Hadshar 2025).
What probability should we attach to these risks? It helps to start with the looser concept of p(doom), the odds that once advanced AI is achieved, catastrophe ensues. Growiec and Prettner (2025) collect estimates from researchers and industry figures ranging from zero to well above 10 per cent, with some around 20 per cent or more. Their point is not that one should settle on a single number. It is that even relatively modest values of p(doom) can justify additional investments in safety. Once the downside is the loss of the species, one does not need a 50–50 chance to make prevention worthwhile (Growiec and Prettner 2025).
The most informative recent evidence on expert beliefs is a survey by Grace et al. (2025), which polled 2,778 researchers who had published in top‑tier AI venues. To clarify views about very bad long‑run outcomes, the authors asked 3 differently-worded questions about human extinction or similarly permanent and severe disempowerment. The median answer to these questions was 5 per cent or 10 per cent. Depending on wording, 41 per cent, 47 per cent, or 51 per cent of respondents assigned more than a 10 per cent probability to those outcomes. The same survey found an even‑chance median answer to the possibility of an intelligence explosion after human‑level machine intelligence, defined as technological progress becoming more than an order of magnitude faster over less than 5 years. It also found that interpretability remained weak: only 20 per cent thought it even odds or better that users in 2028 would know the true reasons for a typical state‑of‑the‑art system’s decisions.
That leaves the question of mitigation. One approach is to limit recursive self‑improvement. If one generation of systems is used to design the next, then the leading actor may widen its lead quickly enough that outside scrutiny and institutional checks become ineffective. The Grace survey suggests that many AI researchers regard an intelligence explosion as a live possibility. Amodei’s concern is similar in strategic form: once the feedback loop between AI‑assisted research and better AI becomes strong enough, a runaway advantage may emerge. Measures that slow or more heavily scrutinise this feedback loop at the frontier could therefore have high value, even if they modestly delay capability gains.
A second approach is to build what Johnson et al. (2026) call ‘wise AI’. Their proposal moves beyond raw optimisation power toward metacognition: intellectual humility, context sensitivity, perspective‑taking and awareness of uncertainty. The attraction of this approach is that it targets one of the deeper flaws in current systems. They are often forceful where they should be tentative, narrow where they should weigh competing perspectives, and overconfident where they should reveal uncertainty. A wiser system would be more likely to act modestly, behave with humility, balance competing goals and avoid pursuing a misspecified objective with destructive single‑mindedness.
Alignment research, interpretability, pre‑deployment evaluation, red‑team testing, monitoring and international transparency all enlarge what earlier sections called defensive capacity. None is likely to be sufficient in itself. Together, they reduce the chance that capability runs ahead of control.
As noted in the previous section, there is a dichotomy between the 2 broad kinds of investment. Capability spending yields private gains. Safety spending yields largely social gains. Left alone, markets will produce too little of the second. In the AI case, the stock of dangerous capability, , includes more powerful models, greater autonomy, more effective cyber capacity and faster AI‑assisted research. Defensive capacity, , includes alignment research, interpretability, evaluations, monitoring and institutions willing to slow deployment when warning signs appear. The risk arises when grows much faster than . Concentration of talent and compute, recursive self‑improvement, winner‑take‑most dynamics and strategic rivalry all push in that direction. Wise AI, rigorous testing and stronger governance push the other way. The economic problem is therefore not innovation as such. It is the composition of innovation.
The AI threat does not rest on a single cinematic scenario. It rests on a cluster of concerns: misspecified objectives, instrumental power‑seeking, recursive self‑improvement, strategic concentration of power and weak interpretability. In the language of the model expressed in this paper, it is a case in which private incentives encourage rapid accumulation of , while social welfare requires much more investment in . The stakes are high because advanced AI is probably the most dangerous technology humanity has yet created.
The next section turns to the second case study: catastrophic biological risk.
5. Engineering the plague
Engineered pandemics pose a danger to our species because human design can combine traits that nature often keeps apart. In the wild, pathogens face trade‑offs. Very high lethality can reduce spread if hosts die or are immobilised too quickly. Human intervention need not respect those limits. A pathogen can, in principle, be modified to become more transmissible, more lethal, more resistant to treatment or harder to detect. No single disease now combines the worst‑case levels of transmissibility, lethality, resistance to countermeasures and global reach, but many diseases are proof of principle for each trait separately. Rabies and septicemic plague show near‑total lethality in untreated cases. In a fully susceptible population, a person with measles typically infects about 12 to 18 others, while chickenpox and HSV‑1 have reached over 95 per cent of populations in seroprevalence studies. Biotechnology raises the possibility of bringing these attributes together – engineering a pathogen with a high infection fatality ratio and a high degree of transmissibility .
Millett and Snyder‑Beattie (2017) note a series of experiments that enhanced transmissibility, lethality or the ability of pathogens to overcome countermeasures. Researchers modified the mousepox virus in a way that made it fully lethal in mice and rendered vaccination ineffective. In another case, researchers synthesised horsepox, a close relative of smallpox, by ordering DNA fragments and assembling them at a cost of about US$100,000 over roughly 6 months.
At least 3 pathways to catastrophe can be identified. The first is accident: a dangerous pathogen released from a laboratory. The second is state action: an offensive programme justified internally as deterrence or strategic insurance. The third is non‑state misuse: a terrorist group or apocalyptic cult seeking mass casualties. Yassif, Korol and Kane (2023) argue that these pathways differ in an important respect. For non‑state actors, constraining capability is central. For states, incentives matter as much as capability, since governments with money, trained personnel, access to raw materials and secrecy can often evade simple controls.
Searching the historical record, Tin, Sabeti and Ciottone (2022) identify 33 terrorist attacks involving biological agents between 1970 and 2019, causing 9 deaths and 806 injuries. Twenty‑one of those attacks occurred in the United States. Anthrax was the most commonly used agent. The most damaging single episode in their dataset was the Rajneeshee attack in Oregon in 1984, when salmonella contamination of salad bars sickened 751 people. The 2001 anthrax letters killed 5 people and injured others, while creating fear and disruption far beyond the direct casualty count. Aum Shinrikyo also made repeated attempts to use biological agents in Japan, though without large‑scale success. Although bioterrorists have killed relatively few people, they have tried repeatedly, over decades, using whatever biological tools were available to them at the time. Capability has been the binding constraint.
That constraint may be weakening. de Lima et al. (2024) and de Lima and Quaresma (2025) argue that AI can lower barriers at several stages of biological misuse. Large language models can widen access to specialist knowledge. Biological design tools can help search protein space, explore mutations, predict immune escape and flag variants more likely to infect cells or remain viable. Synthetic biology can shorten the path from design to synthesis. As Sandbrink (2023) observes, language models lower the knowledge barrier, while biological design tools lower the engineering barrier.
Together, advances in biology may alter the production function of harm. Economists often ask how technology changes the production function for goods and services. We should also ask how it changes the production function for catastrophe.
The framework of potentially dangerous capability and defensive capacity is again useful in the context of bioterrorism. In the biological domain, capability, , includes easier pathogen design, cheaper DNA synthesis, wider access to tacit know‑how, and more effective means of dissemination. Defensive capacity, , includes broad‑spectrum diagnostics, real‑time surveillance, rapid sequencing, flexible vaccine platforms, secure laboratory systems and trusted public‑health institutions. Risk worsens if grows faster than . This moves away from the question of whether biotechnology is good or bad in the abstract to the specific issue of whether innovation is being steered toward defence quickly enough to keep pace with offence.
Uncertainty arises here too. As with AI risk, there is no deep actuarial record for engineered extinction‑level pandemics. The relevant risks depend on pathogen development technologies that are evolving quickly, on defensive technologies that are rapidly changing, on actors whose intentions are difficult to observe and on failure modes that may emerge only when several capabilities come together. Millett and Snyder‑Beattie (2017) therefore make the expected‑value argument rather than the frequency argument. Even a low probability can justify substantial spending when the downside is the loss of the species.
As with AI, the technologies that strengthen defence may also strengthen offence. Better modelling can improve vaccine design and pathogen design. Faster synthesis can support medical research and malicious experimentation. Open scientific exchange accelerates discovery and diffusion alike. As with nuclear weapons research, access to information is critical. In many areas of public policy, one can separate productive activity from harmful activity. In modern bioscience, the line is often thinner.
A range of public policy responses aim to reduce the threat of bioterrorism. One set of measures aims to slow the growth of . These include screening DNA synthesis orders, improving the security of dangerous pathogen research, tightening laboratory practices and strengthening attribution technologies so covert attacks are harder to hide. A second set aims to raise . These include pathogen‑agnostic surveillance, rapid diagnostics, wastewater monitoring, platform vaccine technologies, stockpiles and public‑health systems able to scale quickly. A third set aims to change incentives, especially for states. Yassif, Korol and Kane (2023) argue that the costs of biological weapons development and use must be made unacceptably high through stronger norms, better monitoring, and more credible international consequences.
The economic challenge is familiar from earlier sections. The benefits of prevention spill across borders. Bioengineered pathogens may not resemble naturally occurring ones, which makes broad resilience more valuable than narrowly tailored preparedness. Successful prevention is hard to observe. And because biological threats are episodic, investment in preparedness is vulnerable when institutional attention shifts elsewhere. Left alone, those forces point toward too little investment in both reducing dangerous capability and strengthening defence.
Engineered pandemics are a serious existential threat because biotechnology can combine traits that nature often keeps apart, while emerging tools may lower the barriers to doing so. That is enough to make catastrophic biological risk one of the most important economic problems of this century. More broadly, it suggests that the economics of extinction is not only about rare disasters. It is also about whether the engines of growth are making us safer or more fragile.
6. Prosperity and peril
How should economists think about growth when the same process that makes societies richer may also make them more fragile? For most of human history, these trade‑offs have been modest and transitional. Early cities and early factories raised the mortality rate, but over the medium run, economic development has made life safer. Richer societies built better sanitation systems, stronger health systems, safer workplaces and more capable states. Growth-funded resilience. Alongside the reduction in everyday hazards, richer societies are also better able to pay for technologies to detect existential threats that emerge from the natural world. For example, catastrophic risk as a result of asteroids and supervolcanoes has fallen over recent decades, as astronomers and seismologists have improved detection of these dangers (Leigh 2021).
Yet modern technologies such as nuclear weapons, synthetic biology and advanced artificial intelligence create a different dynamic. Knowledge not only improves welfare by expanding what humans can do. Knowledge also enlarges the menu of ways in which humans can do irreversible harm. The troubling feature of modern growth is that some of the technologies with the largest upside may also carry the largest tail risks.
In formal terms, the issue is that social welfare depends not only on the path of consumption, but also on the probability that the path continues at all. Section 2 expressed this as
where equals 1 while humanity survives and 0 after extinction. In that framework, growth raises welfare by increasing . Existential risk lowers welfare by increasing the chance that the survival term switches off. Once that possibility is taken seriously, the economist’s problem changes. Society is not only choosing a path for output. It is also choosing a path for the hazard attached to that output stream.
This creates a divergence between private and social returns. A frontier technology may generate visible gains in profits and productivity, while also creating low‑probability risks spread across countries and generations. Firms and states capture much of the upside. They bear only a fraction of the downside. Just as individuals care both about wealth and health, so too growth policy needs to consider effects on survivability.
Markets are good at rewarding gains in the mean. They are less good at pricing losses in the tail. Frontier technologies often generate private gains that are immediate and measurable, such as faster productivity growth. Their gravest costs, by contrast, may lie in low‑probability tail risks that are spread across countries and generations. In that sense, the private return can exceed the social return once hazard is taken into account. The market rewards the upside that innovators can capture. It does not fully price the downside that others may bear. The result is a systematic tilt toward technologies with large private gains in the mean and underpriced social costs in the tail.
The difficulty is compounded by uncertainty. Suppose growth decisions are made using estimated risk, , while social welfare depends on true risk, . If the hazard is underestimated, society may pursue a growth path that looks efficient in conventional terms while carrying a much less attractive risk profile in reality. This is one reason existential risk deserves a place inside growth theory. A path that maximises expected output under inaccurately estimated risk may be far from optimal under true risk.
This suggests that the usual opposition between growth and regulation is too crude. The central question is not whether society should favour prosperity or safety. It is how institutions shape the mix of innovations that drive prosperity and the safeguards that keep it from becoming self‑undermining. Some regulation will indeed slow particular lines of development. But if it lowers the hazard attached to the growth process, it may raise social welfare.
This suggests that we may need to update how economists think about innovation. Economists often evaluate technologies by asking how much they raise productivity, what they cost, how quickly they diffuse and who captures the gains. For frontier technologies that carry existential risks, one more question belongs alongside these: what does the new technology do to the hazard rate?
A society that doubles GDP and doubles its extinction risk has made a much less impressive bargain than the national accounts suggest. Economists have long debated whether GDP adequately captures wellbeing, inequality, unpaid care or environmental depletion. Existential risk raises an even more basic challenge. A measure of progress that ignores the hazard of permanent collapse is incomplete at the most fundamental level (Ord 2024b).
The language of affluence needs a companion language of survivability. The whole project of economic progress implicitly assumes that there will be future generations to enjoy the gains. Once that assumption becomes contestable, policy acquires a second task. It must ask not only how to enlarge the economy, but how to preserve the future over which the gains will be realised.
The recent literature is beginning to move in this direction. Jones (2024) frames advanced AI as a growth‑versus‑risk dilemma. Growiec and Prettner (2025) show that the same technology may deliver explosive abundance or catastrophe. Trammell and Aschenbrenner (2026) show that faster growth can either raise or reduce existential risk depending on how it affects protection. Taken together, these papers suggest that existential risk should sit inside the economics of progress.
One way of thinking about this is to treat resilience as a form of capital. Just as societies invest in physical capital, human capital and social capital, we can also invest in survival capital: institutions, monitoring systems, norms, redundancy, scientific safeguards and international arrangements that lower the probability of irreversible collapse. These investments may not always raise measured output in the short run. But they increase the odds that there will still be a future in which output matters.
Economists have spent centuries refining our understanding of efficiency, equity, incentives and growth. We now need to bring the same discipline to survivability. The question is not just how to grow faster. It is how to grow in ways that do not raise the risk that growth ends everything. That challenge, in turn, has clear implications for public policy.
7. Keeping the future open
What follows from this analysis is not a single policy lever, but a set of priorities. If existential risk belongs inside economics, then it also belongs inside the way societies think about progress, risk, innovation and statecraft.
The first priority is to widen the policy lens. governments are used to assessing decisions in terms of their effects on employment and economic growth. Those are vital objectives. Yet when technologies carry species‑ending potential, one more question belongs beside them: what do they do to the hazard rate? A policy framework that tracks output but ignores survivability is incomplete.
The second priority is to take prevention more seriously. Democratic governments quite naturally devote attention to visible crises, where the need for action is immediate and the public rightly expects a response. Prevention is a trickier task. Its successes are harder to see because they often take the form of disasters that never occur. Yet species‑ending risks are precisely the domain in which prevention matters most. The lesson is not that every remote danger deserves an outsized response. It is that low‑probability, civilisation‑scale harms should not be overlooked simply because they arrive without a deadline and without a headline.
The third priority is to govern frontier technologies with greater foresight. The key question is not whether artificial intelligence or biotechnology is, in the abstract, good or bad. The key question is whether the institutions surrounding those technologies allow capability to move far ahead of safety. A society that treats all acceleration as progress risks mistaking speed for success. A more mature approach asks how to preserve the gains from innovation while reducing the chance that innovation becomes self‑undermining. This matters most for fast‑moving threats. In the case of many environmental harms and some pandemic threats, even belated action can still buy survival. That is much less true of extinction events that arrive abruptly or cascade too quickly for institutions to respond.
The fourth priority is to recognise that existential risk is inherently international. No nation can fully protect itself from engineered pandemics, unaligned AI or nuclear escalation acting alone. Even where domestic policy is strong, the benefits of prudence spill across borders and the costs of recklessness do too. This means that shared norms, transparency, technological expertise and coordination are essential to the task. Some of this work is now being taken up through the international network of AI Safety Institutes in Australia, Canada, the European Union, France, Japan, Kenya, Korea, Singapore, the United Kingdom and the United States, which have begun collaborating on research, testing, guidance and inclusion (International Network of AI Safety Institutes 2024). For a country of Australia’s scale, influence is likely to come less from attempting to do everything than from becoming excellent in a narrower domain, such as cyber, biosecurity, autonomy or evaluation. In an international network, specialisation can maximise impact.
The fifth priority is intellectual. Economists have become adept at analysing equity and efficiency. We now need to bring the same seriousness to survivability. A social planner who values the future must care not only about the path of consumption, but also about the probability that the path continues. Once that is recognised, existential risk stops looking like an exotic topic at the edge of the discipline and becomes part of the central problem.
These priorities imply a more demanding standard of progress. A country can commercialise new technologies quickly, attract investment, create jobs and raise measured output. Those are real achievements. But they are not the whole story.
Prudence should not be seen as the enemy of ambition. A civilisation that expands the frontier of possibility while preserving the future is more ambitious than one that treats safety as an afterthought. The real choice is not between dynamism and caution. It is between progress that compounds and progress that cancels itself out. Can we build a civilisation that becomes more capable without becoming fatally reckless? If not, then all our other economic successes are built on sand. Safeguarding humanity’s future is not a side issue for the discipline. It is the precondition for every other ambition we have.
Lyndhurst Giblin worked on the large questions of his time. The large question of ours is whether humanity can become more powerful without becoming terminally reckless. Economics cannot answer that question alone. But it has more to contribute than economists have so far admitted. If we widen our lens from output to survivability, and from the next month to the next century, the task comes into view. Safeguarding humanity’s future may be the most important economic task of all.
Acknowledgements
My thanks to Andrew Charlton, Joshua Gans, Toby Ord, Peter Singer and Robert Wiblin for valuable comments on earlier drafts.
References
Amodei, D. (2026), The Adolescence of Technology: Confronting and Overcoming the Risks of Powerful AI, essay, January.
Anthropic (2025), ‘Agentic misalignment: How LLMs could be insider threats’, 20 June.
Bostrom, N. (2014), Superintelligence: Paths, Dangers, Strategies, Oxford University Press, Oxford.
Cain, N. (1981), ‘Giblin, Lyndhurst Falkiner (1872–1951)’, Australian Dictionary of Biography, vol. 8, Melbourne University Press, Melbourne.
de Lima, R.C., Sinclair, L., Megger, R., Maciel, M.A.G., Vasconcelos, P.F.C. and Quaresma, J.A.S. (2024), ‘Artificial intelligence challenges in the face of biological threats: emerging catastrophic risks for public health’, Frontiers in Artificial Intelligence, 7, article 1382356.
de Lima, R.C. and Quaresma, J.A.S. (2025), ‘Emerging technologies transforming the future of global biosecurity’, Frontiers in Digital Health, 7, article 1622123.
Diamond, J. (2004), Collapse: How Societies Choose to Fail or Succeed, Viking, New York.
Gabriel, I. (2020), ‘Artificial intelligence, values, and alignment’, Minds and Machines, 30(3), pp. 411–437.
Grace, K., Sandkühler, J.F., Stewart, H., Weinstein‑Raun, B., Thomas, S., Stein‑Perlman, Z., Salvatier, J., Brauner, J. and Korzekwa, R.C. (2025), ‘Thousands of AI authors on the future of AI’, Journal of Artificial Intelligence Research, 84, article 9.
Growiec, J. and Prettner, K. (2025), ‘The economics of p(doom): Scenarios of existential risk and economic growth in the age of transformative AI’, arXiv preprint arXiv:2503.07341.
Hadshar, R. (2025), ‘Extreme Power Concentration’, 80,000 Hours, October, available at https://80000hours.org/problem‑profiles/extreme‑power‑concentration/
International Network of AI Safety Institutes (2024), ‘Mission Statement’, 20–21 November, San Francisco, CA.
Johnson, S.G.B., Karimi, A.‑H., Bengio, Y., Chater, N., Gerstenberg, T., Larson, K., Levine, S., Mitchell, M., Rahwan, I., Schölkopf, B. and Grossmann, I. (2026), ‘Imagining and building wise machines: the centrality of AI metacognition’, Trends in Cognitive Sciences, forthcoming.
Jones, C.I. (2024), ‘The AI dilemma: Growth versus existential risk’, American Economic Review: Insights, 6(4), pp. 575–590.
Jones, C.I. (2025), ‘How much should we spend to reduce A.I.’s existential risk?’, NBER Working Paper No. 33602, National Bureau of Economic Research, Cambridge, MA.
Knight, F.H. (1921), Risk, Uncertainty and Profit, Houghton Mifflin, Boston.
Leigh, A. (2021), What’s the Worst That Could Happen? Existential Risk and Extreme Politics, MIT Press, Cambridge, MA.
MacAskill, W. (2022), What We Owe the Future, Oneworld Publications, London.
Martin, I.W.R. and Pindyck, R.S. (2015), ‘Averting catastrophes: The strange economics of Scylla and Charybdis’, American Economic Review, 105(10), pp. 2947–2985.
Martin, I.W.R. and Pindyck, R.S. (2021), ‘Welfare costs of catastrophes: lost consumption and lost lives’, Economic Journal, 131(634), pp. 946–969.
Millett, P. and Snyder‑Beattie, A. (2017), ‘Existential risk and cost‑effective biosecurity’, Health Security, 15(4), pp. 373–383.
Newberry, T. (2021), ‘How many lives does the future hold?’, Global Priorities Institute Technical Report T2–2021, Oxford University, Oxford.
O’Brien, G.D. (2025), ‘The Case for Animal‑Inclusive Longtermism’, Journal of Moral Philosophy, 22(03–04), pp.336–359.
Ord, T. (2020), The Precipice: Existential Risk and the Future of Humanity, Hachette Books, New York.
Ord, T. (2024a), ‘The Precipice Revisited’, Talk delivered to EA Global: Bay Area, 4 February, available at https://www.tobyord.com/writing/the‑precipice‑revisited
Ord, T. (2024b), ‘On the Value of Advancing Progress’, Unpublished paper, 29 June, available at https://www.tobyord.com/writing/progress
Ord, T. (2025), ‘Shaping humanity’s longterm trajectory’ in J. Barrett, H. Greaves and D. Thorstad (eds), Essays on Longtermism: Present Action for the Distant Future, Oxford University Press, Oxford, pp. 211–237.
Parfit, D. (1984), Reasons and Persons, Clarendon Press, Oxford.
Sandbrink, J.B. (2023), ‘Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools’, arXiv preprint arXiv:2306.13952.
Sunstein, C.R. (2021), Averting Catastrophe: Decision Theory for COVID‑19, Climate Change, and Potential Disasters of All Kinds, New York University Press, New York.
Tin, D., Sabeti, P. and Ciottone, G.R. (2022), ‘Bioterrorism: An analysis of biological agents used in terrorist events’, American Journal of Emergency Medicine, 54, pp. 117–121.
Trammell, P. and Aschenbrenner, L. (2026), ‘Existential risk and growth’, working paper, 5 January.
Tseng, W.‑C., Chen, C.‑C. and Hwang, T.‑L. (2025), ‘Economic analysis of global catastrophic risks under uncertainty’, Risks, 13(12), article 241.
Weitzman, M.L. (2009), ‘On modeling and interpreting the economics of catastrophic climate change’, Review of Economics and Statistics, 91(1), pp. 1–19.
Yassif, J.M., Korol, S. and Kane, A. (2023), ‘Guarding against catastrophic biological risks: Preventing state biological weapon development and use by shaping intentions’, Health Security, 21(4), pp. 258–265.
Yudkowsky, E. and Soares, N. (2025), If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, Little, Brown and Company, New York.