The Precipice

These 2020 notes discuss Toby Ord’s estimates and my reactions at the time.

The Precipice by Toby Ord is a great birdseye view of the biggest challenges that we humans face in the future. Challenges that have the possibility of extinguishing our potential, threats that may make us go extinct. As could be predicted, these threats are currently mostly man-made.

I found this review by Scott Alexander very good. Do read it if you want to get a good overview of the whole book.

Also see this review by Theron Pummer on Notre Dame Philosophical Reviews.

The biggest things that stood out to me were:

The book puts the risk from AI at one in ten over the next 100 years, making it one of the largest risks in Ord’s assessment. I can see the case for that figure, though I also see arguments that we might create benevolent AI, or at least systems with ‘good’ goals that we can manage. See Human Compatible for more about AI.

Engineered pandemics felt especially relevant when I wrote this in 2020. Ord gives them a one-in-30 risk. Killing everyone seems difficult to me, but an event that killed 99% of people, or even 70%, might break down society. I wondered how hard the necessary knowledge would be to obtain, whether it would be used only once, and whether we could prevent or cure such a pandemic. My knowledge is thinnest here.

Ord assigns another one-in-30 risk to unforeseen anthropogenic threats. I appreciate that he accounts for these ‘unknown unknowns’ rather than restricting his analysis to named scenarios.

In the book’s estimate, humanity has about a five-in-six chance of making it through the next 100 years. Reducing the remaining one-in-six risk could be enormously valuable, and I felt we were not doing enough at the time. The future potential he describes includes exploring the galaxy, making more art and discovering more about nature. His final chapter made that brighter future feel worth protecting.