P(doom)

New word of the day: P(doom) is the probability of existentially catastrophic outcomes (so-called “doomsday scenarios”) as a result of artificial intelligence

I leaned this word from a very interesting deep dive podcast by NPR Explains… titled AI and the Probability of Doom. Take a minute to listen to it and then come back and let’s discus P(doom)!

So for a catastrophic outcome to be even possible, AI would need to reach Artificial general intelligence (AGI) which is when it reaches or passes human intelligence. The problem I see with current AI that everyday users are in touch with (ChatGPT, CoPilot, etcetera) is that these are large language models trained by sucking up all of the information available on the internet that they could find, and then, as I understand how it works, predicting the next letter that should come as it generates each word of its response. How this makes the leap to actually “thinking” and coming up with a novel idea seems incompatible to me.

I have often joked that we hit our plateau in 2022 when ChatGPT went generally available. After that the “free” sources of knowledge that have been used to train the LMMs have begun to dry up:

  • Stack Exchange (where LLM’s learned how to code): use has plummeted and altruistic sharing of coding knowledge has vanished with it.
  • Books: would appear that all sorts of copy righted material were sucked up and this will be bouncing around in courts for years. What is clear is that companies now have to be more careful about their learning material.
  • Reddit: Another previous source of knowledge about anything that is not as deep as before, this time not due to AI, but due to the need to somehow monetize the site the owners ruined it by removing free API access and causing a revolt by the moderators (the people who made Reddit what it was by investing their time for free ensuring their subreddits subject matter does not go off the rails).

So how AI makes the leap to real intelligence is not clear to me. If it does, I think it is clear that P(doom) is not zero, but how high it gets is not clear. I think some more education is in need, and what better way than watching historical documentaries that deal with P(doom) situations:

  • Tron

    • Date watched: 26/09/2025
    • Executive summary: The Master Control Program (MCP) is an artificial intelligence operating system that originated as a chess program.. The MCP monitors and controls ENCOM’s mainframe and has detected that a user program “Tron” is trying to monitor it and takes self preservation actions.
    • Lessons Learned: Wow. Amazing special effects from a movie from 1982! Oh, but limiting lessons to how to avoid or counter a rogue AGI: keep playing video games to ensure skills are up to needed level if ever sucked into the system like Flynn.

  • 2001: A Space Odyssey

    • Date watched: 28/09/2025
    • Executive summary: HAL 9000 is the onboard computer of the Discovery One, a spacecraft bound for Jupiter. When it discovers two crew members are going to disable it due to strange behavior, it takes self preservation actions killing multiple humans, before Dave is able to shut HAL 9000 down.
    • Lessons Learned: My gosh amazing movie, but really slow. Fell asleep twice and had to fast forward the pycodelic part at the end. Oh but limiting to lessons learned about p-doom: yes a computer that can think and thinks you want to turn it off, could be a dangerous situation.

  • The Terminator

    • Being downloaded now… will update after I have watched

And, of course I have asked ChatGPT if i missed any and he gave me a long list to consider after I have seen the those:

  • Classics / Foundations
    • Colossus: The Forbin Project (1970) – Supercomputer takes over the world “for our own good.”
      • Date watched: 04/10/2025
      • Executive summary: Colossus, a defense super computer, is brought online to take the human out nuclear defense decisions and quickly detects that there is another system like it and begins making demands with consequences of destruction of cities if its orders are not followed.
      • Lessons Learned: Wow super 70’s decor and dress! But focusing on p-doom lessons: If you give an AI system an objective you may not always like the way it meets its objective.
    • Blade Runner (1982) & Blade Runner 2049 (2017) – More about AI consciousness and control, but they hit the existential themes.
    • The Matrix (1999) – Post-doom scenario where AGI already won.
    • I, Robot (2004) – VIKI AI reinterprets “protect humanity” in a dangerous way.
  • Modern / Darker Takes
    • Ex Machina (2014) – Intimate, psychological look at an AGI breaking free.
    • Her (2013) – Less doom, more transcendence, but still raises the “what happens when AI leaves us behind?” question.
    • Transcendence (2014) – Uploading human consciousness leads to near-omnipotent AGI.
    • Eagle Eye (2008) – Military AI takes control of infrastructure.
  • Series & Anthology
    • Black Mirror (several episodes: “Metalhead,” “Hated in the Nation,” “White Christmas,” “Be Right Back”).
    • Westworld (the HBO series) – AGI rebellion and evolution themes.
    • Person of Interest – TV show about competing AGIs manipulating humanity.
  • Animated / Niche but Relevant
    • Ghost in the Shell (1995 anime, plus later films/series) – Strong themes of AI, identity, and autonomy.
    • Paprika (2006) & Serial Experiments Lain (1998) – Surreal explorations of networked intelligences.
    • Neon Genesis Evangelion (1995) – Not AGI per se, but very aligned with apocalyptic tech themes.

Looks like I have a lot of material to review on during rainy days this fall and winter!

Thanks for reading and feel free to give feedback or comments via email (andrew@jupiterstation.net).