P(doom)

Consider this table of P(doom) values from the FoAK. There's a pretty wide range from 0% to 99.9%. But I want to argue that people like Marc Andreessen, Yann LeCun, Richard Sutton and the esteemable Mr Booch are certainly wrong in estimating that P(doom) is ~0%. FMs are already very good at science. So, if you were a postdoc or academic, an FM would be like having a bright PhD student, who doesn't have much taste, but knows a lot and has read everything. That's pretty helpful to a researcher.

The easiest cause of doom is a bioweapon, a weaponised virus or bacterium. Now, of course, FMs have guardrails built into them so that if you ask a system to help you build a bioweapon, it won't. But the point is that the guardrails have to built in. They are not their in the raw model. Training requires a lot of compute, but inference a lot less. So, someone could steal the weights of a raw model from a frontier lab and run the unguardrailed model on commodity hardware. Of course, if you work at a government bioweapons lab, you are going to want your own unguardrailed system. So, systems that are good at helping design bioweapons are going to be out there somewhere and that somewhere is going to include rogue states of various, and - who knows? - bad actors of other kinds (amoksters, criminal gangs, terrorist groups). There are hobbyist biohackers and you can buy a complete home genetic engineering kit for $1664. So, we don't have to believe in a The Machines Hate Us! scenario. We just have to believe that it is going to easier to cook up something nasty in the years and decades to come.

This doesn't mean there aren't necessarily lots of other ways the machines might kill us or that the machine themselves might kill us with bioweapons, it's just that putting powerful unguardrailed systems in the hands of unreliable people (and unreliable institutions). Hence, why I have a very P(doom) of 80-99%. And this leaves aside the absolute impossibility of creating provably aligned A(G|S)I. Because, although it is easier, to look inside the mind of an A(G|S)I, and you can try and interpret and control what it is thinking and doing, you are never going to be absolutely certain.And systems that can think more broadly, deeply and quickly than you are going to an unpredictable factor thrown into our volatile geopolitical mix. I am starting to sound Yudkowsky now. The next thing you know I will be calling for drone strikes on data centres. We just have to hope that the defensive systems can keep up with the offensive ones.