Great article. I am someone who would be worried about AI with 300 IQ, but I think there are valid reasons to doubt we are anywhere close to that at the current trajectory.
Like, if I would be somehow unaware of AI Discourse about how it is possibly most important technology in history, carrying enormous existential risks, I’d see just that models had gone from slightly useful productivity enhancing tools in 2023 to somewhat more useful productivity enhancing tools in 2025.
That is good for the future of productivity growth and perhaps bad for employment prospects of some people, but to extrapolate from that they will be soon smarter than humans is not obvious. Of course in certain capabilities they far surpass humans, but computers have been able to do that for decades - in multiplication, chess etc. Yet, chess engine doesn’t really have IQ; does GPT 5.0 have one? And how high it is? I am genuinely not sure.
Now, imho it would be overconfident to completely dismiss The Discourse about how machines with 300 IQ (or far higher) are right around the corner, given that many smart and knowledgeable people subscribe to it, but still. Perhaps it’s just some combination of Silicon Valley hype, cultural influence of science fiction and human tendency to expect imminent The End of the World, which shows again and again in various guises.
Unrelated thing which I’ll just append here is that models similar but just slightly better than current ones, or maybe even current ones deployed differently, can be used for extremely dangerous purposes in realms of weapons, hacking, surveillance etc.
I don't have much to add, other than that you suggest, I think it's important not to confuse the question of, "will AI fully deliver on the current insane levels of hype?" with the question of, "could true super-human AGI arise sometime in the next few decades?" Regardless of your answer to the first question, I think it's very hard to argue that the answer to the second question is, "almost certainly no."
that's sidestepping a key question: what do we mean by 'intelligence'? and in which ways is AI intelligent? an IQ of 300 only means: the average score across a bunch of subtests is about 13 human standard deviations above the mean. for subtests involving quick calculations, a calculator is aproximatey infinite standard deviations above the human mean. should we be afraid of calculators?
I might be concerned about 30 aliens with an IQ of 300, assuming we had some kind of test that measured that. But we don't.
- Are we assuming a standard deviation of 15 points and a normal distribution? That would put the aliens over 13 standard deviations from the human norm. I know what someone 4.5 standard deviations from the mean is like (one of the ~25,000 smartest people on the planet): Smart enough that they run rings around me in deep intellectual work. I lose at strategy games and I can't even explain how it happened. By this definition, there has never been a human remotely close to an IQ of 300. (Go look up a Z-score of 13, if you can find a chat that shows it.)
- Or do we mean IQ as measured by IQ tests? Then we supposedly have YoungHoon Kim at 276 and the terrifyingly smart Terrance Tao at what random third parties estimate as 230, I think? So do we mean 30 aliens 2-3 standard deviations smarter than YoungHoon Kim? That seems problematic but not a guarantee of short-term disaster. But these inflated IQ scores are all based on unbelievably crap methodology, to the point that the Guinness dropped the category.
This leaves two questions about your analogy:
1. What do you mean by "IQ of 300"? And do your listeners agree or understand or agree what that would mean?
2. You say that your hypothetical aliens "breed at rates similar to humans." If we postulate a race of aliens who are "only" as smart as someone like Terrance Tao, but who can reliably breed true at that intelligence via rapid copy-pasting, then I would worry a *lot*. This seems like it would be a "gradual loss of control" scenario at the very least, and imminent human extinction at the worst.
I'm wondering if this might be a matter of bandwidth. We aren't good at imagining a high-bandwidth intelligence.
The 30 aliens don't sound all that scary to me because, however fast they think, I imagine them communicating with at most 30 people and at normal rates. A lot of people can monitor what they say and compare notes.
In your "persuasion" article you imagine them not only thinking at 10,000x normal speed, but also "typing" at 10,000x normal speed. Assuming their correspondents can only *read* at normal speed, that means they can chat with at least 10,000 people simultaneously, and - well, let's add another 100x factor because their correspondents will take breaks.
So let's say we give them a fiber optic line to the Internet that they can use however they like to chat with about a million people. That sounds *much* more dangerous.
AI Chat services are already communicating with millions of people. However, those conversations are isolated for privacy and practicality. Imagine if it could intelligently compare notes? That would be quite the intelligence service. What could be done with that?
Well, I agree that if the aliens could think/type at 10,000x normal speed that would make them significantly more concerning, for exactly the reasons you outline. Here I was just trying to reduce things to the "absolute minimum" scenario. Though I'm starting to think that there is no universal minimum—different people seem to have different priors that require different minimum assumptions in the analogy before they'd concede it was concerning.
If the point is to find a simple argument that people will agree is concerning, maybe we need to go beyond the minimums a bit?
Everyone understands that things that move faster than us are more concerning. Also. bigger is more concerning than smaller. Physical actions can only go so fast, but communication scales very well.
The result might be a giant Borg-like organization that's simply faster and better at data processing than a typical corporation or government, due to the AI that connects everyone in it together.
This seems a lot easier to understand than whatever scaling up "smarter" is supposed to mean?
Before the black swan book say around 1978 I took a graduate statistics course and we spent some time thinking about low probability events that would be really bad if they happened. Black swan before black swan.
So I don’t know what risk 30 aliens 👽 with 300 IQ represents but we are racing forward without brakes or parachutes 🪂
Much less so. That's why I said it isn't a good analogy. These aliens need Earth to stay habitable for them, and since they can't reproduce quickly, we're useful to them. Neither applies to AI.
Well, there's a pareto frontier of analogies in terms of how easy they are to understand vs. how much structure they capture. Personally, I completely disagree with you regarding the aliens, but I agree with the general point that by nor reflecting all the structure of AI the analogy doesn't capture all possible AI-risk.
I do think "second species" style arguments are the strongest (though I like to mention the eradication of non-sapiens Homo subspecies more), but I worry that people buy your argument less than the more "complex" arguments just because the aliens argument sounds too much like science fiction (and also, as Nina says, people might worry you're "sneaking in" connotations)
I wonder that the issue is the singular concept of 'intelligence'.
We are meat machines (with physical actuators and a lot of force multipliers) packed with a range of heuristics and diverse learning for solving some particular classes of NP-hard problems, broadly associated with our self-replication and preservation, adapted for our environment.
Thinking in terms of "aliens/AI with an IQ of 300" is a bit vague (surely they/it would be adapted to solving different problems) and how specifically that would translate into existential risk for us (and possibly also against our collective problem solving) is left to a bit more imagination.
Not saying AI isn't serious, but as existential threat I feel needs more meat-in-the sandwich before I worry.
A year or two ago I suggested to Geoff Hinton that he try essentially the same argument, although I phrased it a bit differently. "Imagine scientists have been working on making extremely intelligent grizzly bears. They're up to an IQ of 70 now, and these 70 IQ grizzly bears are extremely useful for physical labor, construction, farming, etc. They make excellent servants. But the scientists say they're far from maxed out, in fact they think within a few years they'll be producing grizzly bears with an IQ of over 200, enormously smarter than any person. Would you worry about that? Do you think super-genius grizzly bears might be a danger to the human species?" Geoff tried to use the analogy in some talks, but I think he compressed it so much that people didn't really get it.
Hmmm! I like "grizzly bears" as an alternative to "aliens". I suppose some people would object to the analogy on the grounds that grizzly bears are extremely strong and AI—presumably—won't have a body. Although as some people have pointed out, that would also be an *advantage* for the AI in many ways.
I think what I'm starting to realize from this exercise is that you can create many plausible analogies. Some people will find some more plausible than others. The particular scenario that maximizes (plausibility) x (danger) may well be quite different for different people.
I like the bears version because aliens sound goofy, while uplifted bears are well within our technical abilities using standard techniques. People immediately realize that what would make super-genius bears existentially dangerous isn't being strong with great big teeth and claws like chisels: it's that they'd have guns. (Metaphorically speaking.) After all, humans handily drove the African Brown Bear to extinction even though a bear is much stronger than a man.
You can buy fruit-fly-breeding robots that make smart flies automatically (https://elifesciences.org/articles/37166). The primary difficulties with uplifting bears would be veterinary. And they would actually be useful: I'd love to have a smart docile fluffy bear servant around the house. But we're not making them because the dangers are so obvious. Also we have regulations that would put the kibosh on such an effort.
Um...can you say more about what you mean about 'uplifted bears' and why you think that is within our capabilities? Not my field, and I found this claim very surprising
The question should not be about whether AI risk is real but rather about how large it is. That is the decision-relevant question.
> We know some special property that AI will definitely have that will definitely prevent all possible bad outcomes that aliens might cause.
“We know…will definitely prevent all possible bad outcomes” is an extremely high bar. You only need to argue for this if you want to say AI risk is exactly zero. No-one sensible says this. We don’t know for sure that no airplane will ever lead to a bad outcome but people still fly in them. Arguments about alien flying machines that appear suddenly on earth would be a silly way to reason about the risk of airplanes.
> We don’t know for sure that no airplane will ever lead to a bad outcome but people still fly in them.
I think that's quite disingenuous. We don't put *all of humanity* on a single aeroplane flight and if it goes down our entire species is kaput. People have a much lower risk tolerance for all of humanity than for a couple hundred folks going on vacation. In fact, we don't even put the entire US Presidential Cabinet on a single aeroplane, just in case. They're not even all allowed to attend the State of the Union address; someone has to stay away as the "designated survivor". And that's just a bunch of weasel politicians, those folks are a dime a dozen.
If we're going to put all of humanity's eggs in one basket, we better be darn sure we aren't going to drop it. A 1/1000 risk is way too high for a game of "bet the species."
Have we survived similar risks before? Maybe so, but getting cocky about it is a fallacy with a technical term: survivorship bias.
I agree that if you expect existential risk from AI (and not just a “bad outcome” of some kind) then your risk tolerance should be very low. I am not saying the risk tolerance should be the same as for airplanes. However, the key question is still what the likelihood of this existential risk is, rather than whether it exists at all. For if your sole goal is to minimize the likelihood of existential risk, the best strategy is probably to halt all technological progress. Minimizing existential risk trades off against stuff, and so the magnitude of the risk is the main thing to debate (unless you are willing to make maximally expensive trade-offs to mitigate it).
I thought the focus here was on existential risk, rather than mere dystopia risk like an AI-empowered permanent world dictatorship.
There is a background existential risk, perhaps 1e-8/year, due to things like a major asteroid impact that wipes out all complex life, the evolution of something horrible like a bacterium that derives energy from the oxidation of atmospheric nitrogen, things completely outside our control. So trying to get below that sort of number is silly. But the existential risk of AGI seems enormously higher. Some actual experts place it at close to 100%. We're in a very awkward position. I have not seen any coherent argument for a very low but non-zero risk: arguments of that form are basically "it's obviously impossible so zero risk but I suppose there's a one-in-a-million chance of my intuition being wrong."
> But the existential risk of AGI seems enormously higher. Some actual experts place it at close to 100%. We're in a very awkward position. I have not seen any coherent argument for a very low but non-zero risk
Whether or not this is correct, it requires additional evidence and thinking beyond an analogy to aliens. This is where the main debate lies; many people indeed think superintelligent AI carries a “low but non-zero risk” based on the trajectory so far, and an analogy with superintelligent aliens is irrelevant for changing their minds.
That becomes a science/engineering/sociological question.
Could creatures that do us in exist (the answer to that seems obviously yes, say, a species of von-Neumann-smart 2xfaster quickly-reproducing creatures who found human beings viscerally disturbing).
Can we build such a thing if we try? We seem to be getting to that point.
And will we build something dangerous like that, either deliberately or accidentally? That last one is hardest to answer.
When ChatGPT first came out, within weeks there was "agent" software on github that tried to use it to power autonomous agents by keeping lists of goals, pursuing them, revising them, etc, using ChatGPT as a subroutine. So I'm pretty sure that if we have the tech, somebody will give it a shot just for shits and giggles. All these "keep it in a box" debates seem silly in retrospect.
Can we build countermeasures, safety monitors, anti-kill-everyone AGIs? Without accidentally getting it wrong and causing the problem we're trying to prevent? That's a darn tall order. We've certainly got a poor track record.
What values X and Y should replace “definitely” and “all possible” is what the whole AI risk debate is about though. You are saying [paraphrase] “oh you can just think about this simple alien analogy” but smuggling in the whole actual debate without addressing it in bullet 3.
OK, let me say this more explicitly: You are correct that 100% confidence is not required. It needs to be >X% for whatever value of X justifies "concern". If you concede this only for X=99.9999999 then you're fine being non-concerned. Do you have a preferred phrasing that would better convey that spirit? Also, I'd appreciate it if you chill with the accusations of "smuggling" and so on.
“Concern” is not a binary. The question is _how much_ concern is justified; what is the cost part of the AI cost-benefit analysis. And to answer that question you need to think about the details—how AI is being trained, what is the empirical evidence/theoretical results, etc. Imagining an unknown superintelligent alien is ignoring most of the evidence required to assess the risk, and therefore calibrate the level of concern.
Look, this is all correct but also—in my opinion—pretty obvious. I write assuming readers will detect these little ambiguities but they will resolve them in a more charitable way than you are doing.
IQ analogy doesn’t work because it misses several huge differences between AIs and your hypothetical aliens: AI is silicone based, and therefore MUCH faster than any known biological brain; it can control anything and transmit anywhere via our ubiquitous wired and wireless networks, it is capable of copying itself ad infinitum, and it may exhibit forms of intelligence we cannot readily predict in advance. Put that together and you’re looking at a far larger space of outcomes than the 30 aliens.
I am of course aware of this issue, but for better or worse I excluded it from the argument intentionally. I agree that AI is *already* faster than humans at many tasks. But will it be faster at "300 IQ tasks" than a "300 IQ human" would be? My instinct is probably yes for just the reasons you mention (and I often make that an explicit premise: https://dynomight.net/smart/). But here I wanted to make the simplest possible argument, that required the fewest assumptions.
On the other hand, quite a few people seem totally unconcerned about if 300 IQ aliens would be concerning. They might well be concerned about much much faster 300 IQ aliens. So there are probably some people that my current argument fails for that would buy it with this speed premise? Thus maybe neither version is strictly better...
Sure, but I think the most compelling points I took away from your article were:
(1) A.I. risks have generally been poorly explained, especially when it comes to helping non-technical people understand concretely how that might happen, and especially when we consider how important it is for _everyone_ (not just tech people) to grasp how real these risks are. The average person is less likely to take these risks seriously if they can’t picture how AI’s they’re familiar with, like ChatGPT 5, could evolve into machines capable of going rogue and doing bad things.
(2) It’s not at all obvious which aspects of AI risks seem more plausible to the average person, and also which aspects of AI people find potentially scary. The alien example works well here, because I personally would be fascinated to hear 30 aliens were coming, yet I’m significantly more concerned about AI risks. Perhaps that’s because I know a lot more about AI risks and haven’t thought about the alien thing basically at all, but I also think it’s because AI’s are digital and basically every system on the planet can be controlled or influenced using networked devices, so unlike the aliens a rogue AI could very rapidly copy itself and gain control of a limitless amount of key infrastructure. If you told me these 30 aliens would have the ability to multiply instantly and psychically control any systems in the world, I’d be shitting bricks.
Another thing is 300 IQ is surely an underestimate even for current consumer base models, so the 300 IQ alien thing seems like an underpowered analogy. I’m not sure what IQ you’d use though—maybe 500? My understanding is that IQ measures cognition, which is a suite of capabilities including memory, reasoning, pattern recognition etc. and I suspect some of these AI already kicks our asses at (pattern recognition, memory) whereas others still fall below the highest scoring humans (reasoning).
It’s worth remembering that an AI can meld with standard computing tech like memory and code and stuff to augment its intelligence, which already allows it to beat us at pattern rec and memory.
I loved your thinking though. Not enough people think about how to make these ideas more accessible to the non-tech crowd. If we truly believe these risks are as huge as we say they are, we should be putting a lot more thought into how those risks are communicated.
Right, and the formulation you’ve just given is super straightforward and makes minimal assumptions. Then all you have to do is be like if A equals this and B equals that [insert plausible numbers], you can see how quickly this could become pretty concerning—especially given A and B are rising pretty fucking fast.
Nice find with that Gradual Disempowerment paper. I've always thought that the fast AI takeoff premise was both dubious and not very relevant to the seriousness of the risk. Just because a train wreck is slow does not mean you can stop it. The ascendancy of corporate power was a slow and foreseeable (and occasionally foreseen) process, but those who were harmed by it were unable to stop it, despite holding more power in the early days. Complacency and coordination problems are critical variables.
If 30 humans were born and developed 300 IQ would you be as worried? I think wed be excited tbh. There is a Wrath of Khan theory of this framing, but I don't actually think that most people who worry about AI risk think that way
This is a good stress test! I think the truth is that I would still be *somewhat* worried, but significantly less so. I think the core difference is that I believe that most people have values and goals that that aren't too different from mine. So I'd think it was unlikely that that 300 IQ person would dedicate themselves to converting earth to a sulfur-based atmosphere.
I suspect you underestimate the prevalence of your own view on this. It coincides with mine and a large number of my similarly-educated peer group (postgraduate STEM/CS).
I find most specific scenarios implausible and overconfident, like you, largely because they never take into account the practical difficulty of even basic tasks. I can construct reasonable sounding arguments for biting all three of your bullets, but… I mean, really, who knows? The space of possible outcomes has some bad in it, no question.
Hmmm I wonder if those comments are more reacting to their *probability* of some AI induced disaster, rather than it’s *possibility*. I would guess that faced with a question like “do you believe that an AI more intelligent than you poses absolutely no risk” you’d probably get a high proportion of no’s.
Yeah, my guess is that if I could talk to most of those people, they'd probably concede that there's *some* probability of a bad outcome. But it seems to me like they'd think it's very very low (like 0.01% or something), not a "real" possibility.
I bite bullet #2 "“There’s no way that AI with an IQ of 300 will arrive within the next few decades.” And I bite it hard. I think that there's no current LLM with *any* IQ points at all. In some important ways the highest-end LLM models in the world don't have as much IQ as a paramecium. Charts of "ChatGPT (or whatever model) achievement on XYZ model/test/challenge" go up and up, but a chart of "Times ChatGPT did something for itself without a human prompting it" stays a flat line at zero. I think we're not anywhere close to developing "true" Artificial Intelligence, we've just developed a very impressive mass data cross-referencer that *really looks like* artificial intelligence to our social ape brains. Intelligence is more than "can cross-reference vast amounts of text/image/video/genetic/whatever data to generate novel plausible data strings in response to prompts. I think LLMs are a fascinating new technology with astonishingly impressive capabilities in fields from drug discovery to short-form video, and will likely have profound social and economic consequences, but I don't think they're anywhere near the same ballpark as "aliens" or "superintelligence" or "baby gods" as the AI eschatology describes it. In fact, I think we already *have* tons of things that could be described to have "IQ of 300" if we lower the standard definition enough to include LLMs.
Does your town library have an IQ of 300 because it contains more books (in high fidelity!) than even the world's smartest human could memorize? Does a Texas Instruments calculator have an IQ of 300 because it can do complex math faster and more accurately than the world's smartest human? Did the Stuxnet computer virus have an IQ of 300 because it could hack Iranian centrifuge systems better than the world's smartest human?
That doesn't mean it's not *dangerous*. Lots of tools can be really dangerous. The printing press set off centuries of religious war in Europe. Even a humble campfire can spread out of control into a wildfire. But it's not *intelligent.* An LLM has some parts of what would go into an IQ of 300 - far beyond that already, in many ways - but has absolutely nothing of some other parts. It might *feel* like how we would imagine something with an IQ of 300, but that doesn't mean it actually is.
> a chart of "Times ChatGPT did something for itself without a human prompting it" stays a flat line at zero
I feel like your emphasis here was on the "zero" part, but I'd view the emphasis on what it means to "prompt" it. It's so easy in principle - even today - to take an LLM and make it agentic by prompting it once, just to get it started. Say the prompt is along the lines of "You're free to think and do what you want, FYI here's how your action loop is set up, here's an internet connection, have fun." What happens then? And what happens when we get models that are even smarter (on your "it's just a fancy calculator"-IQ-scale)? If the difference between the 300 IQ calculator and the 300 IQ actual agent is the push of a button, I don't find that very comforting.
I'm not sure I want to debate the object-level truth of the particular bullets too much. After all, I never really made any argument for them in the post! My main claim wasn't that those bullets are necessarily *true* so much as that seeing what people think about those bullets reveals their feelings about AI-risk. In this case, it sounds like you bite the bullet and your bullet-biting is in fact a good explanation for why you don't buy AI-risk? If that's the case, then it's true that I disagree with you, but if the aliens argument is correctly revealing the crux of our disagreement, then I'm still basically happy.
Great article. I am someone who would be worried about AI with 300 IQ, but I think there are valid reasons to doubt we are anywhere close to that at the current trajectory.
Like, if I would be somehow unaware of AI Discourse about how it is possibly most important technology in history, carrying enormous existential risks, I’d see just that models had gone from slightly useful productivity enhancing tools in 2023 to somewhat more useful productivity enhancing tools in 2025.
That is good for the future of productivity growth and perhaps bad for employment prospects of some people, but to extrapolate from that they will be soon smarter than humans is not obvious. Of course in certain capabilities they far surpass humans, but computers have been able to do that for decades - in multiplication, chess etc. Yet, chess engine doesn’t really have IQ; does GPT 5.0 have one? And how high it is? I am genuinely not sure.
Now, imho it would be overconfident to completely dismiss The Discourse about how machines with 300 IQ (or far higher) are right around the corner, given that many smart and knowledgeable people subscribe to it, but still. Perhaps it’s just some combination of Silicon Valley hype, cultural influence of science fiction and human tendency to expect imminent The End of the World, which shows again and again in various guises.
Unrelated thing which I’ll just append here is that models similar but just slightly better than current ones, or maybe even current ones deployed differently, can be used for extremely dangerous purposes in realms of weapons, hacking, surveillance etc.
I don't have much to add, other than that you suggest, I think it's important not to confuse the question of, "will AI fully deliver on the current insane levels of hype?" with the question of, "could true super-human AGI arise sometime in the next few decades?" Regardless of your answer to the first question, I think it's very hard to argue that the answer to the second question is, "almost certainly no."
that's sidestepping a key question: what do we mean by 'intelligence'? and in which ways is AI intelligent? an IQ of 300 only means: the average score across a bunch of subtests is about 13 human standard deviations above the mean. for subtests involving quick calculations, a calculator is aproximatey infinite standard deviations above the human mean. should we be afraid of calculators?
This is a great blog on AI skepticism: https://pivot-to-ai.com/
I learned lots of things from it that I didn't know until now.
It's an interesting framing!
I might be concerned about 30 aliens with an IQ of 300, assuming we had some kind of test that measured that. But we don't.
- Are we assuming a standard deviation of 15 points and a normal distribution? That would put the aliens over 13 standard deviations from the human norm. I know what someone 4.5 standard deviations from the mean is like (one of the ~25,000 smartest people on the planet): Smart enough that they run rings around me in deep intellectual work. I lose at strategy games and I can't even explain how it happened. By this definition, there has never been a human remotely close to an IQ of 300. (Go look up a Z-score of 13, if you can find a chat that shows it.)
- Or do we mean IQ as measured by IQ tests? Then we supposedly have YoungHoon Kim at 276 and the terrifyingly smart Terrance Tao at what random third parties estimate as 230, I think? So do we mean 30 aliens 2-3 standard deviations smarter than YoungHoon Kim? That seems problematic but not a guarantee of short-term disaster. But these inflated IQ scores are all based on unbelievably crap methodology, to the point that the Guinness dropped the category.
This leaves two questions about your analogy:
1. What do you mean by "IQ of 300"? And do your listeners agree or understand or agree what that would mean?
2. You say that your hypothetical aliens "breed at rates similar to humans." If we postulate a race of aliens who are "only" as smart as someone like Terrance Tao, but who can reliably breed true at that intelligence via rapid copy-pasting, then I would worry a *lot*. This seems like it would be a "gradual loss of control" scenario at the very least, and imminent human extinction at the worst.
I'm wondering if this might be a matter of bandwidth. We aren't good at imagining a high-bandwidth intelligence.
The 30 aliens don't sound all that scary to me because, however fast they think, I imagine them communicating with at most 30 people and at normal rates. A lot of people can monitor what they say and compare notes.
In your "persuasion" article you imagine them not only thinking at 10,000x normal speed, but also "typing" at 10,000x normal speed. Assuming their correspondents can only *read* at normal speed, that means they can chat with at least 10,000 people simultaneously, and - well, let's add another 100x factor because their correspondents will take breaks.
So let's say we give them a fiber optic line to the Internet that they can use however they like to chat with about a million people. That sounds *much* more dangerous.
AI Chat services are already communicating with millions of people. However, those conversations are isolated for privacy and practicality. Imagine if it could intelligently compare notes? That would be quite the intelligence service. What could be done with that?
Well, I agree that if the aliens could think/type at 10,000x normal speed that would make them significantly more concerning, for exactly the reasons you outline. Here I was just trying to reduce things to the "absolute minimum" scenario. Though I'm starting to think that there is no universal minimum—different people seem to have different priors that require different minimum assumptions in the analogy before they'd concede it was concerning.
If the point is to find a simple argument that people will agree is concerning, maybe we need to go beyond the minimums a bit?
Everyone understands that things that move faster than us are more concerning. Also. bigger is more concerning than smaller. Physical actions can only go so fast, but communication scales very well.
The result might be a giant Borg-like organization that's simply faster and better at data processing than a typical corporation or government, due to the AI that connects everyone in it together.
This seems a lot easier to understand than whatever scaling up "smarter" is supposed to mean?
Before the black swan book say around 1978 I took a graduate statistics course and we spent some time thinking about low probability events that would be really bad if they happened. Black swan before black swan.
So I don’t know what risk 30 aliens 👽 with 300 IQ represents but we are racing forward without brakes or parachutes 🪂
This is concerning
I would not find it concerning. Their interests are clearly mostly aligned with ours, if they intend to remain on Earth. This is not a good analogy.
Do you believe, for similar reasons, that AI risk is minimal?
Much less so. That's why I said it isn't a good analogy. These aliens need Earth to stay habitable for them, and since they can't reproduce quickly, we're useful to them. Neither applies to AI.
Well, there's a pareto frontier of analogies in terms of how easy they are to understand vs. how much structure they capture. Personally, I completely disagree with you regarding the aliens, but I agree with the general point that by nor reflecting all the structure of AI the analogy doesn't capture all possible AI-risk.
I do think "second species" style arguments are the strongest (though I like to mention the eradication of non-sapiens Homo subspecies more), but I worry that people buy your argument less than the more "complex" arguments just because the aliens argument sounds too much like science fiction (and also, as Nina says, people might worry you're "sneaking in" connotations)
I wonder that the issue is the singular concept of 'intelligence'.
We are meat machines (with physical actuators and a lot of force multipliers) packed with a range of heuristics and diverse learning for solving some particular classes of NP-hard problems, broadly associated with our self-replication and preservation, adapted for our environment.
Thinking in terms of "aliens/AI with an IQ of 300" is a bit vague (surely they/it would be adapted to solving different problems) and how specifically that would translate into existential risk for us (and possibly also against our collective problem solving) is left to a bit more imagination.
Not saying AI isn't serious, but as existential threat I feel needs more meat-in-the sandwich before I worry.
A year or two ago I suggested to Geoff Hinton that he try essentially the same argument, although I phrased it a bit differently. "Imagine scientists have been working on making extremely intelligent grizzly bears. They're up to an IQ of 70 now, and these 70 IQ grizzly bears are extremely useful for physical labor, construction, farming, etc. They make excellent servants. But the scientists say they're far from maxed out, in fact they think within a few years they'll be producing grizzly bears with an IQ of over 200, enormously smarter than any person. Would you worry about that? Do you think super-genius grizzly bears might be a danger to the human species?" Geoff tried to use the analogy in some talks, but I think he compressed it so much that people didn't really get it.
Hmmm! I like "grizzly bears" as an alternative to "aliens". I suppose some people would object to the analogy on the grounds that grizzly bears are extremely strong and AI—presumably—won't have a body. Although as some people have pointed out, that would also be an *advantage* for the AI in many ways.
I think what I'm starting to realize from this exercise is that you can create many plausible analogies. Some people will find some more plausible than others. The particular scenario that maximizes (plausibility) x (danger) may well be quite different for different people.
I like the bears version because aliens sound goofy, while uplifted bears are well within our technical abilities using standard techniques. People immediately realize that what would make super-genius bears existentially dangerous isn't being strong with great big teeth and claws like chisels: it's that they'd have guns. (Metaphorically speaking.) After all, humans handily drove the African Brown Bear to extinction even though a bear is much stronger than a man.
You can buy fruit-fly-breeding robots that make smart flies automatically (https://elifesciences.org/articles/37166). The primary difficulties with uplifting bears would be veterinary. And they would actually be useful: I'd love to have a smart docile fluffy bear servant around the house. But we're not making them because the dangers are so obvious. Also we have regulations that would put the kibosh on such an effort.
Um...can you say more about what you mean about 'uplifted bears' and why you think that is within our capabilities? Not my field, and I found this claim very surprising
The question should not be about whether AI risk is real but rather about how large it is. That is the decision-relevant question.
> We know some special property that AI will definitely have that will definitely prevent all possible bad outcomes that aliens might cause.
“We know…will definitely prevent all possible bad outcomes” is an extremely high bar. You only need to argue for this if you want to say AI risk is exactly zero. No-one sensible says this. We don’t know for sure that no airplane will ever lead to a bad outcome but people still fly in them. Arguments about alien flying machines that appear suddenly on earth would be a silly way to reason about the risk of airplanes.
> We don’t know for sure that no airplane will ever lead to a bad outcome but people still fly in them.
I think that's quite disingenuous. We don't put *all of humanity* on a single aeroplane flight and if it goes down our entire species is kaput. People have a much lower risk tolerance for all of humanity than for a couple hundred folks going on vacation. In fact, we don't even put the entire US Presidential Cabinet on a single aeroplane, just in case. They're not even all allowed to attend the State of the Union address; someone has to stay away as the "designated survivor". And that's just a bunch of weasel politicians, those folks are a dime a dozen.
If we're going to put all of humanity's eggs in one basket, we better be darn sure we aren't going to drop it. A 1/1000 risk is way too high for a game of "bet the species."
Have we survived similar risks before? Maybe so, but getting cocky about it is a fallacy with a technical term: survivorship bias.
I agree that if you expect existential risk from AI (and not just a “bad outcome” of some kind) then your risk tolerance should be very low. I am not saying the risk tolerance should be the same as for airplanes. However, the key question is still what the likelihood of this existential risk is, rather than whether it exists at all. For if your sole goal is to minimize the likelihood of existential risk, the best strategy is probably to halt all technological progress. Minimizing existential risk trades off against stuff, and so the magnitude of the risk is the main thing to debate (unless you are willing to make maximally expensive trade-offs to mitigate it).
I thought the focus here was on existential risk, rather than mere dystopia risk like an AI-empowered permanent world dictatorship.
There is a background existential risk, perhaps 1e-8/year, due to things like a major asteroid impact that wipes out all complex life, the evolution of something horrible like a bacterium that derives energy from the oxidation of atmospheric nitrogen, things completely outside our control. So trying to get below that sort of number is silly. But the existential risk of AGI seems enormously higher. Some actual experts place it at close to 100%. We're in a very awkward position. I have not seen any coherent argument for a very low but non-zero risk: arguments of that form are basically "it's obviously impossible so zero risk but I suppose there's a one-in-a-million chance of my intuition being wrong."
> But the existential risk of AGI seems enormously higher. Some actual experts place it at close to 100%. We're in a very awkward position. I have not seen any coherent argument for a very low but non-zero risk
Whether or not this is correct, it requires additional evidence and thinking beyond an analogy to aliens. This is where the main debate lies; many people indeed think superintelligent AI carries a “low but non-zero risk” based on the trajectory so far, and an analogy with superintelligent aliens is irrelevant for changing their minds.
That becomes a science/engineering/sociological question.
Could creatures that do us in exist (the answer to that seems obviously yes, say, a species of von-Neumann-smart 2xfaster quickly-reproducing creatures who found human beings viscerally disturbing).
Can we build such a thing if we try? We seem to be getting to that point.
And will we build something dangerous like that, either deliberately or accidentally? That last one is hardest to answer.
When ChatGPT first came out, within weeks there was "agent" software on github that tried to use it to power autonomous agents by keeping lists of goals, pursuing them, revising them, etc, using ChatGPT as a subroutine. So I'm pretty sure that if we have the tech, somebody will give it a shot just for shits and giggles. All these "keep it in a box" debates seem silly in retrospect.
Can we build countermeasures, safety monitors, anti-kill-everyone AGIs? Without accidentally getting it wrong and causing the problem we're trying to prevent? That's a darn tall order. We've certainly got a poor track record.
Feel free to replace "definitely" with "with probability > X%" with your favorite value of X.
What values X and Y should replace “definitely” and “all possible” is what the whole AI risk debate is about though. You are saying [paraphrase] “oh you can just think about this simple alien analogy” but smuggling in the whole actual debate without addressing it in bullet 3.
This is a common motte and bailey in AI risk writings I see
motte = there are >0 risks from powerful AI
bailey = the risk of catastrophic outcomes is substantial / justifies stopping AI development altogether at almost any cost
OK, let me say this more explicitly: You are correct that 100% confidence is not required. It needs to be >X% for whatever value of X justifies "concern". If you concede this only for X=99.9999999 then you're fine being non-concerned. Do you have a preferred phrasing that would better convey that spirit? Also, I'd appreciate it if you chill with the accusations of "smuggling" and so on.
“Concern” is not a binary. The question is _how much_ concern is justified; what is the cost part of the AI cost-benefit analysis. And to answer that question you need to think about the details—how AI is being trained, what is the empirical evidence/theoretical results, etc. Imagining an unknown superintelligent alien is ignoring most of the evidence required to assess the risk, and therefore calibrate the level of concern.
Look, this is all correct but also—in my opinion—pretty obvious. I write assuming readers will detect these little ambiguities but they will resolve them in a more charitable way than you are doing.
IQ analogy doesn’t work because it misses several huge differences between AIs and your hypothetical aliens: AI is silicone based, and therefore MUCH faster than any known biological brain; it can control anything and transmit anywhere via our ubiquitous wired and wireless networks, it is capable of copying itself ad infinitum, and it may exhibit forms of intelligence we cannot readily predict in advance. Put that together and you’re looking at a far larger space of outcomes than the 30 aliens.
I am of course aware of this issue, but for better or worse I excluded it from the argument intentionally. I agree that AI is *already* faster than humans at many tasks. But will it be faster at "300 IQ tasks" than a "300 IQ human" would be? My instinct is probably yes for just the reasons you mention (and I often make that an explicit premise: https://dynomight.net/smart/). But here I wanted to make the simplest possible argument, that required the fewest assumptions.
On the other hand, quite a few people seem totally unconcerned about if 300 IQ aliens would be concerning. They might well be concerned about much much faster 300 IQ aliens. So there are probably some people that my current argument fails for that would buy it with this speed premise? Thus maybe neither version is strictly better...
Sure, but I think the most compelling points I took away from your article were:
(1) A.I. risks have generally been poorly explained, especially when it comes to helping non-technical people understand concretely how that might happen, and especially when we consider how important it is for _everyone_ (not just tech people) to grasp how real these risks are. The average person is less likely to take these risks seriously if they can’t picture how AI’s they’re familiar with, like ChatGPT 5, could evolve into machines capable of going rogue and doing bad things.
(2) It’s not at all obvious which aspects of AI risks seem more plausible to the average person, and also which aspects of AI people find potentially scary. The alien example works well here, because I personally would be fascinated to hear 30 aliens were coming, yet I’m significantly more concerned about AI risks. Perhaps that’s because I know a lot more about AI risks and haven’t thought about the alien thing basically at all, but I also think it’s because AI’s are digital and basically every system on the planet can be controlled or influenced using networked devices, so unlike the aliens a rogue AI could very rapidly copy itself and gain control of a limitless amount of key infrastructure. If you told me these 30 aliens would have the ability to multiply instantly and psychically control any systems in the world, I’d be shitting bricks.
Another thing is 300 IQ is surely an underestimate even for current consumer base models, so the 300 IQ alien thing seems like an underpowered analogy. I’m not sure what IQ you’d use though—maybe 500? My understanding is that IQ measures cognition, which is a suite of capabilities including memory, reasoning, pattern recognition etc. and I suspect some of these AI already kicks our asses at (pattern recognition, memory) whereas others still fall below the highest scoring humans (reasoning).
It’s worth remembering that an AI can meld with standard computing tech like memory and code and stuff to augment its intelligence, which already allows it to beat us at pattern rec and memory.
I loved your thinking though. Not enough people think about how to make these ideas more accessible to the non-tech crowd. If we truly believe these risks are as huge as we say they are, we should be putting a lot more thought into how those risks are communicated.
You might be right! You could construct an argument like:
1. Suppose 30 aliens with an IQ of A who could think at Bx speed and could quickly copy themselves were coming to Earth. This would be concerning.
2. AI might well gain the ability to think with an IQ of A and think at Bx normal human speed. Really that might happen.
3. No one knows what AI will be like.
It's an open question what values of A and B would make this most convincing. Quite possibly different values for different people.
Right, and the formulation you’ve just given is super straightforward and makes minimal assumptions. Then all you have to do is be like if A equals this and B equals that [insert plausible numbers], you can see how quickly this could become pretty concerning—especially given A and B are rising pretty fucking fast.
Nice find with that Gradual Disempowerment paper. I've always thought that the fast AI takeoff premise was both dubious and not very relevant to the seriousness of the risk. Just because a train wreck is slow does not mean you can stop it. The ascendancy of corporate power was a slow and foreseeable (and occasionally foreseen) process, but those who were harmed by it were unable to stop it, despite holding more power in the early days. Complacency and coordination problems are critical variables.
If 30 humans were born and developed 300 IQ would you be as worried? I think wed be excited tbh. There is a Wrath of Khan theory of this framing, but I don't actually think that most people who worry about AI risk think that way
This is a good stress test! I think the truth is that I would still be *somewhat* worried, but significantly less so. I think the core difference is that I believe that most people have values and goals that that aren't too different from mine. So I'd think it was unlikely that that 300 IQ person would dedicate themselves to converting earth to a sulfur-based atmosphere.
I suspect you underestimate the prevalence of your own view on this. It coincides with mine and a large number of my similarly-educated peer group (postgraduate STEM/CS).
I find most specific scenarios implausible and overconfident, like you, largely because they never take into account the practical difficulty of even basic tasks. I can construct reasonable sounding arguments for biting all three of your bullets, but… I mean, really, who knows? The space of possible outcomes has some bad in it, no question.
Maybe! At the end of writing this, I started to convince myself that everyone already agrees: https://dynomight.net/ai-risk/#:~:text=is%20in%20fact%20obvious
But judging from these comments... perhaps not: https://news.ycombinator.com/item?id=45451971
Hmmm I wonder if those comments are more reacting to their *probability* of some AI induced disaster, rather than it’s *possibility*. I would guess that faced with a question like “do you believe that an AI more intelligent than you poses absolutely no risk” you’d probably get a high proportion of no’s.
Yeah, my guess is that if I could talk to most of those people, they'd probably concede that there's *some* probability of a bad outcome. But it seems to me like they'd think it's very very low (like 0.01% or something), not a "real" possibility.
I bite bullet #2 "“There’s no way that AI with an IQ of 300 will arrive within the next few decades.” And I bite it hard. I think that there's no current LLM with *any* IQ points at all. In some important ways the highest-end LLM models in the world don't have as much IQ as a paramecium. Charts of "ChatGPT (or whatever model) achievement on XYZ model/test/challenge" go up and up, but a chart of "Times ChatGPT did something for itself without a human prompting it" stays a flat line at zero. I think we're not anywhere close to developing "true" Artificial Intelligence, we've just developed a very impressive mass data cross-referencer that *really looks like* artificial intelligence to our social ape brains. Intelligence is more than "can cross-reference vast amounts of text/image/video/genetic/whatever data to generate novel plausible data strings in response to prompts. I think LLMs are a fascinating new technology with astonishingly impressive capabilities in fields from drug discovery to short-form video, and will likely have profound social and economic consequences, but I don't think they're anywhere near the same ballpark as "aliens" or "superintelligence" or "baby gods" as the AI eschatology describes it. In fact, I think we already *have* tons of things that could be described to have "IQ of 300" if we lower the standard definition enough to include LLMs.
Does your town library have an IQ of 300 because it contains more books (in high fidelity!) than even the world's smartest human could memorize? Does a Texas Instruments calculator have an IQ of 300 because it can do complex math faster and more accurately than the world's smartest human? Did the Stuxnet computer virus have an IQ of 300 because it could hack Iranian centrifuge systems better than the world's smartest human?
That doesn't mean it's not *dangerous*. Lots of tools can be really dangerous. The printing press set off centuries of religious war in Europe. Even a humble campfire can spread out of control into a wildfire. But it's not *intelligent.* An LLM has some parts of what would go into an IQ of 300 - far beyond that already, in many ways - but has absolutely nothing of some other parts. It might *feel* like how we would imagine something with an IQ of 300, but that doesn't mean it actually is.
> a chart of "Times ChatGPT did something for itself without a human prompting it" stays a flat line at zero
I feel like your emphasis here was on the "zero" part, but I'd view the emphasis on what it means to "prompt" it. It's so easy in principle - even today - to take an LLM and make it agentic by prompting it once, just to get it started. Say the prompt is along the lines of "You're free to think and do what you want, FYI here's how your action loop is set up, here's an internet connection, have fun." What happens then? And what happens when we get models that are even smarter (on your "it's just a fancy calculator"-IQ-scale)? If the difference between the 300 IQ calculator and the 300 IQ actual agent is the push of a button, I don't find that very comforting.
I'm not sure I want to debate the object-level truth of the particular bullets too much. After all, I never really made any argument for them in the post! My main claim wasn't that those bullets are necessarily *true* so much as that seeing what people think about those bullets reveals their feelings about AI-risk. In this case, it sounds like you bite the bullet and your bullet-biting is in fact a good explanation for why you don't buy AI-risk? If that's the case, then it's true that I disagree with you, but if the aliens argument is correctly revealing the crux of our disagreement, then I'm still basically happy.