This kind of statistics always makes my head spin. Suppose you give me a dice and tell me it's fair and I believe you (that's my prior). I roll it 62 times and six never comes up. Should I revise my prior to some lower percentage of belief that it's fair? Should I shrug my shoulders and say "welp, sometimes weird stuff happens with sequences of random independent events"? Or should examine the dice more closely and say "wait a minute, this dice has two 1s and no 6!"?
If you really, REALLY believe me that it's fair, then you should never update no matter how many times you roll!
If that sounds crazy, I think what it shows is that you shouldn't 100% believe me. In practice, you should almost always have some small probability that I'm lying. The size of that probability then determines how many rolls you need to be convinced that I was lying after all. Then, the difference between 99.9% believing me and 99.9999% believing me makes a huge difference.
And this is my problem with the whole Bayesian thing: where do priors come from? What possible experience could I have that could make the difference between 99.9% believing you and 99.9999% believing you? Every worked example I have ever seen, both the prior and the degree of confidence in the prior seem to be so vaguely justifiable they might as well have been plucked out of thin air.
And I think that this is a practical problem for anybody trying to use this to make really decisions. Most people can't even interpret what "10% chance of rain today" means, let alone 1% chance of nuclear war. I just don't see how to put a hard number to a prior belief?
Eh, I certainly agree priors can be hard. But what's the alternative? At the end of the day in the dice scenario, you need to decide *somehow* when you want to stop trusting me and start "trusting the dice". To me, whatever policy you decide on will ultimately amount to something fairly similar to a prior. I'd rather just be upfront about it.
Now, not that this detracts from your point... But I do see 99.9% and 99.9999% as very different! Like, if you walk up to my door and give me a pie and tell me that it's safe to eat, I'd believe:
- A random stranger: 80%
- A random acquaintance: 99%
- A good friend: 99.9%
- A lifelong friend who I know to be very conscientious and has documented proof of spending the last 20 years doing nothing but visiting friends and giving them totally safe pies: 99.999%
When you're looking at probabilities near 100%, it's usually easier to think of the opposite probability. If you say person A has a 0.1% probability of lying and person B has a 0.00001% probability, then you trust person B 10,000x more. That's a huge difference!
I am not sure I think that 10,000x difference is a huge difference, because it is a huge difference relative to a super tiny number. I think that is sort of what Jacob is getting at, that the differences are not something humans can make decisions based around. In your pie example it is hard to imagine you making a different decision based on those differences of belief past random stranger (80% seems high there). Like, what might happen to make you trust the lifelong friend but not the good friend? 10 bad pies in a row from the good friend and you stop trusting them, but you would still believe the lifelong friend?
I am not trying to be sarcastic or anything, I honestly just can't imagine what event might split that difference and make those two priors less than equivalent.
Well... Say you drive a moderately dangerous car without a seatbelt and you sell it and get an extremely safe car and start wearing a seatbelt. You're probably changing your changes of "not dying in a car crash each day" from 99.9999% to something like 99.99999%. That's only 10x. But most people consider this worthwhile, and I don't think they're right, because compounded over a lifetime, this makes a real difference.
I suspect the real difference is swamped by just about everything else, like living in a place with fewer accidents, or not being the kind of person who gets in car accidents much, etc.
In fact, I suspect that actuarially, those people in those cars would be entirely indistinguishable.
Or to look at it another way, a 99.9999% chance of not dying in your car going to a 99.99999% is not something meaningful when compared to, say, moving to a safer neighborhood, or exercising a little more, or watching what one eats or not drinking alcohol. People -might- make the choice to go with the safer car and buckle up because of those numbers, but unless they are doing ALL the other things that undoubtedly have a higher return on safety, both in terms of absolute life time expectancy and relative to costs, it is hard to say they are making the decision based on that additional 9. They might hear "10x safer!" but making that decision and not the other higher quality ones is hardly rationally making a decision based on the numbers. In fact, I would be willing to bet they weren't even really counting how many 9's there were, such that if you said 99.999% to 99.9999% one day and 99.9% to 99.99% the next exactly zero people would say "Woah, woah! That was a much better change yesterday! Now it is 10x worse than the base rate!"
I think the obvious concern in the context of this post is that if you applied that principle to the world in October 1963, then I believe it would dictate that "there's a 75% chance that you're in the first 25% of the interval between Cuban-missile-crisis-level-crises", and so there's a 75% chance the next such crisis would come in the next 3 years.
This inspired me to think through the problem some more and come up with a formulation that helps me wrap my head around the limits of the anthropic principle. I'll share it here in case it helps anyone else. (I'm sure I'm not the first person to come up with this idea.)
Suppose you find a box of Russian Roulette Hand Grenades. You know that when you pull the pin on such a grenade, it might explode, or it might turn out to be a dud. You also know that two types were manufactured, one with a 10% probability of exploding and one with a 90% probability of exploding. Let's call them Easy Mode grenades and Hard Mode grenades. You're confident from context that all of the grenades in this box are the same type, but you don't know whether it's the Easy or Hard type.
Assume your prior is that it's equally likely that the box contains either type of grenade. You would predict that if you pull the pin on one grenade, you have a 50% chance of being killed. Suppose you do in fact pull a pin, and the grenade turns out to be a dud. Should you update your belief as to what type of grenade the box contains?
You could imagine thinking that you've gained no new information: of course the grenade was a dud, because in all worlds where it was not a dud, you are no longer around to update beliefs.
Here's the analysis which contradicts that idea: imagine that, prior to pulling the pin, we fork the universe 100 ways. This yields 50 copies of you who are standing in front of a box of Easy Mode grenades, and 50 copies who are standing in front of Hard Mode grenades. After pulling the pin, there are 45 living copies of you standing in front of Easy Mode grenades, and 5 copies in front of Hard Mode grenades. So if you find yourself alive, you should update to believe that there is a 90% chance the box contains Easy Mode grenades, and an 18% chance that the next grenade would explode if you pull its pin.
This feeds nicely into your point that how much you should update depends on the sharpness of your prior. In my example, the prior was broad (the box could have contained Easy Mode or Hard Mode grenades, which are very very different), and so it updated substantially from a single sample. If Hard Mode grenades only had a 12% chance of exploding (i.e. a sharper prior), then there would be 45 copies of you with Easy Mode grenades and 44 copies with Hard Mode grenades, and you'd only update your beliefs a tiny amount.
Here's a restatement that even more directly attacks the incorrect form of anthropic-principle argument: there are more ways to have a universe with an unexploded Easy Mode grenade than a universe with an unexploded Hard Mode grenade. (If you imagine that each grenade generates a random number from 0-99 when you pull the pin, then the Easy Mode universe could feature a grenade showing any number from 10 to 99, while the Hard Mode universe could only feature grenades showing numbers from 90 to 99.) So if you find yourself in a universe with an unexploded grenade, it's probably an Easy Mode grenade. Similarly, if you find yourself in a universe where it is the year 2024 and there has not been a catastrophic nuclear war, that should update you somewhat in the direction that catastrophic nuclear wars are unlikely.
These are good examples. (Sorta similar to the happy puppies, perhaps?) I'd like to find an example that is even simpler (and/or less contrived) but I just haven't been able to come up with one.
FWIW, the problem I was having was this: all of these thought experiments, around happy puppies and so forth, seem quite valid. Logically, they explain why you should update on evidence like "there hasn't yet been a nuclear war". But my *intuition* remained caught on the (incorrect) idea that an experiment with only one not-killing-you outcome doesn't reveal any new information: of course you saw the not-killing-you result! It's hard for a logical argument to overthrow an intuitive idea.
The multiple-universes framing helped me to directly attack that intuition, by pointing out that the distribution of observable universes in which I've pulled the pin on a grenade (and, of course, survived) is a subset of the observable universes in which I haven't yet pulled the pin, so pulling the pin does yield information (the information that I'm in that subset) which I didn't have prior to pulling it. And similarly with regard to the fact that we haven't (yet) had a nuclear war.
How does this sit with you? Maybe part of what's unintuitive is that updating on information that you only get to see in one universe seems "biased". And it *is* biased, if you want an estimate for the average over all universes. But if you need to make a decision based on the state of *your* universe, then that "bias" is a good thing.
Enjoyed this but might add that it seems to me quite misleading to think of probabilities as inherent properties of the world; Probability is a conceptual tool we use to reason about the world. There isn't an underlying truth of "likelihood of nuclear war" though in certain cases it can be quite handy to think of it this way.
If you talk about "the probability that I weigh more than 70kg" that's a very different thing from the "probability that this U-235 atom will decay in the next 100 years". (They reflect "epistemic" vs "aleatory" uncertainty.) Some people don't like using the word "probability" for the first kind of probability but I find that to be a kind of semantic issue as long as everyone understands what's intended.
That said, I think these kinds of debates often disappear when you think about making decisions (or bets). For nuclear war, the question is how much resources to invest in preventing nuclear war or bomb shelters or whatever.
This is essentially the question raised (and discussed at length in the philosophy literature) by the sleeping beauty paradox.
In terms of pure decision theory for you personally, I think everything you said is fine. The problem is that's not the only way we use probability talk -- indeed I suspect that the way we use probability talk conflates two different ideas.
Basically the idea in sleeping beauty is that you are drugged into unconsciousness on Sunday and your captor flips a coin. If the coin is heads the captor wakes you up on Monday then wipes your mind and lets you go. If the coin is tails the captor wakes you up on Monday, wipes your memory and redrugs you and wakes you up again on Tuesday.
Suppose you wake up in the captor's possession one morning and the captor explains the situation (but not what day it is) what should your probability be the coin landed heads? Well it seems obvious it's 50% but now suppose you know the captor will offer you a $100 bet every time you wake about whether the coin landed tails at just below even odds.
Obviously you should take the bet because if the coin lands tails you will win twice but if it lands heads you lose once. That suggests you should assign probability of tails to be 2/3rds.
Ultimately, I think the confusion is over which of two questions one is answering (probability is just fancy counting). Are you trying to count up the situations from your perspective or from some kind of imagined 3rd party perspective?
To put it roughly in another way, there are an equal number of 'ways' the universe turns out where the captor gets a head and a tails. However, there are twice as many 'ways' you wake up in said situation with the coin tails as heads. So it's all just about what you are counting.
I think the point here is not about tightness of priors, but rather about whether a given observation is informative.
If you think that the only possible observation you can get is “Earth still exists”, then you might think that observing that Earth exist gives you no information, so you should not update your priors of yearly rate of Earth destruction, no matter how loose your priors were.
If, on the other hand, you think that the fact of observing Earth is itself contingent, then observing “Earth still exist” is informative, and you should update your priors (how much indeed depends on the tightness of your priors, but that is always the case).
I tend to agree with you observing the Earth existing is informative, and we should update. But probabilistic reasoning about events that only happen once are tricky.
For what it's worth, this is more or less the view that I wrote this post to try to refute! I mean, if an observation isn't information, sure you should update less. But if the *only* reason that an observation seems less information is because you might not exist, then I think the amount you should update depends 100% on your prior and 0% on the fact that you might not exist.
I think the post is a good one (I like the comming up with transparent examples), but I take issue with:
"The crux of the nuclear war scenario isn’t that you might stop existing. That’s just a very salient feature that draws our attention away from the heart of the debate: The confidence of your prior."
I think that part is wrong, and from your answer it seems you agree. Am I wrong? It also seems rather central to the post, so my objection doesn't seem nitpicking.
Also, "if an observation isn't information, sure you should update less.". I think in that case you should update nothing at all.
This kind of statistics always makes my head spin. Suppose you give me a dice and tell me it's fair and I believe you (that's my prior). I roll it 62 times and six never comes up. Should I revise my prior to some lower percentage of belief that it's fair? Should I shrug my shoulders and say "welp, sometimes weird stuff happens with sequences of random independent events"? Or should examine the dice more closely and say "wait a minute, this dice has two 1s and no 6!"?
If you really, REALLY believe me that it's fair, then you should never update no matter how many times you roll!
If that sounds crazy, I think what it shows is that you shouldn't 100% believe me. In practice, you should almost always have some small probability that I'm lying. The size of that probability then determines how many rolls you need to be convinced that I was lying after all. Then, the difference between 99.9% believing me and 99.9999% believing me makes a huge difference.
And this is my problem with the whole Bayesian thing: where do priors come from? What possible experience could I have that could make the difference between 99.9% believing you and 99.9999% believing you? Every worked example I have ever seen, both the prior and the degree of confidence in the prior seem to be so vaguely justifiable they might as well have been plucked out of thin air.
And I think that this is a practical problem for anybody trying to use this to make really decisions. Most people can't even interpret what "10% chance of rain today" means, let alone 1% chance of nuclear war. I just don't see how to put a hard number to a prior belief?
Eh, I certainly agree priors can be hard. But what's the alternative? At the end of the day in the dice scenario, you need to decide *somehow* when you want to stop trusting me and start "trusting the dice". To me, whatever policy you decide on will ultimately amount to something fairly similar to a prior. I'd rather just be upfront about it.
Now, not that this detracts from your point... But I do see 99.9% and 99.9999% as very different! Like, if you walk up to my door and give me a pie and tell me that it's safe to eat, I'd believe:
- A random stranger: 80%
- A random acquaintance: 99%
- A good friend: 99.9%
- A lifelong friend who I know to be very conscientious and has documented proof of spending the last 20 years doing nothing but visiting friends and giving them totally safe pies: 99.999%
When you're looking at probabilities near 100%, it's usually easier to think of the opposite probability. If you say person A has a 0.1% probability of lying and person B has a 0.00001% probability, then you trust person B 10,000x more. That's a huge difference!
I am not sure I think that 10,000x difference is a huge difference, because it is a huge difference relative to a super tiny number. I think that is sort of what Jacob is getting at, that the differences are not something humans can make decisions based around. In your pie example it is hard to imagine you making a different decision based on those differences of belief past random stranger (80% seems high there). Like, what might happen to make you trust the lifelong friend but not the good friend? 10 bad pies in a row from the good friend and you stop trusting them, but you would still believe the lifelong friend?
I am not trying to be sarcastic or anything, I honestly just can't imagine what event might split that difference and make those two priors less than equivalent.
Well... Say you drive a moderately dangerous car without a seatbelt and you sell it and get an extremely safe car and start wearing a seatbelt. You're probably changing your changes of "not dying in a car crash each day" from 99.9999% to something like 99.99999%. That's only 10x. But most people consider this worthwhile, and I don't think they're right, because compounded over a lifetime, this makes a real difference.
I will ask you to do the math on that one :)
I suspect the real difference is swamped by just about everything else, like living in a place with fewer accidents, or not being the kind of person who gets in car accidents much, etc.
In fact, I suspect that actuarially, those people in those cars would be entirely indistinguishable.
Or to look at it another way, a 99.9999% chance of not dying in your car going to a 99.99999% is not something meaningful when compared to, say, moving to a safer neighborhood, or exercising a little more, or watching what one eats or not drinking alcohol. People -might- make the choice to go with the safer car and buckle up because of those numbers, but unless they are doing ALL the other things that undoubtedly have a higher return on safety, both in terms of absolute life time expectancy and relative to costs, it is hard to say they are making the decision based on that additional 9. They might hear "10x safer!" but making that decision and not the other higher quality ones is hardly rationally making a decision based on the numbers. In fact, I would be willing to bet they weren't even really counting how many 9's there were, such that if you said 99.999% to 99.9999% one day and 99.9% to 99.99% the next exactly zero people would say "Woah, woah! That was a much better change yesterday! Now it is 10x worse than the base rate!"
Are you familiar with Gott's Copernican principle?
https://en.m.wikipedia.org/wiki/J._Richard_Gott
I wasn't but thanks for the pointer!
I think the obvious concern in the context of this post is that if you applied that principle to the world in October 1963, then I believe it would dictate that "there's a 75% chance that you're in the first 25% of the interval between Cuban-missile-crisis-level-crises", and so there's a 75% chance the next such crisis would come in the next 3 years.
Probability of at least one 1% chance thing happening after 62 tries is only 45%, not more likely than not... We just need to wait seven more years 👀
Doh, you're right! 1-.99^62 = 46.4%
Changed to "If a button has a 1% chance of activating and you press it 62 times, the odds are almost 50/50 that it would activate." thanks!
This inspired me to think through the problem some more and come up with a formulation that helps me wrap my head around the limits of the anthropic principle. I'll share it here in case it helps anyone else. (I'm sure I'm not the first person to come up with this idea.)
Suppose you find a box of Russian Roulette Hand Grenades. You know that when you pull the pin on such a grenade, it might explode, or it might turn out to be a dud. You also know that two types were manufactured, one with a 10% probability of exploding and one with a 90% probability of exploding. Let's call them Easy Mode grenades and Hard Mode grenades. You're confident from context that all of the grenades in this box are the same type, but you don't know whether it's the Easy or Hard type.
Assume your prior is that it's equally likely that the box contains either type of grenade. You would predict that if you pull the pin on one grenade, you have a 50% chance of being killed. Suppose you do in fact pull a pin, and the grenade turns out to be a dud. Should you update your belief as to what type of grenade the box contains?
You could imagine thinking that you've gained no new information: of course the grenade was a dud, because in all worlds where it was not a dud, you are no longer around to update beliefs.
Here's the analysis which contradicts that idea: imagine that, prior to pulling the pin, we fork the universe 100 ways. This yields 50 copies of you who are standing in front of a box of Easy Mode grenades, and 50 copies who are standing in front of Hard Mode grenades. After pulling the pin, there are 45 living copies of you standing in front of Easy Mode grenades, and 5 copies in front of Hard Mode grenades. So if you find yourself alive, you should update to believe that there is a 90% chance the box contains Easy Mode grenades, and an 18% chance that the next grenade would explode if you pull its pin.
This feeds nicely into your point that how much you should update depends on the sharpness of your prior. In my example, the prior was broad (the box could have contained Easy Mode or Hard Mode grenades, which are very very different), and so it updated substantially from a single sample. If Hard Mode grenades only had a 12% chance of exploding (i.e. a sharper prior), then there would be 45 copies of you with Easy Mode grenades and 44 copies with Hard Mode grenades, and you'd only update your beliefs a tiny amount.
Here's a restatement that even more directly attacks the incorrect form of anthropic-principle argument: there are more ways to have a universe with an unexploded Easy Mode grenade than a universe with an unexploded Hard Mode grenade. (If you imagine that each grenade generates a random number from 0-99 when you pull the pin, then the Easy Mode universe could feature a grenade showing any number from 10 to 99, while the Hard Mode universe could only feature grenades showing numbers from 90 to 99.) So if you find yourself in a universe with an unexploded grenade, it's probably an Easy Mode grenade. Similarly, if you find yourself in a universe where it is the year 2024 and there has not been a catastrophic nuclear war, that should update you somewhat in the direction that catastrophic nuclear wars are unlikely.
These are good examples. (Sorta similar to the happy puppies, perhaps?) I'd like to find an example that is even simpler (and/or less contrived) but I just haven't been able to come up with one.
FWIW, the problem I was having was this: all of these thought experiments, around happy puppies and so forth, seem quite valid. Logically, they explain why you should update on evidence like "there hasn't yet been a nuclear war". But my *intuition* remained caught on the (incorrect) idea that an experiment with only one not-killing-you outcome doesn't reveal any new information: of course you saw the not-killing-you result! It's hard for a logical argument to overthrow an intuitive idea.
The multiple-universes framing helped me to directly attack that intuition, by pointing out that the distribution of observable universes in which I've pulled the pin on a grenade (and, of course, survived) is a subset of the observable universes in which I haven't yet pulled the pin, so pulling the pin does yield information (the information that I'm in that subset) which I didn't have prior to pulling it. And similarly with regard to the fact that we haven't (yet) had a nuclear war.
How does this sit with you? Maybe part of what's unintuitive is that updating on information that you only get to see in one universe seems "biased". And it *is* biased, if you want an estimate for the average over all universes. But if you need to make a decision based on the state of *your* universe, then that "bias" is a good thing.
That makes sense to me... but I am now thoroughly contaminated by just having spent an hour reading this post and thinking about all of this. :-)
Steve, thanks for adding your thoughts; they are extremely clarifying.
I've been subscribing to your blog since March and have read all the archives. Please keep writing (and I'll keep reading).
Enjoyed this but might add that it seems to me quite misleading to think of probabilities as inherent properties of the world; Probability is a conceptual tool we use to reason about the world. There isn't an underlying truth of "likelihood of nuclear war" though in certain cases it can be quite handy to think of it this way.
see e.g. https://metarationality.com/small-world
If you talk about "the probability that I weigh more than 70kg" that's a very different thing from the "probability that this U-235 atom will decay in the next 100 years". (They reflect "epistemic" vs "aleatory" uncertainty.) Some people don't like using the word "probability" for the first kind of probability but I find that to be a kind of semantic issue as long as everyone understands what's intended.
That said, I think these kinds of debates often disappear when you think about making decisions (or bets). For nuclear war, the question is how much resources to invest in preventing nuclear war or bomb shelters or whatever.
This is essentially the question raised (and discussed at length in the philosophy literature) by the sleeping beauty paradox.
In terms of pure decision theory for you personally, I think everything you said is fine. The problem is that's not the only way we use probability talk -- indeed I suspect that the way we use probability talk conflates two different ideas.
Basically the idea in sleeping beauty is that you are drugged into unconsciousness on Sunday and your captor flips a coin. If the coin is heads the captor wakes you up on Monday then wipes your mind and lets you go. If the coin is tails the captor wakes you up on Monday, wipes your memory and redrugs you and wakes you up again on Tuesday.
Suppose you wake up in the captor's possession one morning and the captor explains the situation (but not what day it is) what should your probability be the coin landed heads? Well it seems obvious it's 50% but now suppose you know the captor will offer you a $100 bet every time you wake about whether the coin landed tails at just below even odds.
Obviously you should take the bet because if the coin lands tails you will win twice but if it lands heads you lose once. That suggests you should assign probability of tails to be 2/3rds.
Ultimately, I think the confusion is over which of two questions one is answering (probability is just fancy counting). Are you trying to count up the situations from your perspective or from some kind of imagined 3rd party perspective?
To put it roughly in another way, there are an equal number of 'ways' the universe turns out where the captor gets a head and a tails. However, there are twice as many 'ways' you wake up in said situation with the coin tails as heads. So it's all just about what you are counting.
I think the point here is not about tightness of priors, but rather about whether a given observation is informative.
If you think that the only possible observation you can get is “Earth still exists”, then you might think that observing that Earth exist gives you no information, so you should not update your priors of yearly rate of Earth destruction, no matter how loose your priors were.
If, on the other hand, you think that the fact of observing Earth is itself contingent, then observing “Earth still exist” is informative, and you should update your priors (how much indeed depends on the tightness of your priors, but that is always the case).
I tend to agree with you observing the Earth existing is informative, and we should update. But probabilistic reasoning about events that only happen once are tricky.
For what it's worth, this is more or less the view that I wrote this post to try to refute! I mean, if an observation isn't information, sure you should update less. But if the *only* reason that an observation seems less information is because you might not exist, then I think the amount you should update depends 100% on your prior and 0% on the fact that you might not exist.
I think the post is a good one (I like the comming up with transparent examples), but I take issue with:
"The crux of the nuclear war scenario isn’t that you might stop existing. That’s just a very salient feature that draws our attention away from the heart of the debate: The confidence of your prior."
I think that part is wrong, and from your answer it seems you agree. Am I wrong? It also seems rather central to the post, so my objection doesn't seem nitpicking.
Also, "if an observation isn't information, sure you should update less.". I think in that case you should update nothing at all.
Fantasic explanation! This debate has always confused me. You really clarified it.