I guess because if you just told me "there is a universe with star systems and planets" but didn't tell me anything about Earth or what we see here, I'd think "sure. intelligent species might travel to planets with other intelligent species." However given what we actually see here on Earth, I don't think it's likely that aliens are here.
I don’t think the problem you identify is with priors, but rather with estimating probabilities, particularly of unlikely issues.
In your example,
P(data|aliens)=P(data, weird aliens| aliens) + P(data, normal aliens | aliens) = P(data|weird aliens) P(weird aliens|aliens) + P(data|normal aliens) P(normal aliens|aliens), and if useful, you might want to consider other sub- conditions.
But this happens also without priors. For example:
P(Musk is next US president) = P(Musk is next president, Xai achieves AGi and performs a massive manipulation campaign) + P(Musk is next president, Xai achieves AGi does some other weird thing) + P(Musk is next president, Xai does not achieve AGi and weird thing B happens) + …
Again infinite possible things you could think about. Clearly, u should focus’s on the most likely ones and the ones most like to be linked to Musk becoming president, but nothing about priors here.
It is mostly consistent (the math expandy didn't appear in substack), but I can take issue with "the reason it’s lower is that I have additional prior information that I forgot to include in my original prior." I'd say you didn't forget anything in the prior, it was fine, but when computing P[data | aliens], you naturally need to look at the data and consider what sub-conditions consistent with the alternatives in the prior are likely to produce the data. So I dont get why in the next paragraph you write that this procedure is "very unintuitive". Estimating probabilities is often hard, particularly for unlikely events (see example above).
An issue with "building a prior after looking at the data" is the risk of changing it in whichever way you want to discount the data.
So I'd say, leve the prior alone but be careful when estimating the likelihood.
What I was trying to say (and maybe it's not very clear—I'm open to revising) is that any route you take to fixing model 1 (either by fixing the likelihood as you suggest or building model 2 as I prefer (though I agree you get the same number in the end)) you're forced to come up with an estimate for P[weird aliens | aliens] and P[weird people | no aliens], etc.
I'd just add that I don't think you need to fix the likelihood, just estimate it properly. To estimate P(data | aliens) you quite naturally need to think about weird aliens, similarly to how in order to estimate P(Musk is next pressident) you need to think about weird events which could lead to that.
I totally agree that ultimately that is the mistake in model 1. My estimate of P[data | aliens] was bad because I neglected to think about weird aliens.
If I wanted to retreat to a motte, I think my thesis in this post would just be that
(1) in real problems, you almost always end up needing to estimate stuff like P[weird aliens | aliens]
(2) in real problems, it's extremely difficult to predict what those types of things will be before you've looked at the data
My bailey would be all that plus:
(3) P[weird aliens | aliens] does not involve the data and so should be semantically classified as part of the "prior"
I think you agree on (1) and possibly agree on (2) but perhaps don't agree on (3)?
I would't object too much to (1) and (2), though I have the sense that (1) applies mostly for unusual hypothesis or data.
As to (3), I would push back about "should", but I'd be less squeamish if substituted by "could".
I think that, conceptually, keeping the principle "you can write the prior before seeing the data" is better than losing it but getting "everythin that does not depend on the data goes to the prior" (or perhaps more technically but still probably hard to formalize, "you cannot decompose the likelihood in elements, some of which don't depend on the data"). A matter of taste, maybe.
Anyway, its fun to think about these issues, thanks!
I think the big thing you're missing is that if you do a motivated search for something and try very hard to find evidence of it, and you come up blank, you should decrease your probability in that thing at least a little (put another way, absence of evidence is clearly some evidence of absence).
(But otherwise I agree with a lot of it, humans are not logically omniscient)
if you do a motivated search for something and try very hard to find evidence of it, and you come up blank, this shoul appear in your likelihood. P(data mostly null after hard work looking for aliens | aliens) is presumably low
Excellent post. Setting up good Bayesian priors (that are actually prior!) has always intuitively seemed basically impossible to me for real-world situations and this helps me understand why.
Speaking of which, Manifold.markets gives a 7% chance aliens are real (not really; it's a known problem with prediction markets that they fail to reflect true probabilities that are close to 0% or 100%, particularly with long time horizons):
There's something hilarious about him making a bet at 150:1 odds, and then a prediction market converging to 13:1 odds of him winning that bet.
Personally, I'd take the same side as he did with that bet. Though I must admit that without the precise conditions (past UFO sightings, within 5 years) I'd have to think long and hard. So I guess I don't really rate the odds that much smaller than 1%?
As long as we agree it's something under 1% for the green men scenario it may not be worth the effort to work out how much less than 1%. The trajectory that AI is on means more than 10 times that probability of something equally world-upside-down-turning, right?
I guess I'd give most of that 1% to some scenario where aliens were here (or some kind of alien probes/drones) but for whatever reason they leave no evidence at all of their existence?
Yeah, and in that case does it even matter how high we go for that probability? It reminds me of the Simulation Hypothesis. Aliens were here but they're gone now and left zero trace. Feels like a big deal but there's nothing to actually do with it.
But now that I say that, I think Robin Hanson has in the past bitten the simulation hypothesis bullet as well and concluded that it's important to be as interesting as possible so the simulators don't turn you off. (I'm actually impressed with Robin's ability to take ideas seriously and follow them relentlessly wherever they lead. Maybe sometimes it leads right off the rails but that's better than the other extreme of never entertaining any idea that feels too weird.)
One simple solution is to divide the data into two groups: one for figuring out which categorization makes the most sense, and the other for calculating the posterior.
I guess it is with Bayesianism as it is with Utilitarianism: technically correct as a description, but completely (and will forever be) useless in practice.
Ah thank you for those references, I was not familiar with the author.
Interesting that he writes: "there is only subjective experience", I actually named my substack "Objective Observation" as a kind of pun -- after going through something like the same "losing my religion" he seems to describe and realizing "Subjective Experience" is foundational to everything I believe in, be it life, science, rationality, compassion, whatever.
So utilitarianism is something I hold very lightly at this point. However, that being said, actually not for the reason he seems to suggest: that no amount of stubbed toes can be worse than a single person suffering immensely. That's not so much a point against utilitarianism as much as it's saying that maximizing "average utility" is wrong, which is more a question of how to aggregate utility more than saying it can't be done.
Also I would like to mention that I think he assumes way too much about how consciousness works, and I would advocate more caution. Let's not harm enormous amounts of creatures for very little gain to ourselves, on the hope that they don't suffer.
So I pondered this a little bit after writing my previous comment, and although I don't fully disagree with anything you write, I will accuse you of some form of utilitarianism.
As a non-utilitarian, I say "let's not harm a bunch of tiny, defenseless creatures for no reason, that is not _right_".
And your response is "well but to avoid this supposed harm to many small creatures, we have to not do other interventions which are higher impact, it's a tradeoff you see, and we are more certain about other forms of bigger harms happening, so we should be prioritizing those". Yeah, I guess I agree, but that's a _utilitarian_ argument! Your argument collapses straight back into utilitarianism, it's not that you're not a utilitarian, you're just saying they got the maths wrong.
Also somewhat unrelated to this complaint, I also think you're overstating the case that we know bugs or whatever aren't suffering immensely. It's strikingly similar to people who don't want to be vegetarian,'s assertions that pigs in factory farms don't suffer, because animals aren't conscious, or can't suffer, or whatever. That being said, I also happen to not care that much about bugs or whatever, so I guess we agree in that sense.
Another way to see the same thing is your complaint about x-risk. As a non-utilitarian I think it's good and right and smart to do small efforts that lower the risk that the human race goes permanently extinct. The utilitarians seem to agree, but they also give a model for knowing "how much" effort it's reasonable to expend in that direction. I get the impression you think that number i zero, which doesn't really make any sense, and I'm not sure how you square that?
I don't know anyone who *really* thinks that pigs can't suffer. Over the decades, I've had people tell me they were just taking a position / trying to get a rise out of me when they said that (or things like it).
Re: it being a utilitarian argument, part 2 here might touch on that
Okay, the only thing I really disagree with is I don't think your take on consciousness takes its mystery seriously enough. I think you're a little bit muddled about panpsychism, evolution and robots. You sneer at panpsychism, and I do too, but you also believe evolution "created" consciousness while at the same time robots are not conscious. This is confused. I think you will agree that evolution can't make something that's not available in our universe, or accordance with its physical laws, or something to that effect. So people sometimes say, maybe consciousness emerged. Whatever, okay, but if that's possible, why can't it also emerge for robots? (Both you and Yudkowsky make the point that the fact that "consciousness" as a concept is part of the training corpus, makes it completely impossible to know for LLMS, which I agree with and is highly annoying.) But my point is more that you seem all too certain that robots can't be or at least aren't now conscious, and I just think this is a slightly magical way of thinking.
Another way of thinking about this is that, since suffering is possible in our universe, it is *in some sense* fundamental. Is it made out of non-suffering parts? Maybe, but it's hard to see how. I agree with your emphatic assertion that subjective experience is the only thing that matters. But to say that it "emerges" from complexity or whatever, is a completely nonsensical statement.
That's not exactly my view, but I think it's fair to view this as part of why Bayesianism isn't used more in practice. It also occurs to me (now...) that some people think about similar issues and then instead of interpreting this as evidence that you need to be careful, they conclude that Bayesianism is just wrong/bad. For example https://www.mindthefuture.info/p/why-im-not-a-bayesian
Yes, I like Richard's take and think the conclusion basically is that it's completely untenable in practice. (As a prescriptive thing, even though descriptively it's correct and elegant and alluring.) I'm not sure why your own reasoning (in this very post) doesn't lead you to the same conclusion?
I'd be curious about your thoughts on In the Cells of the Eggplant, it's basically entirely about this.
Overall, I think I'd endorse the pragmatic attitude it seems to endorse towards Bayesianism, and more rationality more generally. In terms of this list of claims
The first part of Richard's argument (fuzzy logic), is I think equivalent to the main thesis of The Eggplant: nebulosity, which I think is also what you kind of stumble upon in this post.
Most of the real world is so nebulous that for any given definition of a phenomenon, there are counter examples which don't fit and also counter examples which shouldn't be included but are. This goes back all the way to the ancient greeks I think? I'm not saying this is some revolutionary new insight, but properly understood I believe it does not only explain why bayesianism is not widely used, but why it never can and never will be. It is just not going to ever be useful "in the real world". (I don't mean this in a disparaging way, I think Bayes' is pretty cool.)
I agree in a certain sense that Bayes is good for defining and thinking about uncertainty (#3), but there is also another, more important sense, in which it doesn't help you with uncertainty: is there water in the fridge? The uncertainty is mostly not in whether I remember what my fridge contains, or whether someone else put water in my fridge since last time I looked (which is how Bayes comes in), but rather in whether the statement itself refers to "a glass or bottle of water" (in which case, no) or "the water in the cells of the eggplant" (in which case, depends on if there's any food in the fridge), or even "suspended as humidity in the air" (in which case, depends on whether my fridge is in a vacuum I guess?).
The point is, what the question exactly means is equivalent to your observation that it's an infinitely dimensional space with arbitrarily high resolution, and how you want to carve it (impose categories) depends crucially on the data itself. So your prior becomes meaningless and Bayes collapses.
I'm not sure why I'm arguing against you here, I completely agree with everything in your post. I just don't think there's anyway to "be careful" or otherwise salvage Bayes as a prescriptive method for most real world stuff?
I'm still working my way through the book myself, but my impression is that its central thesis really is taking this inherent nebulosity to its logical conclusion. It undermines a lot more than just Bayes.
Yeah, I think that the primary value of Bayesianism is as a sort of Platonic ideal to admire: "Here's what would be the exact optimal way to reason if this list of extremely difficult assumptions were all satisfied."
That said, I personally think it is useful in practice at least sometimes! For example, I find it useful for thinking about the probability that we are actually observing alien aircraft. I also think we should also acknowledge that there are problems where Bayesian solutions are basically acknowledged to work the best in practice, e.g. election forecasting.
Edit: I mean best in practice compared to other formal methods. As far as I can tell, the true best method for predicting elections is still to ask a human who has a good track record at predicting elections...
I wonder if the world isn't so nebulous though, that the categories or dimensions of hypotheses you should be considering, actually become fractal in nature. In which case I'm not sure even the platonic ideal holds?
In practice I think things like "testing positive for a rare disease, on a test that is only 90% reliable, so you should not really be _that_ worried you really are sick" is a great use of Bayes in the real world. But it might be better known as Simpsons Paradox, and really only amounts to multiplying literally two numbers. If you get any more uncertain about what the prior "should" be, or what counts as evidence (or what its likelihood given the hypothesis should be), than that it rapidly disintegrates.
So I remain skeptical that it is possible even in principle to salvage e.g. election forecasting using Bayes -- not because I don't believe in the maths, but because the hypothesis space is undifferentiable (in the sense of a fractal function in calculus, which inherently does not have a well defined derivative).
I shared the article with some folks that don't understand why you find the prior 50-50 okay, but the posterior 50-50 surprising / obviously wrong
I guess because if you just told me "there is a universe with star systems and planets" but didn't tell me anything about Earth or what we see here, I'd think "sure. intelligent species might travel to planets with other intelligent species." However given what we actually see here on Earth, I don't think it's likely that aliens are here.
I don’t think the problem you identify is with priors, but rather with estimating probabilities, particularly of unlikely issues.
In your example,
P(data|aliens)=P(data, weird aliens| aliens) + P(data, normal aliens | aliens) = P(data|weird aliens) P(weird aliens|aliens) + P(data|normal aliens) P(normal aliens|aliens), and if useful, you might want to consider other sub- conditions.
But this happens also without priors. For example:
P(Musk is next US president) = P(Musk is next president, Xai achieves AGi and performs a massive manipulation campaign) + P(Musk is next president, Xai achieves AGi does some other weird thing) + P(Musk is next president, Xai does not achieve AGi and weird thing B happens) + …
Again infinite possible things you could think about. Clearly, u should focus’s on the most likely ones and the ones most like to be linked to Musk becoming president, but nothing about priors here.
Is this consistent with this paragraph (and the math expandy box after)? https://dynomight.net/prior/#:~:text=Technically%2C%20the%20fix%20to%20the%20first%20model
It is mostly consistent (the math expandy didn't appear in substack), but I can take issue with "the reason it’s lower is that I have additional prior information that I forgot to include in my original prior." I'd say you didn't forget anything in the prior, it was fine, but when computing P[data | aliens], you naturally need to look at the data and consider what sub-conditions consistent with the alternatives in the prior are likely to produce the data. So I dont get why in the next paragraph you write that this procedure is "very unintuitive". Estimating probabilities is often hard, particularly for unlikely events (see example above).
An issue with "building a prior after looking at the data" is the risk of changing it in whichever way you want to discount the data.
So I'd say, leve the prior alone but be careful when estimating the likelihood.
What I was trying to say (and maybe it's not very clear—I'm open to revising) is that any route you take to fixing model 1 (either by fixing the likelihood as you suggest or building model 2 as I prefer (though I agree you get the same number in the end)) you're forced to come up with an estimate for P[weird aliens | aliens] and P[weird people | no aliens], etc.
Cool, thanks for engaging.
I'd just add that I don't think you need to fix the likelihood, just estimate it properly. To estimate P(data | aliens) you quite naturally need to think about weird aliens, similarly to how in order to estimate P(Musk is next pressident) you need to think about weird events which could lead to that.
I totally agree that ultimately that is the mistake in model 1. My estimate of P[data | aliens] was bad because I neglected to think about weird aliens.
If I wanted to retreat to a motte, I think my thesis in this post would just be that
(1) in real problems, you almost always end up needing to estimate stuff like P[weird aliens | aliens]
(2) in real problems, it's extremely difficult to predict what those types of things will be before you've looked at the data
My bailey would be all that plus:
(3) P[weird aliens | aliens] does not involve the data and so should be semantically classified as part of the "prior"
I think you agree on (1) and possibly agree on (2) but perhaps don't agree on (3)?
I find this a nice way to put it.
I would't object too much to (1) and (2), though I have the sense that (1) applies mostly for unusual hypothesis or data.
As to (3), I would push back about "should", but I'd be less squeamish if substituted by "could".
I think that, conceptually, keeping the principle "you can write the prior before seeing the data" is better than losing it but getting "everythin that does not depend on the data goes to the prior" (or perhaps more technically but still probably hard to formalize, "you cannot decompose the likelihood in elements, some of which don't depend on the data"). A matter of taste, maybe.
Anyway, its fun to think about these issues, thanks!
I think the big thing you're missing is that if you do a motivated search for something and try very hard to find evidence of it, and you come up blank, you should decrease your probability in that thing at least a little (put another way, absence of evidence is clearly some evidence of absence).
(But otherwise I agree with a lot of it, humans are not logically omniscient)
if you do a motivated search for something and try very hard to find evidence of it, and you come up blank, this shoul appear in your likelihood. P(data mostly null after hard work looking for aliens | aliens) is presumably low
Excellent post. Setting up good Bayesian priors (that are actually prior!) has always intuitively seemed basically impossible to me for real-world situations and this helps me understand why.
Hear hear!
Meanwhile we have Robin Hanson chomping on bullets like breakfast cereal: https://www.overcomingbias.com/p/us-war-depts-big-ufo-lie
Speaking of which, Manifold.markets gives a 7% chance aliens are real (not really; it's a known problem with prediction markets that they fail to reflect true probabilities that are close to 0% or 100%, particularly with long time horizons):
https://manifold.markets/Joshua/will-eliezer-yudkowsky-win-his-1500
There's something hilarious about him making a bet at 150:1 odds, and then a prediction market converging to 13:1 odds of him winning that bet.
Personally, I'd take the same side as he did with that bet. Though I must admit that without the precise conditions (past UFO sightings, within 5 years) I'd have to think long and hard. So I guess I don't really rate the odds that much smaller than 1%?
As long as we agree it's something under 1% for the green men scenario it may not be worth the effort to work out how much less than 1%. The trajectory that AI is on means more than 10 times that probability of something equally world-upside-down-turning, right?
I guess I'd give most of that 1% to some scenario where aliens were here (or some kind of alien probes/drones) but for whatever reason they leave no evidence at all of their existence?
Yeah, and in that case does it even matter how high we go for that probability? It reminds me of the Simulation Hypothesis. Aliens were here but they're gone now and left zero trace. Feels like a big deal but there's nothing to actually do with it.
But now that I say that, I think Robin Hanson has in the past bitten the simulation hypothesis bullet as well and concluded that it's important to be as interesting as possible so the simulators don't turn you off. (I'm actually impressed with Robin's ability to take ideas seriously and follow them relentlessly wherever they lead. Maybe sometimes it leads right off the rails but that's better than the other extreme of never entertaining any idea that feels too weird.)
One simple solution is to divide the data into two groups: one for figuring out which categorization makes the most sense, and the other for calculating the posterior.
I guess it is with Bayesianism as it is with Utilitarianism: technically correct as a description, but completely (and will forever be) useless in practice.
I'm totally down with this observation about Bayesianism
https://www.mattball.org/2025/10/this-is-neither-rational-nor-objective.html
(For the SMBC cartoon link)
But not that Utilitarianism is technically correct
https://www.mattball.org/2024/08/utilitarianism-hurts-case-study.html
Ah thank you for those references, I was not familiar with the author.
Interesting that he writes: "there is only subjective experience", I actually named my substack "Objective Observation" as a kind of pun -- after going through something like the same "losing my religion" he seems to describe and realizing "Subjective Experience" is foundational to everything I believe in, be it life, science, rationality, compassion, whatever.
So utilitarianism is something I hold very lightly at this point. However, that being said, actually not for the reason he seems to suggest: that no amount of stubbed toes can be worse than a single person suffering immensely. That's not so much a point against utilitarianism as much as it's saying that maximizing "average utility" is wrong, which is more a question of how to aggregate utility more than saying it can't be done.
Also I would like to mention that I think he assumes way too much about how consciousness works, and I would advocate more caution. Let's not harm enormous amounts of creatures for very little gain to ourselves, on the hope that they don't suffer.
TY.
I would say it isn't hoping certain creatures don't suffer, but rather, focusing our efforts on actual, known suffering, rather than "expected values." https://www.mattball.org/2025/12/a-meaningful-life-for-super-smart-part-3.html
Oh you _are_ the author, I did not notice!
So I pondered this a little bit after writing my previous comment, and although I don't fully disagree with anything you write, I will accuse you of some form of utilitarianism.
As a non-utilitarian, I say "let's not harm a bunch of tiny, defenseless creatures for no reason, that is not _right_".
And your response is "well but to avoid this supposed harm to many small creatures, we have to not do other interventions which are higher impact, it's a tradeoff you see, and we are more certain about other forms of bigger harms happening, so we should be prioritizing those". Yeah, I guess I agree, but that's a _utilitarian_ argument! Your argument collapses straight back into utilitarianism, it's not that you're not a utilitarian, you're just saying they got the maths wrong.
Also somewhat unrelated to this complaint, I also think you're overstating the case that we know bugs or whatever aren't suffering immensely. It's strikingly similar to people who don't want to be vegetarian,'s assertions that pigs in factory farms don't suffer, because animals aren't conscious, or can't suffer, or whatever. That being said, I also happen to not care that much about bugs or whatever, so I guess we agree in that sense.
Another way to see the same thing is your complaint about x-risk. As a non-utilitarian I think it's good and right and smart to do small efforts that lower the risk that the human race goes permanently extinct. The utilitarians seem to agree, but they also give a model for knowing "how much" effort it's reasonable to expend in that direction. I get the impression you think that number i zero, which doesn't really make any sense, and I'm not sure how you square that?
Thanks for this.
I don't know anyone who *really* thinks that pigs can't suffer. Over the decades, I've had people tell me they were just taking a position / trying to get a rise out of me when they said that (or things like it).
Re: it being a utilitarian argument, part 2 here might touch on that
https://mattball.substack.com/p/what-is-consciousness-where-i-part
Although the chapter Biting the Philosophical Bullet in Losing My Religions is more thorough.
Re: X-risks, this is why you can't have me in polite society:
https://www.mattball.org/2024/04/i-welcome-our-robot-overlords-you.html
(Also a chapter in Losing.)
Okay, the only thing I really disagree with is I don't think your take on consciousness takes its mystery seriously enough. I think you're a little bit muddled about panpsychism, evolution and robots. You sneer at panpsychism, and I do too, but you also believe evolution "created" consciousness while at the same time robots are not conscious. This is confused. I think you will agree that evolution can't make something that's not available in our universe, or accordance with its physical laws, or something to that effect. So people sometimes say, maybe consciousness emerged. Whatever, okay, but if that's possible, why can't it also emerge for robots? (Both you and Yudkowsky make the point that the fact that "consciousness" as a concept is part of the training corpus, makes it completely impossible to know for LLMS, which I agree with and is highly annoying.) But my point is more that you seem all too certain that robots can't be or at least aren't now conscious, and I just think this is a slightly magical way of thinking.
Another way of thinking about this is that, since suffering is possible in our universe, it is *in some sense* fundamental. Is it made out of non-suffering parts? Maybe, but it's hard to see how. I agree with your emphatic assertion that subjective experience is the only thing that matters. But to say that it "emerges" from complexity or whatever, is a completely nonsensical statement.
That's not exactly my view, but I think it's fair to view this as part of why Bayesianism isn't used more in practice. It also occurs to me (now...) that some people think about similar issues and then instead of interpreting this as evidence that you need to be careful, they conclude that Bayesianism is just wrong/bad. For example https://www.mindthefuture.info/p/why-im-not-a-bayesian
Yes, I like Richard's take and think the conclusion basically is that it's completely untenable in practice. (As a prescriptive thing, even though descriptively it's correct and elegant and alluring.) I'm not sure why your own reasoning (in this very post) doesn't lead you to the same conclusion?
I'd be curious about your thoughts on In the Cells of the Eggplant, it's basically entirely about this.
I haven't read it in detail. The most related sections seem to be these?
https://metarationality.com/probability-limitations
https://metarationality.com/probability-limitations
Overall, I think I'd endorse the pragmatic attitude it seems to endorse towards Bayesianism, and more rationality more generally. In terms of this list of claims
https://metarationality.com/probabilism
I'd endorse #1 here practically, I'd sorta-kinda endorse #3 in theory. (It's only complete and correct for one kind of uncertainty.)
The first part of Richard's argument (fuzzy logic), is I think equivalent to the main thesis of The Eggplant: nebulosity, which I think is also what you kind of stumble upon in this post.
Most of the real world is so nebulous that for any given definition of a phenomenon, there are counter examples which don't fit and also counter examples which shouldn't be included but are. This goes back all the way to the ancient greeks I think? I'm not saying this is some revolutionary new insight, but properly understood I believe it does not only explain why bayesianism is not widely used, but why it never can and never will be. It is just not going to ever be useful "in the real world". (I don't mean this in a disparaging way, I think Bayes' is pretty cool.)
I agree in a certain sense that Bayes is good for defining and thinking about uncertainty (#3), but there is also another, more important sense, in which it doesn't help you with uncertainty: is there water in the fridge? The uncertainty is mostly not in whether I remember what my fridge contains, or whether someone else put water in my fridge since last time I looked (which is how Bayes comes in), but rather in whether the statement itself refers to "a glass or bottle of water" (in which case, no) or "the water in the cells of the eggplant" (in which case, depends on if there's any food in the fridge), or even "suspended as humidity in the air" (in which case, depends on whether my fridge is in a vacuum I guess?).
The point is, what the question exactly means is equivalent to your observation that it's an infinitely dimensional space with arbitrarily high resolution, and how you want to carve it (impose categories) depends crucially on the data itself. So your prior becomes meaningless and Bayes collapses.
I'm not sure why I'm arguing against you here, I completely agree with everything in your post. I just don't think there's anyway to "be careful" or otherwise salvage Bayes as a prescriptive method for most real world stuff?
I'm still working my way through the book myself, but my impression is that its central thesis really is taking this inherent nebulosity to its logical conclusion. It undermines a lot more than just Bayes.
(Btw you posted the same link twice)
Yeah, I think that the primary value of Bayesianism is as a sort of Platonic ideal to admire: "Here's what would be the exact optimal way to reason if this list of extremely difficult assumptions were all satisfied."
That said, I personally think it is useful in practice at least sometimes! For example, I find it useful for thinking about the probability that we are actually observing alien aircraft. I also think we should also acknowledge that there are problems where Bayesian solutions are basically acknowledged to work the best in practice, e.g. election forecasting.
Edit: I mean best in practice compared to other formal methods. As far as I can tell, the true best method for predicting elections is still to ask a human who has a good track record at predicting elections...
I do like the idea of a platonic ideal.
I wonder if the world isn't so nebulous though, that the categories or dimensions of hypotheses you should be considering, actually become fractal in nature. In which case I'm not sure even the platonic ideal holds?
In practice I think things like "testing positive for a rare disease, on a test that is only 90% reliable, so you should not really be _that_ worried you really are sick" is a great use of Bayes in the real world. But it might be better known as Simpsons Paradox, and really only amounts to multiplying literally two numbers. If you get any more uncertain about what the prior "should" be, or what counts as evidence (or what its likelihood given the hypothesis should be), than that it rapidly disintegrates.
So I remain skeptical that it is possible even in principle to salvage e.g. election forecasting using Bayes -- not because I don't believe in the maths, but because the hypothesis space is undifferentiable (in the sense of a fractal function in calculus, which inherently does not have a well defined derivative).