Discussion about this post

User's avatar
Steve Newman's avatar

> Does it matter how information gets to you? For example, say a paper comes out that tries yelling at kids in school and finds that yelling is not effective at making them learn faster. You might worry: If the experiment had found that yelling was effective, would it have been published? Would anyone dare to talk about it?

> How much should you downweight evidence like this, where the outcome changes the odds you’d see it?

> If you’re a true believer fundamentalist Bayesian, the answer is: None. You ignore all those concerns and give the evidence full weight. At a high level, that’s because Bayesianism is all about probabilities in this world, so what could have happened in some other world doesn’t matter.

I'm confused by this: if you receive information via a channel that you know applies selective filtering, your knowledge of the filtering *must* affect how you update, mustn't it?

I'm not practiced in formal Bayesian reasoning, so I can't express this in the appropriate technical language; with that caveat:

Suppose I believe that the New York Times is selective in their reporting of natural disasters, they are more likely to report disasters in some countries and less likely in others. They report on a disaster in Japan. I can 100% update that there is, in fact, a disaster in Japan.

Then I open my morning edition of Weird School Experiments Daily and I see the report of a finding that yelling at kids is not effective. It's just one study, so even setting aside my concerns regarding reporting bias, I can't fully trust it. I should update somewhat in the direction of believing that yelling at kids is not effective to make them learn faster.

How much should I update? Well, imagine a world where:

- This particular question is the subject of 10 studies each year.

- The reality is that yelling is *slightly* effective, such that any given study has a 50% chance of concluding it is effective.

- My prior happened to be precisely correct – I believed that yelling is slightly effective.

If I update on published studies, and only the studies which find that yelling is ineffective are published, then 5 times each year I'll update somewhat in the direction of yelling being ineffective – pushing me away from the truth. If I update on published studies, and all of the studies are published, then I'll update back and forth but my belief will tend to stay in the vicinity of the truth.

So it seems that it is unambiguously incorrect to ignore known (or strongly suspected) selective filtering in an information channel? If I know about the selective reporting of yelling studies, and I see a study finding that yelling is ineffective, it seems like I could respond to this in one of two ways:

A) Ignore it. It contains zero bits of information, because I know a priori that all studies I read on this subject will contain this conclusion.

B) Perform some complicated analysis of how many studies I suspect are conducted on this topic, how many studies I see published, and what the actual effectiveness of yelling must be to result in the observed number of stories being published.

All of this ignores the fact that when we require a study to find that yelling is ineffective, we have not fully constrained the contents of that study – it might find that the effect size was almost large enough to be significantly significant, or it might find a much smaller (or even negative) effect size; there are all the details of how the study was conducted (e.g. sample size), etc. So I could get more sophisticated than B. But I think the point stands that knowledge of the properties of the information channel is important.

My presumption is that you could fit all this into formal Bayesian reasoning if you expand your analysis to include priors around what experiments are performed and which of those will be published. But also that your head would explode.

Expand full comment
Adam Mastroianni's avatar

#4 strikes me as an exercise in epicycle-fitting, but I mean that in a descriptive way rather than a disparaging way. One clue we’re still dealing with geocentric psychology here is that the taxonomy is based on symptoms rather than causes. Imagine doing this for physical diseases instead—if you get really good at measuring coughing, sneezing, aching, wheezing, etc. you may ultimately get pretty good at distinguishing between, say, colds and flus. But you’d have a pretty hard time distinguishing between flu and covid, and you’d have no chance of ever developing vaccines for them, because you have no concept of the systems that produce the symptoms.

I think approaches like this, which administer questionnaires and then try to squeeze statistics out of them, are going to top out at that level. They’ll probably make us better at treating mental disorders, but not much better.

Expand full comment
26 more comments...

No posts