Keep in mind that "do nothing" is one of the options, so you can't really "not play".
But still, I agree with your broader point: If there's literally no test outcome that could make you change your mind and change any treatment, then you'd actually have that maxₐ 𝔼[utility(a,Y)] = 𝔼[ maxₐ 𝔼[utility(a,Y)｜X]], i.e. the value of information is zero. I suspect that in practice this is rarely true. However I'd think that often the value of information is very low, and in particular lower than the costs of the test itself. (Which, of course, you definitely want to account for when choosing to test or not.)
Thanks for writing this up! It’s definitely given me a lot to think about. My model of the doctors’ minds falls into two real reasons:
1. If you choose not to order a biopsy, or the like, because of base rate estimates, you are incurring a lot of risk. Even if you are entirely correct that it is low risk, I would very much expect that if you follow that pattern for long enough, you will have at least one patient for whom it is actually cancerous, and my understanding is that you could very easily be sued for malpractice in that case.
2. If you know the biopsy is going to be not worth running, or at least “know” to a good probability, before you even do the initial test, you really are wasting money running that test!
Oh my God. I just had this fight with an MD and NP. An old woman suddenly gets demented. Well, urinary tract infections can cause that. Oh, but we can't test old ladies for UTIs because turns out old ladies often have bacteria in their urine that isn't causing any harm. So then we overtreat old ladies, and overtreating is way worse, apparently, than treating the subset of old ladies who could be brought back out of dementia by getting their UTIs treated. My late mother tended to get demented with UTIs and get better after treatment, so I'm cranky about this trend. I'll probably get them too and no one will test and treat and I'll die demented because OTHER old ladies have asymptomatic bacteriuria.
Thank you for writing about this! I've also been very frustrated to hear about it.
I suspect it's largely explained by medical liability. If you're a doctor and you estimate that a weird nodule is probably not worth the painful and risky biopsy, you gain nothing when it's actually nothing, but you'll be sued for ignoring it in the cases where it turns out to be a problem. Scott Alexander has a post about the stories doctors tell each other about this kind of thing (he flags at the top of the post that this particular story is fishy, but that doesn't affect that doctors believe that this kind of thing happens): https://slatestarcodex.com/2014/06/23/court-ing-disaster/
This article, Overtreatment in the United States, describes a survey of doctors from the AMA, and notes that "At 85% of respondents, “fear of malpractice” was the top cited reason for overtreatment."
"I see this all the time in medicine. A patient goes to the hospital with a heart attack. While he’s recovering, he tells his doctor that he’s really upset about all of this. Any normal person would say “You had a heart attack, of course you’re upset, get over it.” But if his doctor says this, and then a year later he commits suicide for some unrelated reason, his family can sue the doctor for “not picking up the warning signs” and win several million dollars. So now the doctor consults a psychiatrist, who does an hour-long evaluation, charges the insurance company $500, and determines using her immense clinical expertise that the patient is upset because he just had a heart attack.
Those outside the field have *no idea* how much of medicine is built on this principle."
Yes this is the reason that jumped to mind. If there's no good reason to do a CT scan and the doctor doesn't do it, but later the patient dies from something that would have been found by a CT scan, the doctor's not going to lose a suit. Then there's other doctor who does a CT scan even though he has no good reason to do so, sees something that might be cancer, but ignores it because it's probably not cancer. That doctor is probably going to get sued. "He did the CT scan, he saw the cancer! But he just sent her home with pain medication! If he followed up like he should have, Mrs. Doe would still be alive today." sounds really good in court.
I feel I'm being foolish: I think the problem is that Big Eddie isn't a liar. And so we could be in a situation where it is rationnal for any given individual to do something that still shouldn't be done by the group of these individuals.
First, it could be that the test (CT scan, blood test...) was only studied among high risk people, which would make non obvious whether we could apply the same sensibility and specificity among low risk people.
Assuming it is the case, the risk of the biopsy would be, hopefully, fairly low. And both in high risk (obviously) and low risk people, the Positive Predictive Value could be fairly high considering the disease tested for. So in both case, one should opt for the biopsy when tested positive.
Yet, testing low risk people on a daily basis, *because* they would rationnaly opt for the biopsy when tested positive, would end up generating *in the population of low risk people* far more iatrogenic complications, undue psychological distress etc.. than would be rationnaly justified by the prevalence of the disease in this (very) huge group.
The obvious thing missing from your beautiful utility calculations is that besides the risks for the patient, there are also risks for the doctor; legal risks, reputational risks, etc., etc. Include them, and everything makes perfect sense (except American medicine itself, of course).
In the argument of the article, the test gives you more information with which to pick the least bad option more reliably.
The expected value doesn’t have to be positive in order to maximize it. If I’m losing a hand of poker, I want to lose the least I can.
Keep in mind that "do nothing" is one of the options, so you can't really "not play".
But still, I agree with your broader point: If there's literally no test outcome that could make you change your mind and change any treatment, then you'd actually have that maxₐ 𝔼[utility(a,Y)] = 𝔼[ maxₐ 𝔼[utility(a,Y)｜X]], i.e. the value of information is zero. I suspect that in practice this is rarely true. However I'd think that often the value of information is very low, and in particular lower than the costs of the test itself. (Which, of course, you definitely want to account for when choosing to test or not.)
Thanks for writing this up! It’s definitely given me a lot to think about. My model of the doctors’ minds falls into two real reasons:
1. If you choose not to order a biopsy, or the like, because of base rate estimates, you are incurring a lot of risk. Even if you are entirely correct that it is low risk, I would very much expect that if you follow that pattern for long enough, you will have at least one patient for whom it is actually cancerous, and my understanding is that you could very easily be sued for malpractice in that case.
2. If you know the biopsy is going to be not worth running, or at least “know” to a good probability, before you even do the initial test, you really are wasting money running that test!
Regardless, very much appreciate the post!
Oh my God. I just had this fight with an MD and NP. An old woman suddenly gets demented. Well, urinary tract infections can cause that. Oh, but we can't test old ladies for UTIs because turns out old ladies often have bacteria in their urine that isn't causing any harm. So then we overtreat old ladies, and overtreating is way worse, apparently, than treating the subset of old ladies who could be brought back out of dementia by getting their UTIs treated. My late mother tended to get demented with UTIs and get better after treatment, so I'm cranky about this trend. I'll probably get them too and no one will test and treat and I'll die demented because OTHER old ladies have asymptomatic bacteriuria.
Thank you for writing about this! I've also been very frustrated to hear about it.
I suspect it's largely explained by medical liability. If you're a doctor and you estimate that a weird nodule is probably not worth the painful and risky biopsy, you gain nothing when it's actually nothing, but you'll be sued for ignoring it in the cases where it turns out to be a problem. Scott Alexander has a post about the stories doctors tell each other about this kind of thing (he flags at the top of the post that this particular story is fishy, but that doesn't affect that doctors believe that this kind of thing happens): https://slatestarcodex.com/2014/06/23/court-ing-disaster/
This article, Overtreatment in the United States, describes a survey of doctors from the AMA, and notes that "At 85% of respondents, “fear of malpractice” was the top cited reason for overtreatment."
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5587107/
I just found a Scott Alexander post talking about this more directly: https://slatestarcodex.com/2017/02/09/considerations-on-cost-disease/
"I see this all the time in medicine. A patient goes to the hospital with a heart attack. While he’s recovering, he tells his doctor that he’s really upset about all of this. Any normal person would say “You had a heart attack, of course you’re upset, get over it.” But if his doctor says this, and then a year later he commits suicide for some unrelated reason, his family can sue the doctor for “not picking up the warning signs” and win several million dollars. So now the doctor consults a psychiatrist, who does an hour-long evaluation, charges the insurance company $500, and determines using her immense clinical expertise that the patient is upset because he just had a heart attack.
Those outside the field have *no idea* how much of medicine is built on this principle."
Yep, it's known as "defensive medicine".
Yes this is the reason that jumped to mind. If there's no good reason to do a CT scan and the doctor doesn't do it, but later the patient dies from something that would have been found by a CT scan, the doctor's not going to lose a suit. Then there's other doctor who does a CT scan even though he has no good reason to do so, sees something that might be cancer, but ignores it because it's probably not cancer. That doctor is probably going to get sued. "He did the CT scan, he saw the cancer! But he just sent her home with pain medication! If he followed up like he should have, Mrs. Doe would still be alive today." sounds really good in court.
I feel I'm being foolish: I think the problem is that Big Eddie isn't a liar. And so we could be in a situation where it is rationnal for any given individual to do something that still shouldn't be done by the group of these individuals.
First, it could be that the test (CT scan, blood test...) was only studied among high risk people, which would make non obvious whether we could apply the same sensibility and specificity among low risk people.
Assuming it is the case, the risk of the biopsy would be, hopefully, fairly low. And both in high risk (obviously) and low risk people, the Positive Predictive Value could be fairly high considering the disease tested for. So in both case, one should opt for the biopsy when tested positive.
Yet, testing low risk people on a daily basis, *because* they would rationnaly opt for the biopsy when tested positive, would end up generating *in the population of low risk people* far more iatrogenic complications, undue psychological distress etc.. than would be rationnaly justified by the prevalence of the disease in this (very) huge group.
The obvious thing missing from your beautiful utility calculations is that besides the risks for the patient, there are also risks for the doctor; legal risks, reputational risks, etc., etc. Include them, and everything makes perfect sense (except American medicine itself, of course).