Parenting feels like one big giant emotional investment in diminishing returns - i.e. every parent willing to go gonzo for whatever small/insignificant effect sizes they can get. esp on things like iq/health/etc. Especially in those first born, early years with all of your best intentions. I wonder what belief updating around these practices look like for second+ born kids?
Also there's a lot to be said around the decision making framework around the results of statistical testing, etc - but no need to go down that fiery road to hell today.
Minor point: you use the phrase "if you can breastfeed" a few times. This is not quite the right way to think about the issue, because whether you "can" breastfeed isn't binary. Most mothers "can" breastfeed to some extent, but it imposes large--but finite--but only partially quantifiable costs. Examples include:
-- Serious nipple pain and injury, including bloody blisters, for the first 6ish weeks (eventually your nipple skin adjusts to the experience, but it takes a while)
-- Having to hide somewhere at work to pump several times a day, for 20-30 minutes at a time, instead of interacting with coworkers like a normal person
-- Having to plan all outings and socialization around opportunities to nurse and/or pump, because it's still awkward to do it in public
-- Never-ending storage/transport/supply logistics for pumped milk
These are only the ones I personally experienced. So "breastfeed if you can" is not an answer to the right question. The right question is whether, for a given person, breastfeeding passes a cost-benefit analysis. This may be very hard to determine even if you succeed in quantifying the benefits!
(I ended up happily sticking with breastfeeding and not doubting my decision much. From this, you can extrapolate how much higher the costs probably are for someone who is really on the fence.)
Totally agree. For what it's worth, this is also how I think about it! I think I just shortened it to "can" because I feel like these tradeoffs are so personal, and I don't want to come across like I'm telling anyone else how to navigate them. (I just want to understand what tradeoff is being made!)
Not sure if there's an easy way to add this kind of nuance without a long digression. Maybe I could just throw in a "...without undue difficulty" with some of them mentions? Even that sounds a little more prescriptive than I'd like. But if I sound prescriptive already, then OK!
This also raises the issue of whether dispensing stored human milk via bottle is equivalent to breastfeeding. It may be that the main advantage of breastfeeding is physical rather than chemical.
Certainly questions of IQ ten or twenty years out beg the question of whether any behavioral benefits of breastfeeding are short or long term.
I'm wondering whether the science of formula-making could possibly have improved. Are the formulas used today different from those used in Belarus in 1996?
Formula has definitely evolved. I'm not exactly sure what formula was common in Belarus in 1996. But in terms of formula that's common in the West, I've found at least three major changes since that time:
1. Addition of docosahexaenoic acid
2. Addition of arachidonic acid
3. Addition of extra nucleotides
I'm not sure if there's rigorous proof that these help, but it's certainly plausible, and I think this is a valid reason to suspect that the benefits of breastfeeding over formula may be smaller now.
(Of course, it all depends on which mechanism you think is most plausible. If it's bio-active stuff like antibodies, then this might not make much of a difference.)
I think the "hospitals faking data" thing deserves more attention. In the Belarus RCT, one hospital was discovered to be faking data, and later another hospital was discovered to be faking data. How many other hospitals faked their data but didn't get caught? This makes me wonder if the original results of the study can be trusted at all.
(How did the known faked data change the results? Maybe faked data makes the results noisier and thus the study will underestimate the true effect size. But there could be some bias in how people faked the data that produces a spurious effect.)
> These results seem more believable. None of the differences are significant, but a non-significant result doesn’t mean the true magnitude is zero. If the true difference in full-scale IQ were 3.1 points as above, there would be no chance of significance with a sample of this size. It’s uncertain, but given this data, the best guess is 3.1 points, not 0 points.
I appreciated this paragraph. I've been reading some scientific papers lately and I keep getting annoyed that they treat a non-significant result in an underpowered study as evidence that the effect is exactly zero.
I'm not 100% sure how to think about the forgery. The fact that two hospitals were caught forging data seems very concerning. They exclude them from the data, but you have to worry that there might be other hospitals that weren't caught. On the other hand, they clearly had a fairly rigorous process to detect fraud and took it quite seriously if they found it.
Regarding effects: Definitely one of my most basic gripes with the statistics in most papers is total obsession with p-values and neglect of effect sizes. Like "correlation is not causation" it's so well-known as to be a cliche (https://en.wikipedia.org/wiki/Evidence_of_absence) but yet it's still so common...
Something I'd be curious about is at what point further encouragement to breastfeed becomes counterproductive. Lots of people have stories of struggling to breastfeed and eventually giving up. A lot of people are so determined to make it work that they end up undernourishing their baby until they switch to formula.
That implies that encouraging people to breastfeed could have very different effects depending on your audience's priors. Telling Belorussian mothers in the 90s to breastfeed probably amounts to "hey at least try it, don't just grab formula right away." But I feel like if you're talking to yuppy US moms in the 2020s, there's a zero percent chance they won't at least try. So the main effect of that intervention is probably to get them to stick at it longer even if they're struggling.
It's an interesting thought. Oddly, one of the suggested benefits of breastfeeding in the correlational studies is obesity. And those same correlations seem to suggest that exclusively breastfed babies are somewhat lighter. Some people suggest that that's an important mechanism for breastfeeding—it reduces obesity because the baby gets fewer total calories!
This seems strange to me for many reasons, among them that if that's really the mechanism, then you could just give less formula. But since PROBIT showed 0 effect on obesity (and seems to basically exclude a large effect) I personally wouldn't worry about this...
Excellent post, as often (we're talking stats here)!
One quibble:
> "If you ran a trial that compared no breastfeeding to exclusive breastfeeding for 12 months, the impact would surely have been much larger."
I'm not sure that's a claim one can make. It makes human logical sense, but I don't think it makes philosophical/mathematical/statistical logical sense - hell, you can't even say that *on average* it would have a larger impact, because we just don't know. Am I missing something?
As for possible confounders off the top of my head - I went over the paper quickly and didn't find that they controlled for total nutrition volume, nor did they show the group difference in infant weight/height/growth rate - but of course I might just have missed it in my perusal. Assuming no strict control for nutrition volume it's quite plausible there were differences in volume that could impact infant growth and thus hormonal and immune development.
I think I'd defend that statement... I mean, *technically*, yes, it's possible that that "none vs exclusive for 12 months" trial could have a lower effect. But the assumptions needed to guarantee a larger effect are quite mild. The issues are a bit complex, though—I spent a looong time in this colonoscopy article (https://asteriskmag.com/issues/04/you-re-invited-to-a-colonoscopy) trying to think about how much smaller an intention to treat analysis is than a full analysis. It's even more complex with breastfeeding (because it's not binary, unlike a 1-time colonoscopy) but the same basic principles apply.
For example, if you assume that
(1) "exclusive breastfeeding at 3 months" is the only thing that matters, and
(2) women in the intervention and treatment groups randomly decided if they should breastfeed for 3 months or not
then you should multiply all the effects by 2.7 (1/.369 since the difference of the groups was 36.9%). Both of those assumptions are surely false, but it gives an order of magnitude of what we should be thinking about, and that's still neglecting that there was some breastfeeding in the control group.
I think we have a philosophical difference then - in your colonoscopy article you seem to suggest the trial shows underwhelming efficacy for colonoscopies, while I see it as underwhelming efficacy for the intention to conduct colonoscopies. So too here - I can't get myself to assume that a theoretical RCT with exclusivity in breast milk vs. formula would inevitably show a higher effect size than the one in this ITT trial because it's comparing correlation apples to causation oranges.
Yes, of course, ITT is different from actually doing the treatment. This is why I spend so much time in the colonoscopy article that you can't divide the 18% reduction by the percentage of people who agreed to colonoscopies (42%) to say that if everyone got colonoscopies the reduction in colon cancer would be 18%/0.422 = 43%. (Which many GI doctors suggested!) In the colonoscopy trial, we *know* this is wrong because the people who refused colonoscopies did better than the control group, meaning people who were at higher risk were much more likely to agree. So I think the benefit is probably somewhat higher than 18% but not as high as 43%.
I think the philosophical position of refusing to make any extrapolation from ITT trails isn't a very practical one. That means you will never have any information about breastfeeding or colonoscopies, because we aren't ever getting a trial where people are forced to do or not do these things.
In any extrapolation, some guesswork is required. It might be higher or lower than the naive extrapolation suggests. With breastfeeding, it seems much less plausible to me that there was a bias in who selected to be breastfed. But who knows! The number is debatable. But I think it's pretty unreasonable to claim that with total breastfeeding vs no breastfeeding the effect size wouldn't be expected to be much larger.
But the benefits “fade out” over time. By the time those kids are in 7th or 8th grade, the benefit is gone.
But the actual article linked seems to say the opposite? Or in a way they are no longer visible academically but result in actual better QOL / Money in the longer term...
(From the article) "He also disagrees with those who say a gain of $1,000 a year is too small to matter, pointing out that these students earned an average of $16,000 a year at age 27, so $1,000 represents a significant 6 percent raise. Chetty adds that the new study merely highlights the benefits of a single year. It’s likely, he says, that each subsequent year with an excellent educator would yield additional pay increases:"
Yeah, I was just referring to measures on standardized tests, where the fade-out seems to be complete. (Which is what seems most relevant to the context of IQ tests.) It would be interesting to see income data on the breastfeeding cohorts!
(Sorry everyone, I seem to have accidentally set the comments on this post to closed. Should be open now.)
Parenting feels like one big giant emotional investment in diminishing returns - i.e. every parent willing to go gonzo for whatever small/insignificant effect sizes they can get. esp on things like iq/health/etc. Especially in those first born, early years with all of your best intentions. I wonder what belief updating around these practices look like for second+ born kids?
Also there's a lot to be said around the decision making framework around the results of statistical testing, etc - but no need to go down that fiery road to hell today.
Confounding is one thing, reverse causation (unhealthy infants latch poorly, so they get bottle fed) another. Check out Vera Wilde‘s work https://wildetruth.substack.com/p/a-recent-article-in-the-leading-breastfeeding
This is much more plausible than my "smart babies manipulate their mothers into breastfeeding" proposal!
Minor point: you use the phrase "if you can breastfeed" a few times. This is not quite the right way to think about the issue, because whether you "can" breastfeed isn't binary. Most mothers "can" breastfeed to some extent, but it imposes large--but finite--but only partially quantifiable costs. Examples include:
-- Serious nipple pain and injury, including bloody blisters, for the first 6ish weeks (eventually your nipple skin adjusts to the experience, but it takes a while)
-- Having to hide somewhere at work to pump several times a day, for 20-30 minutes at a time, instead of interacting with coworkers like a normal person
-- Having to plan all outings and socialization around opportunities to nurse and/or pump, because it's still awkward to do it in public
-- Never-ending storage/transport/supply logistics for pumped milk
These are only the ones I personally experienced. So "breastfeed if you can" is not an answer to the right question. The right question is whether, for a given person, breastfeeding passes a cost-benefit analysis. This may be very hard to determine even if you succeed in quantifying the benefits!
(I ended up happily sticking with breastfeeding and not doubting my decision much. From this, you can extrapolate how much higher the costs probably are for someone who is really on the fence.)
Totally agree. For what it's worth, this is also how I think about it! I think I just shortened it to "can" because I feel like these tradeoffs are so personal, and I don't want to come across like I'm telling anyone else how to navigate them. (I just want to understand what tradeoff is being made!)
Not sure if there's an easy way to add this kind of nuance without a long digression. Maybe I could just throw in a "...without undue difficulty" with some of them mentions? Even that sounds a little more prescriptive than I'd like. But if I sound prescriptive already, then OK!
This also raises the issue of whether dispensing stored human milk via bottle is equivalent to breastfeeding. It may be that the main advantage of breastfeeding is physical rather than chemical.
Certainly questions of IQ ten or twenty years out beg the question of whether any behavioral benefits of breastfeeding are short or long term.
Can you do one of these for SIDS, I recently came across this, but would like a second opinion:
https://open.substack.com/pub/sensiblemed/p/a-new-series-on-the-back-to-sleep?utm_source=share&utm_medium=android&r=fczw9
I'm wondering whether the science of formula-making could possibly have improved. Are the formulas used today different from those used in Belarus in 1996?
Formula has definitely evolved. I'm not exactly sure what formula was common in Belarus in 1996. But in terms of formula that's common in the West, I've found at least three major changes since that time:
1. Addition of docosahexaenoic acid
2. Addition of arachidonic acid
3. Addition of extra nucleotides
I'm not sure if there's rigorous proof that these help, but it's certainly plausible, and I think this is a valid reason to suspect that the benefits of breastfeeding over formula may be smaller now.
(Of course, it all depends on which mechanism you think is most plausible. If it's bio-active stuff like antibodies, then this might not make much of a difference.)
I think the "hospitals faking data" thing deserves more attention. In the Belarus RCT, one hospital was discovered to be faking data, and later another hospital was discovered to be faking data. How many other hospitals faked their data but didn't get caught? This makes me wonder if the original results of the study can be trusted at all.
(How did the known faked data change the results? Maybe faked data makes the results noisier and thus the study will underestimate the true effect size. But there could be some bias in how people faked the data that produces a spurious effect.)
> These results seem more believable. None of the differences are significant, but a non-significant result doesn’t mean the true magnitude is zero. If the true difference in full-scale IQ were 3.1 points as above, there would be no chance of significance with a sample of this size. It’s uncertain, but given this data, the best guess is 3.1 points, not 0 points.
I appreciated this paragraph. I've been reading some scientific papers lately and I keep getting annoyed that they treat a non-significant result in an underpowered study as evidence that the effect is exactly zero.
I'm not 100% sure how to think about the forgery. The fact that two hospitals were caught forging data seems very concerning. They exclude them from the data, but you have to worry that there might be other hospitals that weren't caught. On the other hand, they clearly had a fairly rigorous process to detect fraud and took it quite seriously if they found it.
Regarding effects: Definitely one of my most basic gripes with the statistics in most papers is total obsession with p-values and neglect of effect sizes. Like "correlation is not causation" it's so well-known as to be a cliche (https://en.wikipedia.org/wiki/Evidence_of_absence) but yet it's still so common...
Something I'd be curious about is at what point further encouragement to breastfeed becomes counterproductive. Lots of people have stories of struggling to breastfeed and eventually giving up. A lot of people are so determined to make it work that they end up undernourishing their baby until they switch to formula.
That implies that encouraging people to breastfeed could have very different effects depending on your audience's priors. Telling Belorussian mothers in the 90s to breastfeed probably amounts to "hey at least try it, don't just grab formula right away." But I feel like if you're talking to yuppy US moms in the 2020s, there's a zero percent chance they won't at least try. So the main effect of that intervention is probably to get them to stick at it longer even if they're struggling.
It's an interesting thought. Oddly, one of the suggested benefits of breastfeeding in the correlational studies is obesity. And those same correlations seem to suggest that exclusively breastfed babies are somewhat lighter. Some people suggest that that's an important mechanism for breastfeeding—it reduces obesity because the baby gets fewer total calories!
This seems strange to me for many reasons, among them that if that's really the mechanism, then you could just give less formula. But since PROBIT showed 0 effect on obesity (and seems to basically exclude a large effect) I personally wouldn't worry about this...
Excellent post, as often (we're talking stats here)!
One quibble:
> "If you ran a trial that compared no breastfeeding to exclusive breastfeeding for 12 months, the impact would surely have been much larger."
I'm not sure that's a claim one can make. It makes human logical sense, but I don't think it makes philosophical/mathematical/statistical logical sense - hell, you can't even say that *on average* it would have a larger impact, because we just don't know. Am I missing something?
As for possible confounders off the top of my head - I went over the paper quickly and didn't find that they controlled for total nutrition volume, nor did they show the group difference in infant weight/height/growth rate - but of course I might just have missed it in my perusal. Assuming no strict control for nutrition volume it's quite plausible there were differences in volume that could impact infant growth and thus hormonal and immune development.
I think I'd defend that statement... I mean, *technically*, yes, it's possible that that "none vs exclusive for 12 months" trial could have a lower effect. But the assumptions needed to guarantee a larger effect are quite mild. The issues are a bit complex, though—I spent a looong time in this colonoscopy article (https://asteriskmag.com/issues/04/you-re-invited-to-a-colonoscopy) trying to think about how much smaller an intention to treat analysis is than a full analysis. It's even more complex with breastfeeding (because it's not binary, unlike a 1-time colonoscopy) but the same basic principles apply.
For example, if you assume that
(1) "exclusive breastfeeding at 3 months" is the only thing that matters, and
(2) women in the intervention and treatment groups randomly decided if they should breastfeed for 3 months or not
then you should multiply all the effects by 2.7 (1/.369 since the difference of the groups was 36.9%). Both of those assumptions are surely false, but it gives an order of magnitude of what we should be thinking about, and that's still neglecting that there was some breastfeeding in the control group.
I think we have a philosophical difference then - in your colonoscopy article you seem to suggest the trial shows underwhelming efficacy for colonoscopies, while I see it as underwhelming efficacy for the intention to conduct colonoscopies. So too here - I can't get myself to assume that a theoretical RCT with exclusivity in breast milk vs. formula would inevitably show a higher effect size than the one in this ITT trial because it's comparing correlation apples to causation oranges.
(As a sanity check I went to see if Andrew Gelman had anything to say on the colonoscopy trial and I think he also sees it as apple and oranges - but again, that's one expert opinion who might have also shaped the way that I view statistics myself - https://statmodeling.stat.columbia.edu/2023/05/16/colonoscopy-corner-misleading-reporting-of-intent-to-treat-analysis/)
Yes, of course, ITT is different from actually doing the treatment. This is why I spend so much time in the colonoscopy article that you can't divide the 18% reduction by the percentage of people who agreed to colonoscopies (42%) to say that if everyone got colonoscopies the reduction in colon cancer would be 18%/0.422 = 43%. (Which many GI doctors suggested!) In the colonoscopy trial, we *know* this is wrong because the people who refused colonoscopies did better than the control group, meaning people who were at higher risk were much more likely to agree. So I think the benefit is probably somewhat higher than 18% but not as high as 43%.
I think the philosophical position of refusing to make any extrapolation from ITT trails isn't a very practical one. That means you will never have any information about breastfeeding or colonoscopies, because we aren't ever getting a trial where people are forced to do or not do these things.
In any extrapolation, some guesswork is required. It might be higher or lower than the naive extrapolation suggests. With breastfeeding, it seems much less plausible to me that there was a bias in who selected to be breastfed. But who knows! The number is debatable. But I think it's pretty unreasonable to claim that with total breastfeeding vs no breastfeeding the effect size wouldn't be expected to be much larger.
But the benefits “fade out” over time. By the time those kids are in 7th or 8th grade, the benefit is gone.
But the actual article linked seems to say the opposite? Or in a way they are no longer visible academically but result in actual better QOL / Money in the longer term...
(From the article) "He also disagrees with those who say a gain of $1,000 a year is too small to matter, pointing out that these students earned an average of $16,000 a year at age 27, so $1,000 represents a significant 6 percent raise. Chetty adds that the new study merely highlights the benefits of a single year. It’s likely, he says, that each subsequent year with an excellent educator would yield additional pay increases:"
Linked* article @ https://www.harvardmagazine.com/2010/10/kindergarten-matters
Yeah, I was just referring to measures on standardized tests, where the fade-out seems to be complete. (Which is what seems most relevant to the context of IQ tests.) It would be interesting to see income data on the breastfeeding cohorts!