There's a famous general of blog posts discussing, generally, what went wrong in the early '70s such that GDP growth slowed down, wages slowed down, etc. etc.
The more posts I see like yours the more I wonder if it is really just we hit a degree of regulation-induced bloat that makes it hard to sustain productivity at previous levels. You start with something that makes complete sense like "no more giving people syphilis without their knowledge"
or
"You can't choose not to hire someone because of their race/sex"
or
"You should supply hard hats to your employees"
or
"stop dumping disgusting chemical trash in rivers"
and then before you know it you end up in the situation you outline here, or NEPA reviews, or buildings take 4x as long to build as comparable countries, or whatever analogue you want to draw.
> "no more giving people syphilis without their knowledge"
Incidentally... before looking into all this, I had the impression that the Tuskegee study involved the government actively giving the men syphilis. I wonder how common that is. I think everyone would agree that the actual actions (concealing the diagnosis, giving fake treatments) are still pretty unconscionable.
I considered mentioning it! But I wasn't sure how common the misconception is. (Also, no one wants to be the one to say, "they didn't do that very very very very bad thing, merely this very very bad thing!"...)
I think the only real answer is to write and pass actually detailed laws and then fight it in the courts, with the caveat that those who win against the state get their legal fees covered. That means there will be many fewer rules, possibly less than optimal, but the costs of too few rules (especially national as opposed to state level) seems much lower than the cost of too many.
Another avenue of exploration that I'm sure is incredibly straightforwards and easy to understand: What exactly counts as an IRB? If I want to do a study that needs an IRB, can I gather 5 friends and ask them to say "yup this looks good" and call that enough?
Oh whoops, I missed that. This raises the question then of why haven't people already done this? Surely it would save e.g. drug companies quite a bit of time and money if they had an IRB in their pocket to rubber-stamp anything they want to do.
Oh, I think that if you started approving anything dodgy, HHS and/or the FDA would quickly reject the IRB. And anyway, for serious medical trials, there's so much overhead that IRB requirements probably aren't a major burden. The place I'd see this being most useful would be academics who are doing simple surveys where they ask people "what's your favorite color" and don't want to deal with the horror of their institutional IRB.
My understanding (per the slatestarcodex post linked in another comment) is that there are commercial IRBs, outside of academia, which already more or less fulfill this function. Since they make money from paying customers, they are incentivized to provide an efficient review process.
Indeed there are. (See https://willyreads.substack.com/i/36008067/efficient-compliance for some history.) Some universities don't even have their own IRBs, so if you want to do any research, you have to pay for one out of research funds. They're... kind of efficient, I guess? But they're pretty expensive ($X000). And they tend to take a fairly conservative interpretation of ambiguous rules. (They don't want to give HHS any excuse to shut them down.)
idea: develop a tool that integrates with some online communities (reddit?) that allows interested participants to organize themselves into a community-driven randomized controlled trial.
probably the most expensive part of trials are the recruitment of participants (here: community members) and the follow-up on outcomes (built as part of community values). the delivery and adherence to treatment can be somewhat treated as a sunk cost/equally challenging issue in RCTs. at least you can compute lots of ITTs!
sure there's biased selection into the trial but external validity is usually a pretty fraudulent claim anyway. the internally valid results are still very useful for the community.
Wonderful article, thank you for doing this research. You point out that no one has been prosecuted for violations, which is encouraging, but in reality this is all enforced via social mechanisms. A personal example:
As a hobby, I help officiate ("judge") Magic: The Gathering tournaments. The game is complicated and the tournaments have high prizes, so the Magic judges have their own tests and certifications and whatnot to ensure they know what they're doing. There are always arguments over whether the tests are too hard or too easy, and what level of knowledge is reasonable to expect from the (extremely underpaid) judges.
I was curious myself to get an idea of the baseline, so I started keeping track of the rulings I saw other judges take and how many they got right or wrong. (Note: watching other judges answer questions about the game is normal and in fact encouraged in case they need help; the only thing I was doing differently was writing down the result on a notepad on my phone. This is happening in a large convention center, so I had no private access; any person off the street could have done the exact same thing.)
After I had observed a few hundred rulings from a few dozen different judges, I made a short post on my personal Facebook page, saying "of calls taken by level 1/2/3 judges that I watched, they got X% correct without needing my help".
Dozens of judges and players swarmed my post to tell me that I was doing human subjects research, and thus needed an IRB. I was told that I had violated the Nuremburg Code, and was "bringing Nazism into judging". Several said that, by not asking other judges for their explicit consent to write down their answer on a notepad, I had "violated their consent", and implied that this meant I was in support of rape. Of the ~6 companies that run large Magic tournaments in North America, every one of them proceeded to ban me from judging for them. Most still have me banned now, 3 years later.
So I don't think the core problem here is the government. The average person just really loves exerting control over others and preventing them from trying to learn things about others. Our dystopian regulatory system is just the will of the people in this respect.
>Dozens of judges and players swarmed my post to tell me that I was doing human subjects research, and thus needed an IRB.
There's certainly no drug here, so unless you're affiliated with an institution that takes federal funding, they're definitely wrong that you need an IRB, even if it was research. And even that is quite debatable, as it's hard to see what the "generalizable knowledge" is. (This is a common angle people take when trying not to do "research". Don't create any generalizable conclusions!)
Regardless I wasn't particularly concerned about legal challenges, I was just blindsided that anyone would find this objectionable. (They're also technically right that it violates the Nuremburg code since it's so vaguely written.)
Basically I'm saying that this isn't a case of regulators overreaching, this seems to actually be what people want.
Wonder how much of this is would be true at any time and how much of this is just the bizarreness of SJ culture at its height... well, the elaborations seem like the latter, but I wouldn't expect that to lead to the *initial* objections (which I too would definitely not have expected!), so...? Yeah idk I don't have a good sense of this because I don't feel like I know the sort of person who would react like that! And the problem with asking people is that once they've learned these elaborated reasons they'll incorporate them into their way of thinking and not be able to think well about what caused them to have the reaction initially...
Thank you for writing this! I actually had this on a list of ideas of things to write about for my own blog, and I'm glad I didn't because yours is more comprehensive than mine would have been.
I still have a couple questions that perhaps you've come across during your research for this post?:
-Does IRB approval provide protection against lawsuits from research participants? Like, if I (or an institution I'm at) does a study where we send out surveys to humans, and a year later one of the participants comes and tries to sue me/the institution by saying the survey made them feel bad or their data was used improperly or not anonymized well or something... does the IRB approval provide some form of legal protection? Do IRBs view themselves as necessary due to liability concerns? Or is it purely to protect from, like... regulatory direction/oversight? How do they view their mandate?
-Are there specific guidelines that an IRB must satisfy to be an IRB? Like, if I form my own "Institute for the Study of Research" or whatever, and I create an IRB that is just me, then it's not an "Independent" Review Board. But does that even matter? Can I be my own IRB? Could I find a friend who is smart and ethical and appoint them as the IRB for the Institute for the Study of Research? What constrains an IRB being defined as an IRB? Can my IRB just grant categorical exemptions for broad swathes of research? Can my IRB just say "Approved" to everything, or must they create documents guaranteeing compliance with the various IRB-governing regulations?
-Some IRBs grant exemptions to, say, survey studies, or such. Others do not. Are there limits to what IRBs can grant exemptions to?
-Can a journal choose NOT to mandate IRB use? Like, will the journal be liable in the eyes of the government if they publish non-IRB research, or are journals doing this because they want to / have chosen to?
1. I think it does offer legal protection in that without an IRB there would be an increased risk of being held legally negligent. I think that plus fear of unpredictable HHS enforcement is what keeps people in line.
2. See the links in my "chaotic good" idea at the end. There are some requirements but I think they are *relatively* mild, and there could be some value to doing something similar.
This should be one of those areas for billionaire philanthropy. In addition to decoding burned scrolls and funding defamation lawsuits, I want to see more billionaires paying for people's legal challenges to poorly defined or interpreted federal regulations.
How is the IRB itself regulated? Could one hang out an IRB shingle and provide more "streamlined" review processes?
"As an analogy, driving a car is dangerous. Whenever I drive, I could easily kill someone. But the government doesn’t force me to submit a driving plan any time I want to go somewhere. Instead, if I misbehave, I am punished in retrospect. Why don’t we apply the same policy to research?"
Seems eerily analogous to NEPA. I guess on that note IRBs could be even worse. What if anyone could sue if the IRB didn't cover every potential angle of a study?
I have to admit, the tiny voice that wants to see the world burn is tempted to ask “Did you get IRB approval?” The next time I get an engagement survey at work. I wonder if one could get a company sued for internal surveys or other studies. It seems to fall squarely into the crosshairs of this sort of thing, but I have never heard of HR or say time study projects asking for IRB review.
That's a good point about generalizable knowledge. I wonder if they would fall afoul of the rulers in they tried to take a survey from one plant and apply the lessons to another, or if the company wide survey was sufficiently large. It seems like at some point it would, because otherwise you could always get around IRB by saying "We are surveying the opinions of only these 10,000 people, and the results do not generalize to other Americans we didn't speak to *wink wink*".
I absolutely wish there was some billionaire troll who made it their hobby to test these questions in the legal system and see what happens. Maybe we should start a non-profit org for doing just that.
Well, there's also the issue that unless you work at an institution that takes federal funding for research (or you're in NY/VA/MD), the Common Rule doesn't apply at all, and human subjects research without "drugs" is totally legal!
That's where it gets fun... some of my company's customers are aerospace contracts with the US gov, with locations in VA. A logical human with reasoning skills would say "That's a bit of a stretch" but a government regulator... who knows?
It would be amusing to see how litigation of that would work out. I expect that my company would be given a pass, but it might be close. The next company might be closer, and sooner or later it would have to come down to the regulators prosecuting someone for something obviously insane, or 90% of the affected parties suddenly realizing the rules don't actually apply to them. As you point out, most of the system "works" by virtue of people being afraid to find out what will actually be enforced.
> I used to live in an apartment with an extremely steep driveway. When I had visitors, I’d tell them, “At the bottom of the driveway you must slow to ⅒ of normal speed or your car will scrape the ground. I tell this to everyone and they only slow to ½ of normal speed and scrape their car on the ground. Don’t do that!” Then they’d only slow to ½ of normal speed and scrape their car on the ground.
I mean, if you don't tell them what "normal speed" is so that they have a target to measure against, and they aren't used to thinking about just what "normal speed" is, it may be hard to do better than this -- since you didn't give them numbers, and they don't know numbers, they have to go by feel, and while sure 1/2 and 1/10 will feel different to a first approximation they're both just "slow". Which is what then allows the *room* for the effect you describe. Did you try giving people an actual number? I'm curious if you did because I expect that would work better, by removing the wiggle room created by the need for judgement calls on feel.
(Hopefully you wouldn't have needed an IRB for that :P )
"From oversight to overkill" is another relevant book that dives into the history behind why IRBs are the way they are, and proposes some solutions for how to fix them
As an independent researcher investigating purely irrelevant stuff that that I'm mildly curious about, this Dynomight Internet Newsletter post wins the prize of the century.
See also psychiatrist Scott Alexander's blog post, "My IRB Nightmare", where he describes his surreal, Kafkaesque struggle to conduct a simple survey.
https://slatestarcodex.com/2017/08/29/my-irb-nightmare/
There's a famous general of blog posts discussing, generally, what went wrong in the early '70s such that GDP growth slowed down, wages slowed down, etc. etc.
The more posts I see like yours the more I wonder if it is really just we hit a degree of regulation-induced bloat that makes it hard to sustain productivity at previous levels. You start with something that makes complete sense like "no more giving people syphilis without their knowledge"
or
"You can't choose not to hire someone because of their race/sex"
or
"You should supply hard hats to your employees"
or
"stop dumping disgusting chemical trash in rivers"
and then before you know it you end up in the situation you outline here, or NEPA reviews, or buildings take 4x as long to build as comparable countries, or whatever analogue you want to draw.
Not sure what the solution is though.
> "no more giving people syphilis without their knowledge"
Incidentally... before looking into all this, I had the impression that the Tuskegee study involved the government actively giving the men syphilis. I wonder how common that is. I think everyone would agree that the actual actions (concealing the diagnosis, giving fake treatments) are still pretty unconscionable.
Incidentally...I read your article, cared enough to comment, and still missed the paragraph that said they didn't actually give them syphilis!
I considered mentioning it! But I wasn't sure how common the misconception is. (Also, no one wants to be the one to say, "they didn't do that very very very very bad thing, merely this very very bad thing!"...)
I think the only real answer is to write and pass actually detailed laws and then fight it in the courts, with the caveat that those who win against the state get their legal fees covered. That means there will be many fewer rules, possibly less than optimal, but the costs of too few rules (especially national as opposed to state level) seems much lower than the cost of too many.
> Science journals require IRB approval don’t seem to require details unless an editor requests them.
Confusing sentence structure here, I suspect a typo.
Fixed, thanks!
> that that
Another
Keep them coming!
Another avenue of exploration that I'm sure is incredibly straightforwards and easy to understand: What exactly counts as an IRB? If I want to do a study that needs an IRB, can I gather 5 friends and ask them to say "yup this looks good" and call that enough?
See my "chaotic good" idea at the end. I think this is semi-workable!
Oh whoops, I missed that. This raises the question then of why haven't people already done this? Surely it would save e.g. drug companies quite a bit of time and money if they had an IRB in their pocket to rubber-stamp anything they want to do.
Oh, I think that if you started approving anything dodgy, HHS and/or the FDA would quickly reject the IRB. And anyway, for serious medical trials, there's so much overhead that IRB requirements probably aren't a major burden. The place I'd see this being most useful would be academics who are doing simple surveys where they ask people "what's your favorite color" and don't want to deal with the horror of their institutional IRB.
My understanding (per the slatestarcodex post linked in another comment) is that there are commercial IRBs, outside of academia, which already more or less fulfill this function. Since they make money from paying customers, they are incentivized to provide an efficient review process.
Indeed there are. (See https://willyreads.substack.com/i/36008067/efficient-compliance for some history.) Some universities don't even have their own IRBs, so if you want to do any research, you have to pay for one out of research funds. They're... kind of efficient, I guess? But they're pretty expensive ($X000). And they tend to take a fairly conservative interpretation of ambiguous rules. (They don't want to give HHS any excuse to shut them down.)
idea: develop a tool that integrates with some online communities (reddit?) that allows interested participants to organize themselves into a community-driven randomized controlled trial.
probably the most expensive part of trials are the recruitment of participants (here: community members) and the follow-up on outcomes (built as part of community values). the delivery and adherence to treatment can be somewhat treated as a sunk cost/equally challenging issue in RCTs. at least you can compute lots of ITTs!
sure there's biased selection into the trial but external validity is usually a pretty fraudulent claim anyway. the internally valid results are still very useful for the community.
as someone who just got their research approved after battling IRB for months, this is disturbingly relevant to me.
Wonderful article, thank you for doing this research. You point out that no one has been prosecuted for violations, which is encouraging, but in reality this is all enforced via social mechanisms. A personal example:
As a hobby, I help officiate ("judge") Magic: The Gathering tournaments. The game is complicated and the tournaments have high prizes, so the Magic judges have their own tests and certifications and whatnot to ensure they know what they're doing. There are always arguments over whether the tests are too hard or too easy, and what level of knowledge is reasonable to expect from the (extremely underpaid) judges.
I was curious myself to get an idea of the baseline, so I started keeping track of the rulings I saw other judges take and how many they got right or wrong. (Note: watching other judges answer questions about the game is normal and in fact encouraged in case they need help; the only thing I was doing differently was writing down the result on a notepad on my phone. This is happening in a large convention center, so I had no private access; any person off the street could have done the exact same thing.)
After I had observed a few hundred rulings from a few dozen different judges, I made a short post on my personal Facebook page, saying "of calls taken by level 1/2/3 judges that I watched, they got X% correct without needing my help".
Dozens of judges and players swarmed my post to tell me that I was doing human subjects research, and thus needed an IRB. I was told that I had violated the Nuremburg Code, and was "bringing Nazism into judging". Several said that, by not asking other judges for their explicit consent to write down their answer on a notepad, I had "violated their consent", and implied that this meant I was in support of rape. Of the ~6 companies that run large Magic tournaments in North America, every one of them proceeded to ban me from judging for them. Most still have me banned now, 3 years later.
So I don't think the core problem here is the government. The average person just really loves exerting control over others and preventing them from trying to learn things about others. Our dystopian regulatory system is just the will of the people in this respect.
>Dozens of judges and players swarmed my post to tell me that I was doing human subjects research, and thus needed an IRB.
There's certainly no drug here, so unless you're affiliated with an institution that takes federal funding, they're definitely wrong that you need an IRB, even if it was research. And even that is quite debatable, as it's hard to see what the "generalizable knowledge" is. (This is a common angle people take when trying not to do "research". Don't create any generalizable conclusions!)
That's not what your flowchart says?
Regardless I wasn't particularly concerned about legal challenges, I was just blindsided that anyone would find this objectionable. (They're also technically right that it violates the Nuremburg code since it's so vaguely written.)
Basically I'm saying that this isn't a case of regulators overreaching, this seems to actually be what people want.
Wonder how much of this is would be true at any time and how much of this is just the bizarreness of SJ culture at its height... well, the elaborations seem like the latter, but I wouldn't expect that to lead to the *initial* objections (which I too would definitely not have expected!), so...? Yeah idk I don't have a good sense of this because I don't feel like I know the sort of person who would react like that! And the problem with asking people is that once they've learned these elaborated reasons they'll incorporate them into their way of thinking and not be able to think well about what caused them to have the reaction initially...
Thank you for writing this! I actually had this on a list of ideas of things to write about for my own blog, and I'm glad I didn't because yours is more comprehensive than mine would have been.
I still have a couple questions that perhaps you've come across during your research for this post?:
-Does IRB approval provide protection against lawsuits from research participants? Like, if I (or an institution I'm at) does a study where we send out surveys to humans, and a year later one of the participants comes and tries to sue me/the institution by saying the survey made them feel bad or their data was used improperly or not anonymized well or something... does the IRB approval provide some form of legal protection? Do IRBs view themselves as necessary due to liability concerns? Or is it purely to protect from, like... regulatory direction/oversight? How do they view their mandate?
-Are there specific guidelines that an IRB must satisfy to be an IRB? Like, if I form my own "Institute for the Study of Research" or whatever, and I create an IRB that is just me, then it's not an "Independent" Review Board. But does that even matter? Can I be my own IRB? Could I find a friend who is smart and ethical and appoint them as the IRB for the Institute for the Study of Research? What constrains an IRB being defined as an IRB? Can my IRB just grant categorical exemptions for broad swathes of research? Can my IRB just say "Approved" to everything, or must they create documents guaranteeing compliance with the various IRB-governing regulations?
-Some IRBs grant exemptions to, say, survey studies, or such. Others do not. Are there limits to what IRBs can grant exemptions to?
-Can a journal choose NOT to mandate IRB use? Like, will the journal be liable in the eyes of the government if they publish non-IRB research, or are journals doing this because they want to / have chosen to?
Yay, questions I think I can answer!
1. I think it does offer legal protection in that without an IRB there would be an increased risk of being held legally negligent. I think that plus fear of unpredictable HHS enforcement is what keeps people in line.
2. See the links in my "chaotic good" idea at the end. There are some requirements but I think they are *relatively* mild, and there could be some value to doing something similar.
3. All the exemptions are theoretically spelled out here: https://www.ecfr.gov/on/2018-07-19/title-45/part-46#p-46.104(d) I think anything on that list is not exempt by default.
4. AFAIK there is zero legal requirement for journals to mandate IRBs. (And I think in practice the mandates aren't as strict as some people believe.)
This should be one of those areas for billionaire philanthropy. In addition to decoding burned scrolls and funding defamation lawsuits, I want to see more billionaires paying for people's legal challenges to poorly defined or interpreted federal regulations.
How is the IRB itself regulated? Could one hang out an IRB shingle and provide more "streamlined" review processes?
"As an analogy, driving a car is dangerous. Whenever I drive, I could easily kill someone. But the government doesn’t force me to submit a driving plan any time I want to go somewhere. Instead, if I misbehave, I am punished in retrospect. Why don’t we apply the same policy to research?"
Seems eerily analogous to NEPA. I guess on that note IRBs could be even worse. What if anyone could sue if the IRB didn't cover every potential angle of a study?
I have to admit, the tiny voice that wants to see the world burn is tempted to ask “Did you get IRB approval?” The next time I get an engagement survey at work. I wonder if one could get a company sued for internal surveys or other studies. It seems to fall squarely into the crosshairs of this sort of thing, but I have never heard of HR or say time study projects asking for IRB review.
You might scare them, but legally, I think they're in the clear because the engagement survey isn't searching for "generalizable" knowledge.
That's a good point about generalizable knowledge. I wonder if they would fall afoul of the rulers in they tried to take a survey from one plant and apply the lessons to another, or if the company wide survey was sufficiently large. It seems like at some point it would, because otherwise you could always get around IRB by saying "We are surveying the opinions of only these 10,000 people, and the results do not generalize to other Americans we didn't speak to *wink wink*".
I absolutely wish there was some billionaire troll who made it their hobby to test these questions in the legal system and see what happens. Maybe we should start a non-profit org for doing just that.
Well, there's also the issue that unless you work at an institution that takes federal funding for research (or you're in NY/VA/MD), the Common Rule doesn't apply at all, and human subjects research without "drugs" is totally legal!
That's where it gets fun... some of my company's customers are aerospace contracts with the US gov, with locations in VA. A logical human with reasoning skills would say "That's a bit of a stretch" but a government regulator... who knows?
It would be amusing to see how litigation of that would work out. I expect that my company would be given a pass, but it might be close. The next company might be closer, and sooner or later it would have to come down to the regulators prosecuting someone for something obviously insane, or 90% of the affected parties suddenly realizing the rules don't actually apply to them. As you point out, most of the system "works" by virtue of people being afraid to find out what will actually be enforced.
I happened to read Philip K. Howard's book "Everyday Freedom" today and it really rhymes with this post: tldr: this is just a subset of a broader movement post-1960s to move from giving people authority to instead adding rule-based processes https://www.amazon.com/Everyday-Freedom-Designing-Framework-Flourishing/dp/1957588209
> I used to live in an apartment with an extremely steep driveway. When I had visitors, I’d tell them, “At the bottom of the driveway you must slow to ⅒ of normal speed or your car will scrape the ground. I tell this to everyone and they only slow to ½ of normal speed and scrape their car on the ground. Don’t do that!” Then they’d only slow to ½ of normal speed and scrape their car on the ground.
I mean, if you don't tell them what "normal speed" is so that they have a target to measure against, and they aren't used to thinking about just what "normal speed" is, it may be hard to do better than this -- since you didn't give them numbers, and they don't know numbers, they have to go by feel, and while sure 1/2 and 1/10 will feel different to a first approximation they're both just "slow". Which is what then allows the *room* for the effect you describe. Did you try giving people an actual number? I'm curious if you did because I expect that would work better, by removing the wiggle room created by the need for judgement calls on feel.
(Hopefully you wouldn't have needed an IRB for that :P )
>near-fractal complexity
Brilliant!
"From oversight to overkill" is another relevant book that dives into the history behind why IRBs are the way they are, and proposes some solutions for how to fix them
As an independent researcher investigating purely irrelevant stuff that that I'm mildly curious about, this Dynomight Internet Newsletter post wins the prize of the century.