Long detailed posts chronicling seemingly obscure topics that turn out to affect us all: THIS is why so many of us got excited about the Internet back in the 90s. Thanks for making full use of the medium.
This reminded me of the time Vsauce did A "real-life" trolley problem experiment. IIRC, he decided he didn't NEED to go to an IRB but wanted to talk with one anyway to hear their opinions. https://www.youtube.com/watch?v=1sl5KJ69qiA
As a long-standing member and 4-yr Chair of an IRB at a state university, I found this to be the best explanation of IRBs that I've read - and will share it wildly to help my colleagues understand how confusing it is. Please add the 2023 GAO report on IRBs (which is at least clear -- and includes the fact that private IRBs are few and most owned by hedge funds oh my!) at https://www.gao.gov/products/gao-23-104721 for as long as the website remains. Happy research!
Thanks, wow, those flowcharts of private equity gobbling up the independent IRBs are something else! I've added a link and also mirrored the report at https://dynomight.net/img/irb/gao-23-104721.pdf for safekeeping.
As an independent researcher investigating purely irrelevant stuff that that I'm mildly curious about, this Dynomight Internet Newsletter post wins the prize of the century.
"From oversight to overkill" is another relevant book that dives into the history behind why IRBs are the way they are, and proposes some solutions for how to fix them
> I used to live in an apartment with an extremely steep driveway. When I had visitors, I’d tell them, “At the bottom of the driveway you must slow to ⅒ of normal speed or your car will scrape the ground. I tell this to everyone and they only slow to ½ of normal speed and scrape their car on the ground. Don’t do that!” Then they’d only slow to ½ of normal speed and scrape their car on the ground.
I mean, if you don't tell them what "normal speed" is so that they have a target to measure against, and they aren't used to thinking about just what "normal speed" is, it may be hard to do better than this -- since you didn't give them numbers, and they don't know numbers, they have to go by feel, and while sure 1/2 and 1/10 will feel different to a first approximation they're both just "slow". Which is what then allows the *room* for the effect you describe. Did you try giving people an actual number? I'm curious if you did because I expect that would work better, by removing the wiggle room created by the need for judgement calls on feel.
(Hopefully you wouldn't have needed an IRB for that :P )
I have to admit, the tiny voice that wants to see the world burn is tempted to ask “Did you get IRB approval?” The next time I get an engagement survey at work. I wonder if one could get a company sued for internal surveys or other studies. It seems to fall squarely into the crosshairs of this sort of thing, but I have never heard of HR or say time study projects asking for IRB review.
That's a good point about generalizable knowledge. I wonder if they would fall afoul of the rulers in they tried to take a survey from one plant and apply the lessons to another, or if the company wide survey was sufficiently large. It seems like at some point it would, because otherwise you could always get around IRB by saying "We are surveying the opinions of only these 10,000 people, and the results do not generalize to other Americans we didn't speak to *wink wink*".
I absolutely wish there was some billionaire troll who made it their hobby to test these questions in the legal system and see what happens. Maybe we should start a non-profit org for doing just that.
Well, there's also the issue that unless you work at an institution that takes federal funding for research (or you're in NY/VA/MD), the Common Rule doesn't apply at all, and human subjects research without "drugs" is totally legal!
That's where it gets fun... some of my company's customers are aerospace contracts with the US gov, with locations in VA. A logical human with reasoning skills would say "That's a bit of a stretch" but a government regulator... who knows?
It would be amusing to see how litigation of that would work out. I expect that my company would be given a pass, but it might be close. The next company might be closer, and sooner or later it would have to come down to the regulators prosecuting someone for something obviously insane, or 90% of the affected parties suddenly realizing the rules don't actually apply to them. As you point out, most of the system "works" by virtue of people being afraid to find out what will actually be enforced.
"As an analogy, driving a car is dangerous. Whenever I drive, I could easily kill someone. But the government doesn’t force me to submit a driving plan any time I want to go somewhere. Instead, if I misbehave, I am punished in retrospect. Why don’t we apply the same policy to research?"
Seems eerily analogous to NEPA. I guess on that note IRBs could be even worse. What if anyone could sue if the IRB didn't cover every potential angle of a study?
This should be one of those areas for billionaire philanthropy. In addition to decoding burned scrolls and funding defamation lawsuits, I want to see more billionaires paying for people's legal challenges to poorly defined or interpreted federal regulations.
How is the IRB itself regulated? Could one hang out an IRB shingle and provide more "streamlined" review processes?
Thank you for writing this! I actually had this on a list of ideas of things to write about for my own blog, and I'm glad I didn't because yours is more comprehensive than mine would have been.
I still have a couple questions that perhaps you've come across during your research for this post?:
-Does IRB approval provide protection against lawsuits from research participants? Like, if I (or an institution I'm at) does a study where we send out surveys to humans, and a year later one of the participants comes and tries to sue me/the institution by saying the survey made them feel bad or their data was used improperly or not anonymized well or something... does the IRB approval provide some form of legal protection? Do IRBs view themselves as necessary due to liability concerns? Or is it purely to protect from, like... regulatory direction/oversight? How do they view their mandate?
-Are there specific guidelines that an IRB must satisfy to be an IRB? Like, if I form my own "Institute for the Study of Research" or whatever, and I create an IRB that is just me, then it's not an "Independent" Review Board. But does that even matter? Can I be my own IRB? Could I find a friend who is smart and ethical and appoint them as the IRB for the Institute for the Study of Research? What constrains an IRB being defined as an IRB? Can my IRB just grant categorical exemptions for broad swathes of research? Can my IRB just say "Approved" to everything, or must they create documents guaranteeing compliance with the various IRB-governing regulations?
-Some IRBs grant exemptions to, say, survey studies, or such. Others do not. Are there limits to what IRBs can grant exemptions to?
-Can a journal choose NOT to mandate IRB use? Like, will the journal be liable in the eyes of the government if they publish non-IRB research, or are journals doing this because they want to / have chosen to?
1. I think it does offer legal protection in that without an IRB there would be an increased risk of being held legally negligent. I think that plus fear of unpredictable HHS enforcement is what keeps people in line.
2. See the links in my "chaotic good" idea at the end. There are some requirements but I think they are *relatively* mild, and there could be some value to doing something similar.
Wonderful article, thank you for doing this research. You point out that no one has been prosecuted for violations, which is encouraging, but in reality this is all enforced via social mechanisms. A personal example:
As a hobby, I help officiate ("judge") Magic: The Gathering tournaments. The game is complicated and the tournaments have high prizes, so the Magic judges have their own tests and certifications and whatnot to ensure they know what they're doing. There are always arguments over whether the tests are too hard or too easy, and what level of knowledge is reasonable to expect from the (extremely underpaid) judges.
I was curious myself to get an idea of the baseline, so I started keeping track of the rulings I saw other judges take and how many they got right or wrong. (Note: watching other judges answer questions about the game is normal and in fact encouraged in case they need help; the only thing I was doing differently was writing down the result on a notepad on my phone. This is happening in a large convention center, so I had no private access; any person off the street could have done the exact same thing.)
After I had observed a few hundred rulings from a few dozen different judges, I made a short post on my personal Facebook page, saying "of calls taken by level 1/2/3 judges that I watched, they got X% correct without needing my help".
Dozens of judges and players swarmed my post to tell me that I was doing human subjects research, and thus needed an IRB. I was told that I had violated the Nuremburg Code, and was "bringing Nazism into judging". Several said that, by not asking other judges for their explicit consent to write down their answer on a notepad, I had "violated their consent", and implied that this meant I was in support of rape. Of the ~6 companies that run large Magic tournaments in North America, every one of them proceeded to ban me from judging for them. Most still have me banned now, 3 years later.
So I don't think the core problem here is the government. The average person just really loves exerting control over others and preventing them from trying to learn things about others. Our dystopian regulatory system is just the will of the people in this respect.
Wow, great example and I’m sorry that happened to you!
I don’t think your conclusion “The average person just really loves exerting control over others and preventing them from trying to learn things about others.” is right tho, I think it’s “The average person really hates the thought of being measured and found not-perfect by someone who isn’t an authority figure”
As a non-authority individual, writing down judges results and announcing them is an (unjustified-from-their-POV) claim to higher status (in the Impro sense). You got got by the status-slap down effect.
Wonder how much of this is would be true at any time and how much of this is just the bizarreness of SJ culture at its height... well, the elaborations seem like the latter, but I wouldn't expect that to lead to the *initial* objections (which I too would definitely not have expected!), so...? Yeah idk I don't have a good sense of this because I don't feel like I know the sort of person who would react like that! And the problem with asking people is that once they've learned these elaborated reasons they'll incorporate them into their way of thinking and not be able to think well about what caused them to have the reaction initially...
>Dozens of judges and players swarmed my post to tell me that I was doing human subjects research, and thus needed an IRB.
There's certainly no drug here, so unless you're affiliated with an institution that takes federal funding, they're definitely wrong that you need an IRB, even if it was research. And even that is quite debatable, as it's hard to see what the "generalizable knowledge" is. (This is a common angle people take when trying not to do "research". Don't create any generalizable conclusions!)
Regardless I wasn't particularly concerned about legal challenges, I was just blindsided that anyone would find this objectionable. (They're also technically right that it violates the Nuremburg code since it's so vaguely written.)
Basically I'm saying that this isn't a case of regulators overreaching, this seems to actually be what people want.
idea: develop a tool that integrates with some online communities (reddit?) that allows interested participants to organize themselves into a community-driven randomized controlled trial.
probably the most expensive part of trials are the recruitment of participants (here: community members) and the follow-up on outcomes (built as part of community values). the delivery and adherence to treatment can be somewhat treated as a sunk cost/equally challenging issue in RCTs. at least you can compute lots of ITTs!
sure there's biased selection into the trial but external validity is usually a pretty fraudulent claim anyway. the internally valid results are still very useful for the community.
Another avenue of exploration that I'm sure is incredibly straightforwards and easy to understand: What exactly counts as an IRB? If I want to do a study that needs an IRB, can I gather 5 friends and ask them to say "yup this looks good" and call that enough?
My understanding (per the slatestarcodex post linked in another comment) is that there are commercial IRBs, outside of academia, which already more or less fulfill this function. Since they make money from paying customers, they are incentivized to provide an efficient review process.
Indeed there are. (See https://willyreads.substack.com/i/36008067/efficient-compliance for some history.) Some universities don't even have their own IRBs, so if you want to do any research, you have to pay for one out of research funds. They're... kind of efficient, I guess? But they're pretty expensive ($X000). And they tend to take a fairly conservative interpretation of ambiguous rules. (They don't want to give HHS any excuse to shut them down.)
Oh whoops, I missed that. This raises the question then of why haven't people already done this? Surely it would save e.g. drug companies quite a bit of time and money if they had an IRB in their pocket to rubber-stamp anything they want to do.
Oh, I think that if you started approving anything dodgy, HHS and/or the FDA would quickly reject the IRB. And anyway, for serious medical trials, there's so much overhead that IRB requirements probably aren't a major burden. The place I'd see this being most useful would be academics who are doing simple surveys where they ask people "what's your favorite color" and don't want to deal with the horror of their institutional IRB.
Long detailed posts chronicling seemingly obscure topics that turn out to affect us all: THIS is why so many of us got excited about the Internet back in the 90s. Thanks for making full use of the medium.
This reminded me of the time Vsauce did A "real-life" trolley problem experiment. IIRC, he decided he didn't NEED to go to an IRB but wanted to talk with one anyway to hear their opinions. https://www.youtube.com/watch?v=1sl5KJ69qiA
As a long-standing member and 4-yr Chair of an IRB at a state university, I found this to be the best explanation of IRBs that I've read - and will share it wildly to help my colleagues understand how confusing it is. Please add the 2023 GAO report on IRBs (which is at least clear -- and includes the fact that private IRBs are few and most owned by hedge funds oh my!) at https://www.gao.gov/products/gao-23-104721 for as long as the website remains. Happy research!
Thanks, wow, those flowcharts of private equity gobbling up the independent IRBs are something else! I've added a link and also mirrored the report at https://dynomight.net/img/irb/gao-23-104721.pdf for safekeeping.
As an independent researcher investigating purely irrelevant stuff that that I'm mildly curious about, this Dynomight Internet Newsletter post wins the prize of the century.
"From oversight to overkill" is another relevant book that dives into the history behind why IRBs are the way they are, and proposes some solutions for how to fix them
>near-fractal complexity
Brilliant!
> I used to live in an apartment with an extremely steep driveway. When I had visitors, I’d tell them, “At the bottom of the driveway you must slow to ⅒ of normal speed or your car will scrape the ground. I tell this to everyone and they only slow to ½ of normal speed and scrape their car on the ground. Don’t do that!” Then they’d only slow to ½ of normal speed and scrape their car on the ground.
I mean, if you don't tell them what "normal speed" is so that they have a target to measure against, and they aren't used to thinking about just what "normal speed" is, it may be hard to do better than this -- since you didn't give them numbers, and they don't know numbers, they have to go by feel, and while sure 1/2 and 1/10 will feel different to a first approximation they're both just "slow". Which is what then allows the *room* for the effect you describe. Did you try giving people an actual number? I'm curious if you did because I expect that would work better, by removing the wiggle room created by the need for judgement calls on feel.
(Hopefully you wouldn't have needed an IRB for that :P )
I happened to read Philip K. Howard's book "Everyday Freedom" today and it really rhymes with this post: tldr: this is just a subset of a broader movement post-1960s to move from giving people authority to instead adding rule-based processes https://www.amazon.com/Everyday-Freedom-Designing-Framework-Flourishing/dp/1957588209
I have to admit, the tiny voice that wants to see the world burn is tempted to ask “Did you get IRB approval?” The next time I get an engagement survey at work. I wonder if one could get a company sued for internal surveys or other studies. It seems to fall squarely into the crosshairs of this sort of thing, but I have never heard of HR or say time study projects asking for IRB review.
You might scare them, but legally, I think they're in the clear because the engagement survey isn't searching for "generalizable" knowledge.
That's a good point about generalizable knowledge. I wonder if they would fall afoul of the rulers in they tried to take a survey from one plant and apply the lessons to another, or if the company wide survey was sufficiently large. It seems like at some point it would, because otherwise you could always get around IRB by saying "We are surveying the opinions of only these 10,000 people, and the results do not generalize to other Americans we didn't speak to *wink wink*".
I absolutely wish there was some billionaire troll who made it their hobby to test these questions in the legal system and see what happens. Maybe we should start a non-profit org for doing just that.
Well, there's also the issue that unless you work at an institution that takes federal funding for research (or you're in NY/VA/MD), the Common Rule doesn't apply at all, and human subjects research without "drugs" is totally legal!
That's where it gets fun... some of my company's customers are aerospace contracts with the US gov, with locations in VA. A logical human with reasoning skills would say "That's a bit of a stretch" but a government regulator... who knows?
It would be amusing to see how litigation of that would work out. I expect that my company would be given a pass, but it might be close. The next company might be closer, and sooner or later it would have to come down to the regulators prosecuting someone for something obviously insane, or 90% of the affected parties suddenly realizing the rules don't actually apply to them. As you point out, most of the system "works" by virtue of people being afraid to find out what will actually be enforced.
"As an analogy, driving a car is dangerous. Whenever I drive, I could easily kill someone. But the government doesn’t force me to submit a driving plan any time I want to go somewhere. Instead, if I misbehave, I am punished in retrospect. Why don’t we apply the same policy to research?"
Seems eerily analogous to NEPA. I guess on that note IRBs could be even worse. What if anyone could sue if the IRB didn't cover every potential angle of a study?
This should be one of those areas for billionaire philanthropy. In addition to decoding burned scrolls and funding defamation lawsuits, I want to see more billionaires paying for people's legal challenges to poorly defined or interpreted federal regulations.
How is the IRB itself regulated? Could one hang out an IRB shingle and provide more "streamlined" review processes?
Thank you for writing this! I actually had this on a list of ideas of things to write about for my own blog, and I'm glad I didn't because yours is more comprehensive than mine would have been.
I still have a couple questions that perhaps you've come across during your research for this post?:
-Does IRB approval provide protection against lawsuits from research participants? Like, if I (or an institution I'm at) does a study where we send out surveys to humans, and a year later one of the participants comes and tries to sue me/the institution by saying the survey made them feel bad or their data was used improperly or not anonymized well or something... does the IRB approval provide some form of legal protection? Do IRBs view themselves as necessary due to liability concerns? Or is it purely to protect from, like... regulatory direction/oversight? How do they view their mandate?
-Are there specific guidelines that an IRB must satisfy to be an IRB? Like, if I form my own "Institute for the Study of Research" or whatever, and I create an IRB that is just me, then it's not an "Independent" Review Board. But does that even matter? Can I be my own IRB? Could I find a friend who is smart and ethical and appoint them as the IRB for the Institute for the Study of Research? What constrains an IRB being defined as an IRB? Can my IRB just grant categorical exemptions for broad swathes of research? Can my IRB just say "Approved" to everything, or must they create documents guaranteeing compliance with the various IRB-governing regulations?
-Some IRBs grant exemptions to, say, survey studies, or such. Others do not. Are there limits to what IRBs can grant exemptions to?
-Can a journal choose NOT to mandate IRB use? Like, will the journal be liable in the eyes of the government if they publish non-IRB research, or are journals doing this because they want to / have chosen to?
Yay, questions I think I can answer!
1. I think it does offer legal protection in that without an IRB there would be an increased risk of being held legally negligent. I think that plus fear of unpredictable HHS enforcement is what keeps people in line.
2. See the links in my "chaotic good" idea at the end. There are some requirements but I think they are *relatively* mild, and there could be some value to doing something similar.
3. All the exemptions are theoretically spelled out here: https://www.ecfr.gov/on/2018-07-19/title-45/part-46#p-46.104(d) I think anything on that list is not exempt by default.
4. AFAIK there is zero legal requirement for journals to mandate IRBs. (And I think in practice the mandates aren't as strict as some people believe.)
Wonderful article, thank you for doing this research. You point out that no one has been prosecuted for violations, which is encouraging, but in reality this is all enforced via social mechanisms. A personal example:
As a hobby, I help officiate ("judge") Magic: The Gathering tournaments. The game is complicated and the tournaments have high prizes, so the Magic judges have their own tests and certifications and whatnot to ensure they know what they're doing. There are always arguments over whether the tests are too hard or too easy, and what level of knowledge is reasonable to expect from the (extremely underpaid) judges.
I was curious myself to get an idea of the baseline, so I started keeping track of the rulings I saw other judges take and how many they got right or wrong. (Note: watching other judges answer questions about the game is normal and in fact encouraged in case they need help; the only thing I was doing differently was writing down the result on a notepad on my phone. This is happening in a large convention center, so I had no private access; any person off the street could have done the exact same thing.)
After I had observed a few hundred rulings from a few dozen different judges, I made a short post on my personal Facebook page, saying "of calls taken by level 1/2/3 judges that I watched, they got X% correct without needing my help".
Dozens of judges and players swarmed my post to tell me that I was doing human subjects research, and thus needed an IRB. I was told that I had violated the Nuremburg Code, and was "bringing Nazism into judging". Several said that, by not asking other judges for their explicit consent to write down their answer on a notepad, I had "violated their consent", and implied that this meant I was in support of rape. Of the ~6 companies that run large Magic tournaments in North America, every one of them proceeded to ban me from judging for them. Most still have me banned now, 3 years later.
So I don't think the core problem here is the government. The average person just really loves exerting control over others and preventing them from trying to learn things about others. Our dystopian regulatory system is just the will of the people in this respect.
Wow, great example and I’m sorry that happened to you!
I don’t think your conclusion “The average person just really loves exerting control over others and preventing them from trying to learn things about others.” is right tho, I think it’s “The average person really hates the thought of being measured and found not-perfect by someone who isn’t an authority figure”
As a non-authority individual, writing down judges results and announcing them is an (unjustified-from-their-POV) claim to higher status (in the Impro sense). You got got by the status-slap down effect.
Wonder how much of this is would be true at any time and how much of this is just the bizarreness of SJ culture at its height... well, the elaborations seem like the latter, but I wouldn't expect that to lead to the *initial* objections (which I too would definitely not have expected!), so...? Yeah idk I don't have a good sense of this because I don't feel like I know the sort of person who would react like that! And the problem with asking people is that once they've learned these elaborated reasons they'll incorporate them into their way of thinking and not be able to think well about what caused them to have the reaction initially...
>Dozens of judges and players swarmed my post to tell me that I was doing human subjects research, and thus needed an IRB.
There's certainly no drug here, so unless you're affiliated with an institution that takes federal funding, they're definitely wrong that you need an IRB, even if it was research. And even that is quite debatable, as it's hard to see what the "generalizable knowledge" is. (This is a common angle people take when trying not to do "research". Don't create any generalizable conclusions!)
That's not what your flowchart says?
Regardless I wasn't particularly concerned about legal challenges, I was just blindsided that anyone would find this objectionable. (They're also technically right that it violates the Nuremburg code since it's so vaguely written.)
Basically I'm saying that this isn't a case of regulators overreaching, this seems to actually be what people want.
as someone who just got their research approved after battling IRB for months, this is disturbingly relevant to me.
idea: develop a tool that integrates with some online communities (reddit?) that allows interested participants to organize themselves into a community-driven randomized controlled trial.
probably the most expensive part of trials are the recruitment of participants (here: community members) and the follow-up on outcomes (built as part of community values). the delivery and adherence to treatment can be somewhat treated as a sunk cost/equally challenging issue in RCTs. at least you can compute lots of ITTs!
sure there's biased selection into the trial but external validity is usually a pretty fraudulent claim anyway. the internally valid results are still very useful for the community.
Another avenue of exploration that I'm sure is incredibly straightforwards and easy to understand: What exactly counts as an IRB? If I want to do a study that needs an IRB, can I gather 5 friends and ask them to say "yup this looks good" and call that enough?
My understanding (per the slatestarcodex post linked in another comment) is that there are commercial IRBs, outside of academia, which already more or less fulfill this function. Since they make money from paying customers, they are incentivized to provide an efficient review process.
Indeed there are. (See https://willyreads.substack.com/i/36008067/efficient-compliance for some history.) Some universities don't even have their own IRBs, so if you want to do any research, you have to pay for one out of research funds. They're... kind of efficient, I guess? But they're pretty expensive ($X000). And they tend to take a fairly conservative interpretation of ambiguous rules. (They don't want to give HHS any excuse to shut them down.)
See my "chaotic good" idea at the end. I think this is semi-workable!
Oh whoops, I missed that. This raises the question then of why haven't people already done this? Surely it would save e.g. drug companies quite a bit of time and money if they had an IRB in their pocket to rubber-stamp anything they want to do.
Oh, I think that if you started approving anything dodgy, HHS and/or the FDA would quickly reject the IRB. And anyway, for serious medical trials, there's so much overhead that IRB requirements probably aren't a major burden. The place I'd see this being most useful would be academics who are doing simple surveys where they ask people "what's your favorite color" and don't want to deal with the horror of their institutional IRB.