My personal heuristic/litmus test for whether some particular substance has a psychological effect is whether it's possible to take too much and have a bad time. Weed, alcohol, caffeine, amphetamines, etc, all pass easily. Theanine, chamomile, etc, not so much. Failing this test doesn't conclusively mean that something has no effect, but it sows doubt in my mind.
I think there's an important analysis you missed: How your estimate of whether you took Theanine influenced the result. Here's the result of my analysis:
The correlation looks quite strong! Eyeballing the steepness, it's a stress reduction of 0.25 for every 10 percentage points of increased certainty you took Theanine. I think this supports the role of a placebo effect quite strongly.
(Linear regression gives a line steepness of -0.0283, per percentage point.)
Note that this analysis marginalizes out time, which might matter.
Here's the code I've used if anyone is interested:
Originally, the idea of the T/D prediction is I thought that I might be able to do better than just a prediction based on the change in stress levels. For example, I thought that I might be able to internally calibrate for how stressful/relaxing my life was during the one hour period, or I might be able to detect some other kind of effects of theanine other than stress. It seemed plausible that there would be a much stronger signal for T/D vs. prediction than vs. Δstress. But things didn't turn out that way...
I don't know why this is confusing. The physiological effects of theanine and caffeine are time varying. They are nutrients. They trigger physiological effects that are also time varying. So, your study should control for when and how much caffeine you ingested.
For instance, it's well known that caffeine increases heart rate and blood pressure. How long does this effect take to be induced? How might another chemical like theanine mediate this effect? This might happen because theanine blocks the heart rate increase, but theanine may not have the capacity to lower heart rate once it is already established (or its effect may be weaker). This is just one hypothesis, but it is an example of a reason to doubt your conclusions.
You could simplify your experiment. You could just test for whether taking caffeine and theanine simultaneously reduce the heart rate or blood pressure effects of caffeine alone. This would be more accurate to measure and could more easily control for other effects.
> I’ve seen many other self-experiments (including for theanine), but they’re non-blinded and I’d be doing you a disservice if I liked to them. People often mention that hypothetically this means the results aren’t scientific, but treat it like a small niggling technicality. It’s not.
As someone who published a non-blinded caffeine self-study, I feel attacked :P
In my defense, I was testing the time-series effect of caffeine, not the effect of caffeine vs. no caffeine, and I don't know if there's any way to blind myself to the passage of time. Only solution would be to run the test on other people and not tell them the purpose of the experiment, which is indeed best practice but it's much harder than a self-experiment.
To be honest, if you're testing for non-actute effects and/or you're testing something that has symptoms that are quite easy to observe, I don't think blinding practically makes that much of a difference. That is, if you blinded me to spend a month taking either caffeine pills or placebo, I feel like I would quickly figure out which it was with almost 100% accuracy?
So... for this experiment, the placebo (or whatever) effect does worry me, but I don't really think blinding would make all that much difference, nor do I see any other easy way to solve it. The truth is, the cases where easy randomized blinding can really solve all placebo-esque problems are a minority!
Congratulations on doing the work! I truly admire when someone follows through on their curiosity and puts the fad of the day under a little bit of scrutiny.
I hate to say that I'm not sure this was an effective test?
I have not heard of theanine as an acute anxiety relief.
It seems to help cutoff excess activity in some regards. I'm hoping other people will have more sophisticated comments here, and if not I'll circle back with some paper about limiting dopamine spikes.
Anecdotally it vastly diminishes some side effects of high caffeine doses for me. But yes not very blinded. So if theres nothing compelling in the comments I'll try to swing back with the dopamine cutoff filter paper.
Very curious what kind of job @Dynomight has. Is he stressed out at 9:00am when the emails are flying, and then automatically less stressed on his lunch break 2.5 hours later?
Notably, I once took like 2000 mg of theanine and definitely felt something. Nothing crazy but I felt pretty chill. This was years ago but I likened it to maybe 0.5 mg xanax with a bit more sleepiness and less euphoria. You did 200 mg. Your theanine aspirations may be achievable by upping the dosage
I'm considering doing 400mg. The biggest reason I used 200mg is that blinding gets very complicated at higher doses. How do you randomly ensure that you *either* get 2 theanine pills or 2 D pills? (It's certainly possible, just annoying.)
Separate the theanine pills and the vitamin D pills into two separate bags and put a folded-up note inside that identifies them. Then mix up the bags (or have someone else do it for you), so you don't know which bag is which. Then label the outside "1" and "2". On alternate days, take two capsules from a bag, and note whether it was bag #1 or bag #2. At the end of the experiment, open the bags and read the notes to tell you which pills you were taking on which day. Of course, you have to be sure that the pills are sufficiently similar to each other that you can't identify them just by looking at them. Under this protocol, your theanine doses occur every-other day -- you just don't know which days.
That's very nice and non-annoying. It does have the downside that over time I might develop an intuition for what bag #1 and bag #2 contain.
Instead, I think this is what I'd do:
- Get 100 envelopes
- Write #1 on the back of half, put 2 theanine pills inside
- Write #2 on the back of others, put 2 placebos inside
- Shuffle all the envelopes, write numbers 1-100 on the front
- Do the experiment
- Look at the backs
There's still a (very) small issue that this isn't technically IID since you're guaranteed exactly a 50/50 split over time, though! I can think of ways to overcome that, too, but I haven't thought of a non-annoying way yet.
Perhaps the easiest thing would be to just put 2 theanine and 1 placebo in 100 numbered envelopes and take 2 capsules each day, so there's a 3-armed trial with 0/200/400 mg.
This seems reasonable, and my only thought is wondering whether theanine or vitamin D have any taste, and if not, maybe there is an approach that involves mixing them in a liquid.
Alternatively, you could buy powdered theanine and vitamine D, then manually fill up some bigger capsules. I figure that weighing and filling up about a hundred or so should take about an hour? (I notably don’t see any issue with a range in dosage between 400-500 mg, since you wouldn’t be perfectly accurate)
For what it's worth, I think it's a significant problem that you were seeing the results as they came in:
- If you were invested in the result I'd imagine you'd get anxious when you were getting closer to finding today's result
- Once you started to realise that getting a meaningful effect size was unlikely, your motivation around conducting the experiment consistently/thoroughly had a potential to drop
Yeah, in retrospect this was a mistake. (I think a lot*of blind self experiments have the same mistake but don't highlight it.) I don't think the first one was an issue. But I did start to doubt my ability to detect D/T later in the experiment. I assure you I was still very thorough! But I doubted my ability to detect D/T and it's easy to imagine that affected my predictions.
In the future, you may just want to just visualize things first with a simple bar graph: two pre means, two post means, and error bars (or if you'd like, one of those many bar graph spinoffs a box plot). Any boutique visualization method will be harder to processes even for a decently competent and motivated audience
We did an n=1 experiment with our teenage son for Adderall and caffeine that we have yet to write up. Rough conclusion was Adderall works better than caffeine, which works better than placebo.
This echoes my experience with Theanine: no detectable difference. I was not blinded but I expected an effect and did not experience one. In my personal experience a good night's sleep has an effect, meditation has an effect, and a ten minute walk can help to reduce a sense of anxiety/stress. So can dealing directly with whatever work I am late on or a conversation I need to have with someone. But I appreciate your rigor and would have given theanine a second look if you had seen an effect.
Regarding the data analysis: There's a fascinating book called "The Cult of Statistical Significance," one of whose themes is that "(effect) size matters". That is, even if p < 0.001 for some intervention, the magnitude of the effect might be so small as to be clinically irrelevant.
So I concur with your "just look at the data" perspective. If the effect for a trial like this isn't large enough to leap out at you from the raw data, then the intervention is probably pointless, no matter what the p-value is.
tl;dr: I don't care about a weight loss remedy with an expected loss of 2 +/- 0.8 lbs, or an anti-anxiety treatment that reduces my anxiety 4% more than a placebo does.
One thing I wonder is whether what people are experiencing when they say theanine helps them (e.g., by drinking tea) is that they are just having less coffee. I.e., the variable is the coffee consumption vs. something in the tea
For coffee vs tea it's hard to say (because most tea has only a tiny amount of theanine, and also because it's hard to measure how much caffeine is actually in any given coffee or tea). But the reports from people who supplement theanine... well... they're all over the place:
I performed a self-blinded experiment on creatine supplements to see if they would improve my reaction time (I chose reaction time because it's inversely correlated with cognitive ability):
I did seem to have faster reaction times when I was taking the creatine supplements, but the effect was either small or not significant (and I'm too much of a statistical ignoramus to perform the necessary analysis to get a more rigorous answer than that).
(I should have linked to them... I've promoted these to numerous people.)
Regarding creatine, do I understand correctly that you randomized over blocks of weeks? I.e. did you create 2 sets each of placebo and creatine capsules, and you just happened to get them in the order placebo / creatine / placebo / creatine?
Incidentally, what's the status of the mentoring project you mentioned in January? I'd love to have someone teach me how to analyze the results of these types of experiments.
I got 300 amazing applications and ended up choosing completely randomly. I tried to email everyone who applied but kept getting blocked as spam, even when I tried to email people in batches of 10 :( I ended up putting an update at the end of this post: https://dynomight.net/bayes/
Yes, that is correct. It could have turned out the other way (placebo / creatine / placebo / creatine). I used large, empty, opaque capsules and filled them either with the smaller creatine capsules or an inert substance (allulose). They looked identical and weighed the same (or almost the same, anyway).
Upon edit: There were only two sets of capsules, placebo and creatine. I alternated them for the two pairs of phases, thereby ensuring that I wouldn't have two creatine phases in a row (or two placebo phases). Does that make sense?
Stimulants work, but are worse than nothing if they are abused or poorly timed relative to your sleep schedule.
Melatonin works, but shouldn't be necessary if you wake up at a consistent time and get at least moderate exercise during the day. This can be expanded to "getting good sleep works." Unless of course you have a medical condition, at which point Melatonin is probably not enough.
Eating healthy and exercise work, but are so self-evidently good we don't really need experiments to verify their positive effects.
I think everything else is mostly irrelevant. Almost everyone has habits that breaking would do far more good than adding any niche chemical ever could.
Yeah, I've considered some self-experiments with stimulants but it seems extremely difficult to do properly! If you want to account for tolerance, it seems like what you'd need to do is randomize entire months to taking caffeine or placebo to measure adaptation.
Anyway I definitely agree that diet/exercise/sleep are the big wins. (To that list, I might add: friends?)
Personally I've gotten into a groove where I have a detox week every 6 weeks or so, and don't take stimulants on weekends (unless I'm working). I haven't done any hard experiments with this, but compared to when I was younger without intentional stimulant use, I'm multiple times more productive.
Never heard of that last big win but I'll do some research if I have the time, thanks.
Lots of people seem convinced that you get "full adaptation" to caffeine and that if you take it for months at a time, you're just getting back to baseline. My best guess is that that's never quite true (even without tolerance breaks) though I've never seen convincing proof either way.
Personal experience tells me that it’s not, or at least there’s a significant difference between the me who has abstained from caffeine for a long time (multiple weeks at least) and the me who has that first cup of coffee after a long tolerance break.
It also helps regulate “when” I feel awake, which absent it is hit or miss. Even if caffeine just becomes a requirement for baseline, being able to rapidly hit baseline in the morning, and avoid the post-lunch fatigue is valuable in itself. So long as I get good sleep I can feel normal the whole day with caffeine, whereas without it there’s times of the day I’ll just feel lethargic.
I see a lot of people treat it as a substitute for sleep though, so they end with sleep deprivation —> Consume more caffeine —> Get more sleep deprived —> etc.
My personal heuristic/litmus test for whether some particular substance has a psychological effect is whether it's possible to take too much and have a bad time. Weed, alcohol, caffeine, amphetamines, etc, all pass easily. Theanine, chamomile, etc, not so much. Failing this test doesn't conclusively mean that something has no effect, but it sows doubt in my mind.
I think there's an important analysis you missed: How your estimate of whether you took Theanine influenced the result. Here's the result of my analysis:
https://i.postimg.cc/dV9cpKx1/analysis.png
The correlation looks quite strong! Eyeballing the steepness, it's a stress reduction of 0.25 for every 10 percentage points of increased certainty you took Theanine. I think this supports the role of a placebo effect quite strongly.
(Linear regression gives a line steepness of -0.0283, per percentage point.)
Note that this analysis marginalizes out time, which might matter.
Here's the code I've used if anyone is interested:
https://drive.google.com/file/d/11wmD_f-hwf31e-7VfUFsNb5UG28G33Oc/view?usp=sharing
This is helpful, thanks!
Originally, the idea of the T/D prediction is I thought that I might be able to do better than just a prediction based on the change in stress levels. For example, I thought that I might be able to internally calibrate for how stressful/relaxing my life was during the one hour period, or I might be able to detect some other kind of effects of theanine other than stress. It seemed plausible that there would be a much stronger signal for T/D vs. prediction than vs. Δstress. But things didn't turn out that way...
I don't know why this is confusing. The physiological effects of theanine and caffeine are time varying. They are nutrients. They trigger physiological effects that are also time varying. So, your study should control for when and how much caffeine you ingested.
For instance, it's well known that caffeine increases heart rate and blood pressure. How long does this effect take to be induced? How might another chemical like theanine mediate this effect? This might happen because theanine blocks the heart rate increase, but theanine may not have the capacity to lower heart rate once it is already established (or its effect may be weaker). This is just one hypothesis, but it is an example of a reason to doubt your conclusions.
You could simplify your experiment. You could just test for whether taking caffeine and theanine simultaneously reduce the heart rate or blood pressure effects of caffeine alone. This would be more accurate to measure and could more easily control for other effects.
> I’ve seen many other self-experiments (including for theanine), but they’re non-blinded and I’d be doing you a disservice if I liked to them. People often mention that hypothetically this means the results aren’t scientific, but treat it like a small niggling technicality. It’s not.
As someone who published a non-blinded caffeine self-study, I feel attacked :P
In my defense, I was testing the time-series effect of caffeine, not the effect of caffeine vs. no caffeine, and I don't know if there's any way to blind myself to the passage of time. Only solution would be to run the test on other people and not tell them the purpose of the experiment, which is indeed best practice but it's much harder than a self-experiment.
To be honest, if you're testing for non-actute effects and/or you're testing something that has symptoms that are quite easy to observe, I don't think blinding practically makes that much of a difference. That is, if you blinded me to spend a month taking either caffeine pills or placebo, I feel like I would quickly figure out which it was with almost 100% accuracy?
So... for this experiment, the placebo (or whatever) effect does worry me, but I don't really think blinding would make all that much difference, nor do I see any other easy way to solve it. The truth is, the cases where easy randomized blinding can really solve all placebo-esque problems are a minority!
Congratulations on doing the work! I truly admire when someone follows through on their curiosity and puts the fad of the day under a little bit of scrutiny.
I hate to say that I'm not sure this was an effective test?
I have not heard of theanine as an acute anxiety relief.
It seems to help cutoff excess activity in some regards. I'm hoping other people will have more sophisticated comments here, and if not I'll circle back with some paper about limiting dopamine spikes.
Anecdotally it vastly diminishes some side effects of high caffeine doses for me. But yes not very blinded. So if theres nothing compelling in the comments I'll try to swing back with the dopamine cutoff filter paper.
Very curious what kind of job @Dynomight has. Is he stressed out at 9:00am when the emails are flying, and then automatically less stressed on his lunch break 2.5 hours later?
Notably, I once took like 2000 mg of theanine and definitely felt something. Nothing crazy but I felt pretty chill. This was years ago but I likened it to maybe 0.5 mg xanax with a bit more sleepiness and less euphoria. You did 200 mg. Your theanine aspirations may be achievable by upping the dosage
I'm considering doing 400mg. The biggest reason I used 200mg is that blinding gets very complicated at higher doses. How do you randomly ensure that you *either* get 2 theanine pills or 2 D pills? (It's certainly possible, just annoying.)
Separate the theanine pills and the vitamin D pills into two separate bags and put a folded-up note inside that identifies them. Then mix up the bags (or have someone else do it for you), so you don't know which bag is which. Then label the outside "1" and "2". On alternate days, take two capsules from a bag, and note whether it was bag #1 or bag #2. At the end of the experiment, open the bags and read the notes to tell you which pills you were taking on which day. Of course, you have to be sure that the pills are sufficiently similar to each other that you can't identify them just by looking at them. Under this protocol, your theanine doses occur every-other day -- you just don't know which days.
That's very nice and non-annoying. It does have the downside that over time I might develop an intuition for what bag #1 and bag #2 contain.
Instead, I think this is what I'd do:
- Get 100 envelopes
- Write #1 on the back of half, put 2 theanine pills inside
- Write #2 on the back of others, put 2 placebos inside
- Shuffle all the envelopes, write numbers 1-100 on the front
- Do the experiment
- Look at the backs
There's still a (very) small issue that this isn't technically IID since you're guaranteed exactly a 50/50 split over time, though! I can think of ways to overcome that, too, but I haven't thought of a non-annoying way yet.
Perhaps the easiest thing would be to just put 2 theanine and 1 placebo in 100 numbered envelopes and take 2 capsules each day, so there's a 3-armed trial with 0/200/400 mg.
OK, I think I've got it! Here's the protocol:
- Get 100 numbered envelopes
- Each day put 2 theanine pills and 2 placebos in two identical cups
- Eat the pills from one cup, store the others in an envelope
- At the end of the experiment, loop in the envelopes
This seems reasonable, and my only thought is wondering whether theanine or vitamin D have any taste, and if not, maybe there is an approach that involves mixing them in a liquid.
Alternatively, you could buy powdered theanine and vitamine D, then manually fill up some bigger capsules. I figure that weighing and filling up about a hundred or so should take about an hour? (I notably don’t see any issue with a range in dosage between 400-500 mg, since you wouldn’t be perfectly accurate)
For what it's worth, I think it's a significant problem that you were seeing the results as they came in:
- If you were invested in the result I'd imagine you'd get anxious when you were getting closer to finding today's result
- Once you started to realise that getting a meaningful effect size was unlikely, your motivation around conducting the experiment consistently/thoroughly had a potential to drop
Yeah, in retrospect this was a mistake. (I think a lot*of blind self experiments have the same mistake but don't highlight it.) I don't think the first one was an issue. But I did start to doubt my ability to detect D/T later in the experiment. I assure you I was still very thorough! But I doubted my ability to detect D/T and it's easy to imagine that affected my predictions.
In the future, you may just want to just visualize things first with a simple bar graph: two pre means, two post means, and error bars (or if you'd like, one of those many bar graph spinoffs a box plot). Any boutique visualization method will be harder to processes even for a decently competent and motivated audience
Thank you for the illuminating science!
We did an n=1 experiment with our teenage son for Adderall and caffeine that we have yet to write up. Rough conclusion was Adderall works better than caffeine, which works better than placebo.
This echoes my experience with Theanine: no detectable difference. I was not blinded but I expected an effect and did not experience one. In my personal experience a good night's sleep has an effect, meditation has an effect, and a ten minute walk can help to reduce a sense of anxiety/stress. So can dealing directly with whatever work I am late on or a conversation I need to have with someone. But I appreciate your rigor and would have given theanine a second look if you had seen an effect.
Excellent experiment.
Regarding the data analysis: There's a fascinating book called "The Cult of Statistical Significance," one of whose themes is that "(effect) size matters". That is, even if p < 0.001 for some intervention, the magnitude of the effect might be so small as to be clinically irrelevant.
So I concur with your "just look at the data" perspective. If the effect for a trial like this isn't large enough to leap out at you from the raw data, then the intervention is probably pointless, no matter what the p-value is.
tl;dr: I don't care about a weight loss remedy with an expected loss of 2 +/- 0.8 lbs, or an anti-anxiety treatment that reduces my anxiety 4% more than a placebo does.
One thing I wonder is whether what people are experiencing when they say theanine helps them (e.g., by drinking tea) is that they are just having less coffee. I.e., the variable is the coffee consumption vs. something in the tea
For coffee vs tea it's hard to say (because most tea has only a tiny amount of theanine, and also because it's hard to measure how much caffeine is actually in any given coffee or tea). But the reports from people who supplement theanine... well... they're all over the place:
https://old.reddit.com/r/Supplements/comments/x7u5mg/wtf_i_tried_theanine_and_it_actually_works/
https://old.reddit.com/r/Supplements/comments/1ethltn/ltheanine_is_changing_my_life/
https://old.reddit.com/r/Supplements/comments/15kpfl9/ltheanine_wow/
https://old.reddit.com/r/Supplements/comments/1hvfase/do_not_sleep_on_ltheanine/
https://old.reddit.com/r/Nootropics/comments/wvc781/ltheanine_is_the_best_thing_i_ever_bought/
https://old.reddit.com/r/Nootropics/comments/1pzq7a/ltheanine_the_only_nootropic_i_have_faith_in/
https://old.reddit.com/r/Nootropics/comments/1b2jh0o/ltheanine_is_insanely_useful_for_me/
I performed a self-blinded experiment on creatine supplements to see if they would improve my reaction time (I chose reaction time because it's inversely correlated with cognitive ability):
https://www.self-experiments.org/
I did seem to have faster reaction times when I was taking the creatine supplements, but the effect was either small or not significant (and I'm too much of a statistical ignoramus to perform the necessary analysis to get a more rigorous answer than that).
This is great! I'm also a big fan of your earlier experiments on caffeine:
https://www.self-experiments.org/caffeine-experiment/
(I should have linked to them... I've promoted these to numerous people.)
Regarding creatine, do I understand correctly that you randomized over blocks of weeks? I.e. did you create 2 sets each of placebo and creatine capsules, and you just happened to get them in the order placebo / creatine / placebo / creatine?
Incidentally, what's the status of the mentoring project you mentioned in January? I'd love to have someone teach me how to analyze the results of these types of experiments.
I got 300 amazing applications and ended up choosing completely randomly. I tried to email everyone who applied but kept getting blocked as spam, even when I tried to email people in batches of 10 :( I ended up putting an update at the end of this post: https://dynomight.net/bayes/
Yes, that is correct. It could have turned out the other way (placebo / creatine / placebo / creatine). I used large, empty, opaque capsules and filled them either with the smaller creatine capsules or an inert substance (allulose). They looked identical and weighed the same (or almost the same, anyway).
Upon edit: There were only two sets of capsules, placebo and creatine. I alternated them for the two pairs of phases, thereby ensuring that I wouldn't have two creatine phases in a row (or two placebo phases). Does that make sense?
Stimulants work, but are worse than nothing if they are abused or poorly timed relative to your sleep schedule.
Melatonin works, but shouldn't be necessary if you wake up at a consistent time and get at least moderate exercise during the day. This can be expanded to "getting good sleep works." Unless of course you have a medical condition, at which point Melatonin is probably not enough.
Eating healthy and exercise work, but are so self-evidently good we don't really need experiments to verify their positive effects.
I think everything else is mostly irrelevant. Almost everyone has habits that breaking would do far more good than adding any niche chemical ever could.
Yeah, I've considered some self-experiments with stimulants but it seems extremely difficult to do properly! If you want to account for tolerance, it seems like what you'd need to do is randomize entire months to taking caffeine or placebo to measure adaptation.
Anyway I definitely agree that diet/exercise/sleep are the big wins. (To that list, I might add: friends?)
Personally I've gotten into a groove where I have a detox week every 6 weeks or so, and don't take stimulants on weekends (unless I'm working). I haven't done any hard experiments with this, but compared to when I was younger without intentional stimulant use, I'm multiple times more productive.
Never heard of that last big win but I'll do some research if I have the time, thanks.
Lots of people seem convinced that you get "full adaptation" to caffeine and that if you take it for months at a time, you're just getting back to baseline. My best guess is that that's never quite true (even without tolerance breaks) though I've never seen convincing proof either way.
Personal experience tells me that it’s not, or at least there’s a significant difference between the me who has abstained from caffeine for a long time (multiple weeks at least) and the me who has that first cup of coffee after a long tolerance break.
It also helps regulate “when” I feel awake, which absent it is hit or miss. Even if caffeine just becomes a requirement for baseline, being able to rapidly hit baseline in the morning, and avoid the post-lunch fatigue is valuable in itself. So long as I get good sleep I can feel normal the whole day with caffeine, whereas without it there’s times of the day I’ll just feel lethargic.
I see a lot of people treat it as a substitute for sleep though, so they end with sleep deprivation —> Consume more caffeine —> Get more sleep deprived —> etc.