I think there are two reasonable ways to do a Bayesian analysis on this data:
1. Calculate the likelihood ratio: the probability of getting this result given that the mean equals the sample mean, divided by the probability of getting this result given that the mean equals zero.
2. If you just want to know whether the effect is positive or zero (or negative), then your prior needs to include a point mass at zero. I did this for Luis Costigan's data and found that the outcome was evidence in *favor* of a zero effect (53% posterior vs. 50% prior). Your results were about as strong as his so my guess is your data is also evidence in favor of zero effect.
Believe it or not, I actually started writing a little rant about that in the post but I decided no one would care and deleted it. I agree! My prior is that there's a significant chance theanine does something *very* close to zero, which just isn't reflected in the current analysis.
I didn't do this mostly because, for better or worse, this kind of prior without the point mass is standard and I didn't want to give the theanine fanatics anything to complain about.
Does theanine have an effect? One strategy is to keep increasing the dosage until it has an obvious effect. Presumably this doesn't work and everyone believes that the effect saturates. But if the effect saturates, maybe it already saturates at cup of tea levels and that is the difference between tea and coffee.
If you think it's about caffeine, you should get caffeine pills, to distinguish coffee is bad from tea is good.
I thought that might be possible. That was one of the reasons for switching from tea to coffee for this experiment.
But after seeing how strong the placebo effect is, I'm now starting to wonder if caffeine really causes anxiety at all! I'd like to do a blind trial of caffeine vs. placebo and see if even that makes a difference. (Taking some kind of coffee extract vs. tea extract in capsule form would also be interesting.)
Lovely. I'm considering doing a self-experiment with caffeine / decaf coffee, just to get a benchmark on how reliable self-reporting is on a substance with a proven effect.
Oh cool, exactly what I had in mind. A bit suspicious because there are no steps to eliminate flavor recognition. If I taste a decaf now it is pretty distinct. My plan is to have a "reset period" in which I drink a half-decaf powder mix. Then in the blind experiment I will be able to tell them apart, but hopefully not recognize which is decaf vs caf.
This assumes that decaffeination removes only caffeine and not other things ... and it almost certainly removes other things because most decaf tastes different - which could be because it's less bitter due to less caffeine, but I don't think that's it
This may or may not help: my doctor recommended theanine to help me sleep more restfully, at a time when I was working up to 60-70 hours a week. She specifically said to buy the Science Naturals brand with "Relora".
It had an effect; I slept better, waking up less or at least noticing it less. I assumed Relora was marketing more that an active substance and bought normal theanine supplements - and the effect went away. My wife uses a CPAP machine and started taking the Relora-branded product; she slept better, as reported by the CPAP machine's reported stats.
Unfortunately. this is a sample size of 2 and not blind, so good chance it's worthless info. However, it does suggest testing at the end of the day, to see if it has a calm-down effect, rather than as a coffee antidote?
Anecdotally n=1 on a some things... I also tried theanine, had no noticeable effect.
5000iu Vitamin D daily gives me a steady "greater wellbeing" feeling, noticeable a few days after starting.
About 15-ish years ago i took lysine before going to clubs to reduce my nervousness in meeting people. But it reduced my internal drive so much i didn't talk, didn't move and didn't care.... had to stop because then what's the point in leaving home. Can there a "psychopath" amino to maintain energy and focus while cooling emotion? (Asking for a friend)
I really just want to thank you for these theanine self-experiments. I took the stuff for years (stacked with caffeine from coffee) and I really wanted to believe it was the nootropic powerhouse it's been claimed to be, but I could never notice any difference from the combo vs coffee alone. Really pleased that (i) other people have the same doubts, and (ii) you set out to measure this and most importantly share the results.
I identified two caveats to your experiment I'd like to mention:
1. As some others here have mentioned, it could be that it has anxiolytic effects for some individuals but not others.
2. The more important point I think: maybe our ability to rate our level of perceived anxiety at any one moment is not reliable, or at least the variance in our ability to do so is larger than the effect size of theanine. This may sound odd as you might say anxiety is itself a perception so it's only our felt experience of it that matters, therefore what you rate it as *is* what it is. But I would argue that this needn't be the case. Additionally, theanine could reduce the range of physiological symptoms of anxiety (not only emotional/psychological symptoms), like increased heart rate, increased blood pressure, etc. To this end it would be interesting if someone would do a similar experiment but measure physiological metrics in addition to perceived anxiety.
I totally agree with both of these caveats. (https://dynomight.net/theanine/#thoughts) In fact, for what it's worth, I think I would rate myself as somewhere on the lower end of the bell curve in terms of my ability to rate my perceived anxiety.
I think there's a related issue, which is: Even if you can tell your perceived anxiety, how well are you able to identify the causes for it. Can you tell if it's a lack of sleep vs. caffeine vs. relationship stress? I think if you're good at this, then you'll have an easier time identifying the "signal" that theanine provides in all the "noise" from other causes. And again, I'd rate myself on the lower end in terms of this ability. :)
I used to naively subscribe to the "anxiety or depression *is* your perception, so self reports can't be wrong" view, but after suffering some severe anxiety for an extended period (and now being back to normal for several years), I've changed my mind. At least in my case where it changed very slowly, even though the magnitude was rather large between then and now, I didn't really notice it and probably would have rated my base rate og anxiety pretty much the same the entire time. But looking back at specific episodic memories and *really* remembering what it was like, the difference is night and day. I think the problem is that it's so hard to remember feelings (at least for me?), and you don't really have any reference point except your current emotional state at any one time.
How about different people continue performing small sample size experiments until someone finds with >95% confidence that Theanine has a meaningful positive effect?
More seriously, if the effect is just placebo, maybe this sort of statistical trickery would have the positive effect of making theanine work, because people believe theanine works, because a study tells them it works, because it does work, because people believe it works, because...
Ha. On the one hand, yes. On the other hand, when I did exactly the same analysis Luis did with my (larger) dataset, I also got an 80% posterior probability. If more and more people keep reproducing that, don't we eventually have to believe it? Maybe it depends on how much we worry about publication bias...
That's fair, but if the benefit is only a 1-2% reduction in anxiety, I'm not sure it's worth it even if it was scientifically proven to an arbitrarily high degree of confidence to reduce anxiety. That's also assuming no tolerance develops, which it probably does with repeated use, if there is an effect at all.
For a 1-2% reduction, I think basically anything that commonly is associated with good mental health will probably work just as well or better, like taking a short walk, sitting in sunlight, exercise, or something benign like being told "Nice hat!" by a stranger.
True, true. I think Luis's mean effect (0.8 points on a 10 point scale) is decent/respectable. But my effect (0.08 points on a 5 point scale) is puny even if real and could probably be achieved with just about anything.
> "Nice hat!" by a stranger.
For me this is at least 0.5 points on a 1-5 scale. :)
My observation is that for roughly equivalent amounts of caffeine, coffee has a much higher chance of making me feel "jittery" or "wired" than black tea (neither tea nor coffee have much effect on my anxiety level). Taking theanine with coffee seems to help somewhat. So I think there's something in coffee (and not in tea) other than caffeine that affects me and theanine partially counteracts it; to test this, I should try (double-blind) comparison of decaf and regular coffee, but the bad effects have become bad enough that I've mostly quit coffee (and I haven't found a decaf that I like).
An additional observation is that black tea affects me more than green tea or matcha (for theoretically equivalent amounts of caffeine). There are a large number of variables and I don't see an easy way to validate this. (how much and what quality of sleep I've had, when I woke up, other stressors, the weather, the season, etc. etc.)
I also suspect that this is very much a "depends on the individual" kind of thing - I used to drink 4 coffees a day without a problem (or so it seemed) but my sensitivity has changed a lot over the years (and I'm now retired). I also suspect that I'm a slow metabolizer of caffeine but I haven't done any blood or genetic tests to confirm this.
I also strongly experience that caffeine makes me much more nervous than tea, although I have no idea why this should be the case. Many people suggest it might be the theanine in tea, but I find this very unlikely given how small it is (in most teas).
(Given my attitude in this post, maybe I should attempt a blind experiment with extracts or something to see if coffee REALLY makes me more anxious...)
Supposedly, there are 1,000 compounds in coffee; so extracting the substance(s) that affects both you and me could be a challenge.
From reading the literature, theanine supplements are 10x as much theanine compared to tea, so there's probably something else going on (unless theanine has decreasing effect with higher dosage - this is apparently the case with melatonin, for example).
One other observation: matcha supposedly has more theanine than other teas (and more caffeine - the legend is that Bodhidharma feel asleep meditating, threw his eyelids on the ground, and a tea plant sprang up), yet it doesn't make me jittery. And I know one person who says that matcha reduces the trembling in his hands. So, perhaps there's yet another substance that's in matcha (maybe in green tea) and not in black tea or coffee.
[One more thing: ~10% of people (especially children) who take Benadryl react to it by becoming restless and excitable rather than drowsy ... clearly there are some highly individualistic reactions to some substances.]
Yeah, this is my experience as well. The teas that contain the highest amounts of theanine (e.g. matcha) seem to be particularly relaxing. But the amount of theanine is still apparently quite small (maybe 25 mg/cup?) and even taking 400 mg here seemed to make little difference. I don't know how to explain any of this!
In general. I don't care much about either p-values or Bayesian probabilities. What I do care about are effect sizes.
Even in Costigan's experiment reporting an 83% chance theanine reduced anxiety, the difference between 4.2 and 5.0 on a 10-point scale is small enough that my conclusion would be, "Why bother?"
The most interesting treatments or supplements are those for which the effects are so obvious that you don't even need to do the analysis to see them -- just plot the data and look. Sadly, such treatments are hard to come by.
Totally agree re: effect sizes. That being said, if theanine really did make a 0.8 point difference on a 10 point scale, I would personally consider that worth it, and probably take it every day! Unfortunately, the best estimate in my experiment is 0.08 points on a 5 point scale, which even I think is pretty dismal.
The implementation of Bayesian methodology was kind of bugging me but I wasn't sure how to improve on it. I spent too long thinking about it and ended up writing some comments on the Manifold market: https://manifold.markets/LuisCostigan/nof1-blinded-experiment-will-210mg#wgvd60w806n
I think there are two reasonable ways to do a Bayesian analysis on this data:
1. Calculate the likelihood ratio: the probability of getting this result given that the mean equals the sample mean, divided by the probability of getting this result given that the mean equals zero.
2. If you just want to know whether the effect is positive or zero (or negative), then your prior needs to include a point mass at zero. I did this for Luis Costigan's data and found that the outcome was evidence in *favor* of a zero effect (53% posterior vs. 50% prior). Your results were about as strong as his so my guess is your data is also evidence in favor of zero effect.
Believe it or not, I actually started writing a little rant about that in the post but I decided no one would care and deleted it. I agree! My prior is that there's a significant chance theanine does something *very* close to zero, which just isn't reflected in the current analysis.
I didn't do this mostly because, for better or worse, this kind of prior without the point mass is standard and I didn't want to give the theanine fanatics anything to complain about.
I'm not sure why I forgot to put my data online, but in case you're interested: https://dynomight.net/img/theanine-2/data.csv
Does theanine have an effect? One strategy is to keep increasing the dosage until it has an obvious effect. Presumably this doesn't work and everyone believes that the effect saturates. But if the effect saturates, maybe it already saturates at cup of tea levels and that is the difference between tea and coffee.
If you think it's about caffeine, you should get caffeine pills, to distinguish coffee is bad from tea is good.
I thought that might be possible. That was one of the reasons for switching from tea to coffee for this experiment.
But after seeing how strong the placebo effect is, I'm now starting to wonder if caffeine really causes anxiety at all! I'd like to do a blind trial of caffeine vs. placebo and see if even that makes a difference. (Taking some kind of coffee extract vs. tea extract in capsule form would also be interesting.)
Lovely. I'm considering doing a self-experiment with caffeine / decaf coffee, just to get a benchmark on how reliable self-reporting is on a substance with a proven effect.
Please let me know your results.
See also this (very nice) experiment: https://stephenmalina.com/post/2020-03-01-coffee-rct/
Oh cool, exactly what I had in mind. A bit suspicious because there are no steps to eliminate flavor recognition. If I taste a decaf now it is pretty distinct. My plan is to have a "reset period" in which I drink a half-decaf powder mix. Then in the blind experiment I will be able to tell them apart, but hopefully not recognize which is decaf vs caf.
The brute-force solution would be to always drink decaf, but either take caffeine pills or placebo.
This assumes that decaffeination removes only caffeine and not other things ... and it almost certainly removes other things because most decaf tastes different - which could be because it's less bitter due to less caffeine, but I don't think that's it
This may or may not help: my doctor recommended theanine to help me sleep more restfully, at a time when I was working up to 60-70 hours a week. She specifically said to buy the Science Naturals brand with "Relora".
It had an effect; I slept better, waking up less or at least noticing it less. I assumed Relora was marketing more that an active substance and bought normal theanine supplements - and the effect went away. My wife uses a CPAP machine and started taking the Relora-branded product; she slept better, as reported by the CPAP machine's reported stats.
Unfortunately. this is a sample size of 2 and not blind, so good chance it's worthless info. However, it does suggest testing at the end of the day, to see if it has a calm-down effect, rather than as a coffee antidote?
16 months is super hardcore.
Anecdotally n=1 on a some things... I also tried theanine, had no noticeable effect.
5000iu Vitamin D daily gives me a steady "greater wellbeing" feeling, noticeable a few days after starting.
About 15-ish years ago i took lysine before going to clubs to reduce my nervousness in meeting people. But it reduced my internal drive so much i didn't talk, didn't move and didn't care.... had to stop because then what's the point in leaving home. Can there a "psychopath" amino to maintain energy and focus while cooling emotion? (Asking for a friend)
I really just want to thank you for these theanine self-experiments. I took the stuff for years (stacked with caffeine from coffee) and I really wanted to believe it was the nootropic powerhouse it's been claimed to be, but I could never notice any difference from the combo vs coffee alone. Really pleased that (i) other people have the same doubts, and (ii) you set out to measure this and most importantly share the results.
I identified two caveats to your experiment I'd like to mention:
1. As some others here have mentioned, it could be that it has anxiolytic effects for some individuals but not others.
2. The more important point I think: maybe our ability to rate our level of perceived anxiety at any one moment is not reliable, or at least the variance in our ability to do so is larger than the effect size of theanine. This may sound odd as you might say anxiety is itself a perception so it's only our felt experience of it that matters, therefore what you rate it as *is* what it is. But I would argue that this needn't be the case. Additionally, theanine could reduce the range of physiological symptoms of anxiety (not only emotional/psychological symptoms), like increased heart rate, increased blood pressure, etc. To this end it would be interesting if someone would do a similar experiment but measure physiological metrics in addition to perceived anxiety.
I totally agree with both of these caveats. (https://dynomight.net/theanine/#thoughts) In fact, for what it's worth, I think I would rate myself as somewhere on the lower end of the bell curve in terms of my ability to rate my perceived anxiety.
I think there's a related issue, which is: Even if you can tell your perceived anxiety, how well are you able to identify the causes for it. Can you tell if it's a lack of sleep vs. caffeine vs. relationship stress? I think if you're good at this, then you'll have an easier time identifying the "signal" that theanine provides in all the "noise" from other causes. And again, I'd rate myself on the lower end in terms of this ability. :)
I used to naively subscribe to the "anxiety or depression *is* your perception, so self reports can't be wrong" view, but after suffering some severe anxiety for an extended period (and now being back to normal for several years), I've changed my mind. At least in my case where it changed very slowly, even though the magnitude was rather large between then and now, I didn't really notice it and probably would have rated my base rate og anxiety pretty much the same the entire time. But looking back at specific episodic memories and *really* remembering what it was like, the difference is night and day. I think the problem is that it's so hard to remember feelings (at least for me?), and you don't really have any reference point except your current emotional state at any one time.
How about different people continue performing small sample size experiments until someone finds with >95% confidence that Theanine has a meaningful positive effect?
More seriously, if the effect is just placebo, maybe this sort of statistical trickery would have the positive effect of making theanine work, because people believe theanine works, because a study tells them it works, because it does work, because people believe it works, because...
Ha. On the one hand, yes. On the other hand, when I did exactly the same analysis Luis did with my (larger) dataset, I also got an 80% posterior probability. If more and more people keep reproducing that, don't we eventually have to believe it? Maybe it depends on how much we worry about publication bias...
That's fair, but if the benefit is only a 1-2% reduction in anxiety, I'm not sure it's worth it even if it was scientifically proven to an arbitrarily high degree of confidence to reduce anxiety. That's also assuming no tolerance develops, which it probably does with repeated use, if there is an effect at all.
For a 1-2% reduction, I think basically anything that commonly is associated with good mental health will probably work just as well or better, like taking a short walk, sitting in sunlight, exercise, or something benign like being told "Nice hat!" by a stranger.
True, true. I think Luis's mean effect (0.8 points on a 10 point scale) is decent/respectable. But my effect (0.08 points on a 5 point scale) is puny even if real and could probably be achieved with just about anything.
> "Nice hat!" by a stranger.
For me this is at least 0.5 points on a 1-5 scale. :)
In lieu of Theanine for improving mood, please reference;
"The Dynomight Internet Newsletter is definitely in my top ten favorite blogs." - Sol Hando
♡
My observation is that for roughly equivalent amounts of caffeine, coffee has a much higher chance of making me feel "jittery" or "wired" than black tea (neither tea nor coffee have much effect on my anxiety level). Taking theanine with coffee seems to help somewhat. So I think there's something in coffee (and not in tea) other than caffeine that affects me and theanine partially counteracts it; to test this, I should try (double-blind) comparison of decaf and regular coffee, but the bad effects have become bad enough that I've mostly quit coffee (and I haven't found a decaf that I like).
An additional observation is that black tea affects me more than green tea or matcha (for theoretically equivalent amounts of caffeine). There are a large number of variables and I don't see an easy way to validate this. (how much and what quality of sleep I've had, when I woke up, other stressors, the weather, the season, etc. etc.)
I also suspect that this is very much a "depends on the individual" kind of thing - I used to drink 4 coffees a day without a problem (or so it seemed) but my sensitivity has changed a lot over the years (and I'm now retired). I also suspect that I'm a slow metabolizer of caffeine but I haven't done any blood or genetic tests to confirm this.
I also strongly experience that caffeine makes me much more nervous than tea, although I have no idea why this should be the case. Many people suggest it might be the theanine in tea, but I find this very unlikely given how small it is (in most teas).
(Given my attitude in this post, maybe I should attempt a blind experiment with extracts or something to see if coffee REALLY makes me more anxious...)
Supposedly, there are 1,000 compounds in coffee; so extracting the substance(s) that affects both you and me could be a challenge.
From reading the literature, theanine supplements are 10x as much theanine compared to tea, so there's probably something else going on (unless theanine has decreasing effect with higher dosage - this is apparently the case with melatonin, for example).
One other observation: matcha supposedly has more theanine than other teas (and more caffeine - the legend is that Bodhidharma feel asleep meditating, threw his eyelids on the ground, and a tea plant sprang up), yet it doesn't make me jittery. And I know one person who says that matcha reduces the trembling in his hands. So, perhaps there's yet another substance that's in matcha (maybe in green tea) and not in black tea or coffee.
[One more thing: ~10% of people (especially children) who take Benadryl react to it by becoming restless and excitable rather than drowsy ... clearly there are some highly individualistic reactions to some substances.]
Yeah, this is my experience as well. The teas that contain the highest amounts of theanine (e.g. matcha) seem to be particularly relaxing. But the amount of theanine is still apparently quite small (maybe 25 mg/cup?) and even taking 400 mg here seemed to make little difference. I don't know how to explain any of this!
In general. I don't care much about either p-values or Bayesian probabilities. What I do care about are effect sizes.
Even in Costigan's experiment reporting an 83% chance theanine reduced anxiety, the difference between 4.2 and 5.0 on a 10-point scale is small enough that my conclusion would be, "Why bother?"
The most interesting treatments or supplements are those for which the effects are so obvious that you don't even need to do the analysis to see them -- just plot the data and look. Sadly, such treatments are hard to come by.
I agree completely with your final sentence.
Totally agree re: effect sizes. That being said, if theanine really did make a 0.8 point difference on a 10 point scale, I would personally consider that worth it, and probably take it every day! Unfortunately, the best estimate in my experiment is 0.08 points on a 5 point scale, which even I think is pretty dismal.
Maybe zembrin would be a good follow up? https://www.astralcodexten.com/p/nootropics-survey-2020-results
Good suggestion. I'm intrigued, but I think I'm too conservative to mess around with a possible SSRI.