Exposing Faulty Studies – On Causation vs. Correlation


I am always stumbling across new studies with these outlandish headings.

Let’s take for instance the following article:

Diets high in meat, eggs and dairy could be as harmful to health as smoking

Wow. That’s really making a claim! I can’t tell you how many times I’ve heard someone make a persuasive statement or share an article based on one of these types of studies – with only reading the headline. This frustrates me. And take a look at this subheading:

People under 65 who eat a lot of meat, eggs and dairy are four times as likely to die from cancer or diabetes, study suggests

Let’s break down what’s going on here.

Based on this headline, I expect tests with meat, eggs, and dairy being the only isolated variable in the controlled group of people. I want to know details about the group tested (age/health/history) and I want it to be a controlled study. I want to see detailed evidence on how the quantity and quality of these variables affect a human’s health. I expect conclusive evidence that people under 65 are more likely to die based on the animal proteins they are consuming. I also want to see the studies about smoking they have performed to make this comparison.

The test group:

  • 6,381 Adults all over the age of 50
  • Broken up into high protein group (over 20% of diet protein), moderate group (10-19%), and low group (less than 10%)

The claimed results:

  • High protein intake is linked to increased cancer, diabetes, and overall mortality
  • Higher protein consumption may be protective for older adults (over 65)
  • Plant-derived proteins are associated with lower mortality than animal-derived proteins

These results make me cringe. So the claim is that high protein is linked to these types of mortality. Out of this whole group of 6,381 people over an 18 year period, there were only 6 deaths. Of these 6 deaths, more came from the range of 50-65 than from over 65. This activated my bullshit detector almost as much as reading the headline of this article. It just doesn’t make sense. The study found that animal proteins (meat, eggs, and dairy) were associated with these deaths and not protein in general (it claims they used both animal proteins and other natural types of proteins in the study). Based on how I understood the study, I don’t see how they can truly link animal proteins in the association with these deaths.

Also, this study never explains the reasoning behind comparing animal proteins with smoking. It appears the creator of the study just added in smoking because people associate it with ill-health. There is no other reason I could find for this to be in the study, let alone the headline.

Here is a quote from the creator of the study:

“People need to switch to a diet where only around nine or ten percent of their calories come from protein, and the ideal sources are plant-based,” Longo told the Guardian. “We are not saying go and do some crazy diet we came up with. If we are wrong, there is no harm done, but if we are right you are looking at an incredible effect that in general is about as bad as smoking.”

This is an illogical recommendation, even based on his study’s findings. The study says that eating animal proteins over the age of 65 is good, but from the range of 50-65 is bad. The creator of the study is telling everyone to consume an amount of protein on the lower end of their test. This, again, makes absolutely no sense to me.

Let’s get into talking about the topic of a controlled study. This does not come across as a controlled study, but more of an observational study. A true controlled study would have all static variables and only change one variable and see the results. It is extremely difficult to perform this on humans with so many factors coming into play. This makes it difficult to trust almost any study on humans. Rats are much easier to control, which is one reason we see so many studies on these animals.

Most studies are either observational studies or controlled studies gone wrong.

Another factor is the age of the control group. Why are you testing only people over 50 years old? Why are you only focused on deaths instead of nutritional data? How are you not taking into effect the additional exposure older people have to sicknesses and other negative-effecting variables? Questionable.

My take-away:

I find this study to be extremely irresponsible. To me. the author is looking only for page-views with this title. This article could have easily been called ” There is no association between protein intake and mortality.” Instead, he chose this current title. Ultimately, my take-away is disregard everything in this article.

Let’s clear it up once and for all

Correlation shows relationships between 2 variables. Ex: how many bathrooms a house has vs. the price of the house. There will be a relationship here – maybe every extra bathroom the house has, the house is worth $4,000 more. However, this does not imply causation. If you add an extra bathroom, your house’s value will not jump $4,000.

Causation is much more difficult to prove in a study. There needs to be isolated variables in a controlled group. Causation is when (A) actually causes (B) to occur.

I’m breaking this down because I see too many people jump to the conclusion that a study proves causation. This is very often not the case. I want people to be wary the next time they see a similar headline.

I always try to check my sources, check the background of the study, and check the data. I also try to make sure the founders of the study have no vested interest in the outcome.

  • Ex: A Coca-Cola study about how Aspartame (sugar substitute) is not harmful to our health.

It is often more dangerous to be misinformed than to not be knowledgeable about a certain topic.

The goal of this process is to at least briefly question information you come across and eliminate being easily fooled. Previously, I was greatly influenced by anything that came from a position of authority. After being proven wrong too many times, I decided to make a change. I have made an effort to self-experiment for a while now, finding my own results instead of relying these types of studies. I find this to be more effective, as everyone is different and there is more poor advice floating around with the recent rise of technology. However, the truth is being exposed as well…you just have to find it.

So even if you decide not to test things out yourself, be wary of the studies you come across.

A better system of evaluation will help you weed through the BS out there.

 “If you see fraud and don’t shout fraud, you are a fraud”. – Nassim Taleb


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s