Misleading headlines fuels the news on nutrition

Wow! I can’t believe its photo Friday again.  We had a great time last week having a few neighbors over for dinner, but forgot to take any photos.  Medically I’m feeling good and just waiting for a few more test results to come in.  I’ll post as soon as they do.

I spend hours researching the web for information on cancer/nutrition.  While searching the web I came across the work of Dr. Peter Attia, M.D.  Here’s a piece off his blog eatingacademy.com that I thought I’d share.  It’s his response to a New York Times article about eating red meat.

The reason Gary Taubes and I founded the Nutrition Science Initiative (NuSI), which we hope to launch this summer is to help the field of nutrition science join the other scientific fields.  Most disciplines of science — such as physics, chemistry, and biology — use something called the Scientific Method to answer questions.

A simple figure of this approach is shown below:

 

The figure is pretty self-explanatory, so let me get to the part where nutrition science is making the biggest mistakes: “Conduct an experiment.”  There is no shortage of observations, questions, or hypotheses in the nutrition science world – so we’re doing well on that front.  It’s that pesky experiment part we’re getting hung up on.   Without doing controlled experiments it is not possible to distinguish the relationship between cause and effect.  [Just a heads up – this is going to be a recurring theme this week.] 

What is an experiment?

There are several types of experiments and they are not all equally effective at determining the cause and effect relationship.  Climate scientists and social economists (like one of my favorites, Steven Levitt, whom I’ve been fortunate enough to spend a day with), for example, often carry out natural experiments.  Why?  Because the “laboratory” they study can’t actually be manipulated in a controlled setting.  For example, when Levitt and his colleagues tried to figure out if swimming pools or guns were more dangerous to children – i.e., Was a child more likely to drown in a house with a swimming pool or be shot by a gun in a home with a gun? – they could only look at historical, or observational, data.  They could not design an experiment to study this question prospectively and in a controlled manner.

How would one design such an experiment?  In a “dream” world you would find, say, 100,000 families and you would split them into two groups – group 1 and group 2.  Group 1 and 2 would be statistically identical in every way once divided.  Because of the size of the population, any differences between them would cancel out (e.g., socioeconomic status, number of kids, parenting styles, geography).  The 50,000 group 1 families would then have a swimming pool installed in their backyard and the 50,000 group 2 families would be given a gun to keep in their house.

For a period of time, say 5 years, the scientists would observe the differences in child death rates from these two causes (accidental drowning and gunshot wounds).  At the conclusion, provided the study was powered appropriately, the scientists would know which was more hazardous to the life of a child, a home swimming pool or a home gun.

Unfortunately questions like this (and the other questions studied by folks like Levitt) can’t be studied in a controlled way. Such studies are just impractical, if not impossible, to do.

Similarly, to rigorously study the anthropogenic CO₂ – climate change hypothesis, for example, we would need another planet earth with the same number of humans, cows, lakes, oceans, and kittens that did NOT burn fossil fuels for 50 years.  But, since these scenarios are never going to happen the folks that carry out natural experiments do the best they can to statistically manipulate data to separate as many confounding factors as possible in every effort to identify the relationship between cause and effect.  Cause and effect.  Say it with me one more time…cause and effect.

Enter the holy grail of experiments: the controlled experiment. In a controlled experiment, as the name suggests, the scientists have control over all variables between the groups (typically what we call a “control” group and a “treatment” group). Furthermore, they study subjects prospectively (rather than backwards looking, or retrospectively) while only changing one variable at a time.  Even a well-designed experiment, if it changes too many variables (for example), prevents the investigator from making the important link: cause and effect.  Remember my silly example from this post a few weeks back?

Imagine a clinical experiment for patients with colon cancer.  One group gets randomized to no treatment (“control group”). The other group gets randomized to a cocktail of 14 different chemotherapy drugs, plus radiation, plus surgery, plus hypnosis treatments, plus daily massages, plus daily ice cream sandwiches, plus daily visits from kittens (“treatment group”).  A year later the treatment group has outlived the control group, and therefore the treatment has worked.  But how do we know EXACTLY what led to the survival benefit?  Was it 3 of the 14 drugs?  The surgery?  The kittens? We cannot know from this experiment.  The only way to know for certain if a treatment works is to isolate it from all other variables and test it in a randomized prospective fashion.

As you can see, even doing a prospective controlled experiment is not enough, like the one above, if you fail to design the trial correctly.  Technically, the fictitious experiment I describe above is not “wrong,” unless someone – for example, the scientist who carried out the trial or the newspapers who report on it – misrepresented it.

If the New York Times and CNN reported the following: New study proves that kittens cure cancer!, would it be accurate? Not even close.  Sadly, most folks would never read the actual study to understand why this bumper-sticker conclusion is categorically false. Sure, it is possible, based on this study, that kittens can cure cancer.  But the scientists in this hypothetical study have wasted a lot of time and money if their goal was to determine if kittens could cure cancer.  The best thing this study did was to reiterate a hypothesis. Nothing more.  In other words, this experiment (even assuming it was done perfectly well from a technical standpoint) learned nothing other than the combination of 20 interventions was better than none because of an experimental design problem As you can see; even doing a prospective controlled experiment is not enough, like the one above, if you fail to design the trial correctly.  Technically, the fictitious experiment I describe above is not “wrong,” unless someone – for example, the scientist who carried out the trial or the newspapers who report on it – misrepresented it.

So what does all of this have to do with eating red meat?

In effect, I’ve already told you everything you need to know.  I’m not actually going to spend any time dissecting the actual study published last week that led to the screaming headlines about how red meat eaters are at greater risk of death from all causes (yes, “all causes,” according to this study) because it’s already been done a number of times by others this week alone.  Three critical posts on this specific paper can be found here, here, and here.
I can’t suggest strongly enough that you read them all if you really want to understand the countless limitations of this particular study, and why its conclusion should be completely disregarded.   If you want bonus points, read the paper first, see if you can understand the failures of it, then check your “answer” against these post.  As silly as this sounds, it’s actually the best way to know if you’ve really internalized what I’m describing.
Now, I know what you might be thinking:  Oh, come on Peter, you’re just upset because this study says something completely opposite to what you say!
Not so.  In fact, I have the same criticism of similarly conducted studies that “find” conclusions I agree with.  For example, on the exact same day the red meat study was published online (March 12, 2012) in the journal Archives of Internal Medicine, the same group of authors from Harvard’s School of Public Health published another paper in the journal Circulation.  This second paper reported on the link between sweetened beverage consumption and heart disease, which “showed” that consumption of sugar-sweetened beverages increased the risk of heart disease in men. [On another day I will give you my thoughts on why the media chose to report on the red meat study rather than the sugar study, despite them both coming out on the same day, by the same authors, from the same prestigious university.]
I agree that sugar-sweetened beverages increase the risk of heart disease (not just in men, of course, but in women, too) along with a whole host of other diseases like cancer, diabetes, and Alzheimer’s disease.  But, the point remains that this study does nothing to add to the body of evidence implicating sugar because it was not a controlled experiment.  This was a waste of time and money reiterating an already strong hypothesis. That effort should have been spent on a controlled experiment.

This problem is actually rampant in nutrition

We’ve got studies “proving” that eating more grains protect men from colon cancer, that light-to-moderate alcohol consumption reduces the risk of stroke in women, and that low levels of polyunsaturated fats, including omega-6 fats, increase the risk of hip fractures in women.  Are we to believe these studies? They sure sound authoritative, and the way the press reports on them it’s hard to argue, right?
How are these studies typically done?
Let’s talk nuts and bolts for a moment.  I know some of you might already be zoning out with the detail, but if you want to understand why and how you’re being misled, you actually need to “double-click” (i.e., get one layer deeper) a bit.  What the researchers do in these studies is follow a cohort of several tens of thousands of people — nurses, health care professionals, AARP members, etcetera — and they ask them what they eat with a food frequency questionnaire (FFQ) that is known to be almost fatally flawed in terms of its ability to accurately acquire data about what people really eat.  Next, the researchers correlate disease states, morbidity, and maybe even mortality with food consumption, or at least reported food consumption (which is NOT the same thing). So, the end products are correlations — eating food X is associated with a gain of Y pounds, for example. Or eating red meat three-times a week is associated with a 50% increase in the risk of death from falling pianos or heart attacks or cancer.
The catch, of course, is that correlations hold no (i.e., ZERO) causal information.  Just because two events occur in step does not mean you can conclude one causes the other.  Often in these articles you’ll hear people give the obligatory, “correlation doesn’t necessarily imply causality.” But saying that suggests a slight disconnect from the real issue. A more accurate statement is “correlation does not imply causality” or “correlations contain no causal information.”
So what explains the findings of studies like this (and virtually every single one of these studies coming out of massive health databases like Harvard’s)?
For starters, the foods associated with weight gain (or whichever disease they are studying) are also the foods associated with “bad” eating habits in the United States — french fries, sweets, red meat, processed meat, etc.  Foods associated with weight loss are those associated with “good” eating habits –fruit, low-fat products, vegetables, etc.  But, that’s not because these foods cause weight gain or loss, it’s because they are markers for who the people who eat a certain way and live a certain way.
Think about who eats a lot of french fries (or a lot of processed meats). They are people who eat at fast food restaurants regularly (or in the case of processed meats, people who are more likely to be economically disadvantaged).  So, eating lots of french fries, hamburgers, or processed meats is generally a marker for people with poor eating habits, which is often the case when people are less economically advantaged and less educated than people who buy their food fresh at the local farmer’s market or at Whole Foods (or Whole Paycheck as I like to call it).  Furthermore, people eating more french fries and red meat are less health conscious in general (or they wouldn’t be eating french fries and red meat – remember, those of us who do eat red meat regularly are in the slim minority of health conscious folks).  These studies are rife with methodological flaws, and I could devote an entire Ph.D. thesis to this topic alone.

What should we do about this?

I’m guessing most of you – and most physicians and policy makers in the United States for that matter – are not actually browsing the American Journal of Epidemiology (where one can find studies like this all day long).  But occasionally, like last week, the New York Times, Wall Street Journal, Washington Post, CBS, ABC, CNN, and everyone else gets wind of a study like the now-famous red meat study and comments in a misleading fashion.  Health policy in the United States – and by extension much of the world – is driven by this. It’s not a conspiracy theory, by the way.  It’s incompetence.  Big difference.  Keep Hanlon’s razor in mind – Never attribute to malice that which is adequately explained by stupidity.
This behavior, in my opinion, is unethical and the journalists who report on it (along with the scientists who stand by not correcting them) are doing humanity no favors.
I do not dispute that observational epidemiology has played a role in helping to elucidate “simple” linkages in health sciences (e.g., a great example is that of contaminated water and cholera or the linkage between scrotal cancer and chimney sweeps).  However, multifaceted or highly complex pathways (e.g., cancer, heart disease) rarely pan out, unless the disease is virtually unheard of without the implicated cause.  A great example of this is the elucidation of the linkage between small-cell lung cancer and smoking – we didn’t need a controlled experiment to link smoking to this particular variant of lung cancer because nothing else has ever been shown to even approach the rate of this type of lung cancer the way smoking has (by a factor of over 100). As a result of this unique fact, Richard Doll and Austin Bradford Hill were able to design a clever observational analysis to correctly identify the cause and effect linkage between tobacco and lung cancer.  But this sort of example is actually the exception and not the rule when it comes to epidemiology.

I trust by now you have a better understanding of why the “science” of nutrition is so bankrupt.  It is based on almost a complete reliance on these observational studies. Virtually every piece of nutritional dogma we suffer from today stems from – you guessed it – an observational study.   Whether it’s Ancel Keys’ observations and correlations of saturated fat intake and heart disease in his famous Seven Countries Study, which “proved” saturated fat is harmful or Denis Burkitt’s observation that people in Africa ate more fiber than people in England and had less colon cancer “proving” that eating fiber is the key to preventing colon cancer, virtually all of the nutritional dogma we are exposed to has not actually been scientifically tested.   Perhaps the most influential current example of observational epidemiology is the work of T. Colin Campbell, lead author of The China Study, which claims, “the science is clear” and “the results are unmistakable.”  Really?  Not if you define science the way scientists do.  This doesn’t mean Colin Campbell is wrong (though I wholeheartedly believe he is wrong on about 75% of what he says based on current data).  It means he has not done any real science to advance the discussion and hypotheses he espouses.  If you want to read the most remarkable and detailed critiques of this work, please look no further than here (Denise Minger) and here (Michael Eades).
I can only imagine the contribution to mankind Dr. Campbell could have given had he spent the same amount of time and money doing actual scientific experiments to elucidate the impact of dietary intake and chronic disease. [For example, Campbell would have designed a prospective study following subjects randomized to one of two different types of diets for 10 years: plant-based and animal-based, but with all other factors controlled for.] This is one irony of enormous observational epidemiology studies.  Not only are they of little value, in a world of finite resources, they actually detracts from real science being done.
It’s time to start doing real science.  We do actually have the luxury of doing controlled experiments (in fact, several have been done, they just tend to get ignored), in contrast to climate scientists and social economists.  Isn’t it time we stop guessing and find out what foods really increase our risk of dying prematurely? That’s exactly our intent with NuSI, but more on that another day…

.