SEIZE THE $50 BILLION SITE-OF-CARE SHIFT OPPORTUNITY
Get the tools, data, and insights to drive growth.
Learn more

 We are pausing publication of The Daily Briefing out of respect for the tragic passing of Brian Thompson. We will resume publication of this daily newsletter in the coming days.

Library

| Daily Briefing

Why so many diet studies are bunk


Conflicting studies have almost become the norm in nutrition research, and that in large part is thanks to a method known as data dredging, or p-hacking, Anahad O'Connor reports for the New York Times.

Cheat sheet series: Evidence-based medicine 101

The problem of 'data dredging'

"Data dredging" is a process by which researchers run exhaustive analyses on various data sets to discover patterns that can be deemed statistically significant without first coming up with a hypothesis. Critics of the process argue that it's equivalent to "casting a wide net and then creating a hypothesis to support whatever cherry-picked findings seem interesting," O'Connor writes.

Studies developed using data-dredging can lead to important findings, O'Connor writes, but they also have significant limitations. For example, these studies can't prove a causal relationship, and they can often uncover spurious or coincidental correlations that may grab headlines but don't have any real-world significance, O'Connor writes.

Ivan Oransky, co-founder of Retraction Watch and a teacher of medical journalism at New York University (NYU), said data dredging is "a really serious problem," and can throw "into question the very statistical basis of what we're reading as science journalists and as the public."

Why data dredging is rampant in nutrition research

Critics have argued that data dredging is rampant throughout nutrition research, leading to misleading, unscientific, and conflicting studies in the field, O'Connor writes.

For example, it's common to see a nutritional study that claims a specific food or beverage can lead to an increased risk of cancer—only to see a separate study a week later that shows the food or beverage has health benefits.

John Ioannidis, chair of disease prevention at Stanford University, in 2012 co-authored a study that showcased just how misleading data dredging can be. The researchers randomly selected 50 recipes from a cookbook and found that 80% of the ingredients had been linked to either an increased or decreased risk of cancer in a number of studies.

In fact, a single ingredient was frequently found to be the subject of questionable cancer claims in at least 10 studies, and the majority of those studies "were based on weak statistical evidence," the study found.

Marion Nestle, a professor of nutrition, food studies, and public health at NYU, explained that researchers often feel significant pressure to publish papers, which could be one reason why many resort to data dredging. "You can't get a job if you don't have papers," she said. "I see this at my university. We expect assistant professors to be hired with already a record of scholarship."

Calls to end the practice

Many researchers have called for reforms to end data dredging, arguing that researchers should publicly register their protocols before completing research to reduce the potential for data dredging.

They also say researchers should share their raw data for transparency and focus more on large, randomized controlled trials rather than cherry-picking findings from large, observational data sets. "You can analyze observational studies in very different ways and, depending on what your belief is … you can get some very dramatic patterns," Ioannidis said.

The onus for reducing misleading coverage isn't just on researchers, however, O'Connor writes, but science journalists as well. Reporters are often encouraged to produce articles that will generate a lot of clicks, which in turn can put pressure on researchers to put out studies with flashy findings.

Ted Kyle, an obesity expert who runs a health site called ConscienHealth, said, "[W]e're all drinking from the same well, and we're all contributing to poisoning the water. At every step along the way there are folks who are culpable. I would suspect that we're all complicit" (O'Connor, New York Times, 9/29).

Learn more about why study design matters

Been awhile since your last statistics class? It can be difficult to judge the quality of studies, the significance of data, or the importance of new findings when you don't know the basics.

Download our cheat sheets to get a quick, one-page refresher on some of the foundational components of evidence-based medicine.

Get all 6 Cheat Sheets


SPONSORED BY

INTENDED AUDIENCE

AFTER YOU READ THIS

AUTHORS

TOPICS

INDUSTRY SECTORS

Don't miss out on the latest Advisory Board insights

Create your free account to access 1 resource, including the latest research and webinars.

Want access without creating an account?

   

You have 1 free members-only resource remaining this month.

1 free members-only resources remaining

1 free members-only resources remaining

You've reached your limit of free insights

Become a member to access all of Advisory Board's resources, events, and experts

Never miss out on the latest innovative health care content tailored to you.

Benefits include:

Unlimited access to research and resources
Member-only access to events and trainings
Expert-led consultation and facilitation
The latest content delivered to your inbox

You've reached your limit of free insights

Become a member to access all of Advisory Board's resources, events, and experts

Never miss out on the latest innovative health care content tailored to you.

Benefits include:

Unlimited access to research and resources
Member-only access to events and trainings
Expert-led consultation and facilitation
The latest content delivered to your inbox
AB
Thank you! Your updates have been made successfully.
Oh no! There was a problem with your request.
Error in form submission. Please try again.