Conflicting studies have almost become the norm in nutrition research, and that in large part is thanks to a method known as data dredging, or p-hacking, Anahad O'Connor reports for the New York Times.

Cheat sheet series: Evidence-based medicine 101

The problem of 'data dredging'

"Data dredging" is a process by which researchers run exhaustive analyses on various data sets to discover patterns that can be deemed statistically significant without first coming up with a hypothesis. Critics of the process argue that it's equivalent to "casting a wide net and then creating a hypothesis to support whatever cherry-picked findings seem interesting," O'Connor writes.

Understand the wellness spectrum

Studies developed using data-dredging can lead to important findings, O'Connor writes, but they also have significant limitations. For example, these studies can't prove a causal relationship, and they can often uncover spurious or coincidental correlations that may grab headlines but don't have any real-world significance, O'Connor writes.

Ivan Oransky, co-founder of Retraction Watch and a teacher of medical journalism at New York University (NYU), said data dredging is "a really serious problem," and can throw "into question the very statistical basis of what we're reading as science journalists and as the public."

Why data dredging is rampant in nutrition research

Critics have argued that data dredging is rampant throughout nutrition research, leading to misleading, unscientific, and conflicting studies in the field, O'Connor writes.

For example, it's common to see a nutritional study that claims a specific food or beverage can lead to an increased risk of cancer—only to see a separate study a week later that shows the food or beverage has health benefits.

John Ioannidis, chair of disease prevention at Stanford University, in 2012 co-authored a study that showcased just how misleading data dredging can be. The researchers randomly selected 50 recipes from a cookbook and found that 80% of the ingredients had been linked to either an increased or decreased risk of cancer in a number of studies.

In fact, a single ingredient was frequently found to be the subject of questionable cancer claims in at least 10 studies, and the majority of those studies "were based on weak statistical evidence," the study found.

Marion Nestle, a professor of nutrition, food studies, and public health at NYU, explained that researchers often feel significant pressure to publish papers, which could be one reason why many resort to data dredging. "You can't get a job if you don't have papers," she said. "I see this at my university. We expect assistant professors to be hired with already a record of scholarship."

Calls to end the practice

Many researchers have called for reforms to end data dredging, arguing that researchers should publicly register their protocols before completing research to reduce the potential for data dredging.

They also say researchers should share their raw data for transparency and focus more on large, randomized controlled trials rather than cherry-picking findings from large, observational data sets. "You can analyze observational studies in very different ways and, depending on what your belief is … you can get some very dramatic patterns," Ioannidis said.

The onus for reducing misleading coverage isn't just on researchers, however, O'Connor writes, but science journalists as well. Reporters are often encouraged to produce articles that will generate a lot of clicks, which in turn can put pressure on researchers to put out studies with flashy findings.

Ted Kyle, an obesity expert who runs a health site called ConscienHealth, said, "[W]e're all drinking from the same well, and we're all contributing to poisoning the water. At every step along the way there are folks who are culpable. I would suspect that we're all complicit" (O'Connor, New York Times, 9/29).

Learn more about why study design matters

Been awhile since your last statistics class? It can be difficult to judge the quality of studies, the significance of data, or the importance of new findings when you don't know the basics.

Download our cheat sheets to get a quick, one-page refresher on some of the foundational components of evidence-based medicine.

• Evidence-based practice (EBP)
• Observational studies
• Randomized control trials (RCT's)
• Systematic reviews
• Meta-analyses
• Statistical significance

Get all 6 Cheat Sheets