The U.S. system for awarding medical research grants through NIH "in many ways … may be failing" to "fund work that spurs innovation and fosters research careers" by favoring proposals for low-risk research and those submitted by more experienced researchers, Aaron Carroll writes in the New York Times.
After proposals are submitted to NIH, Carroll explains, they are sorted by topic and later delivered to a group of experts, called a study section. Three members of the study section score the proposal in a number of areas from a scale of 1, representing the best, to 9, representing the worst.
The scores for the proposals are then averaged to determine which proposals ranked in the bottom and top half of the applications. For the proposals in the top half, reviewers will provide a presentation and bring the proposal for review by the entire study section using the same nine-point scale.
After the scoring is complete, the proposals are ranked by score. Those with scores higher than the so-called "payline," which represents a cutoff point set based on funds available, receive funding. Currently, Carroll writes, that means tat between 10% and 15% of proposals are funded.
Problems with the current system
But according to Carroll, the current system "may be unreliable" at funding the best possible research.
In a recent study, for example, researchers examined 25 proposals that had been funded by the National Cancer Institute. Of the 25 proposals:
- 16 received funding the first time they were submitted for review, which means they could be considered "excellent;" and
- Nine received funding after they were submitted for review a second time, which means they could be considered "very good."
The researchers then created mock study sections to replicate NIH's model for reviewing grant proposals. First, they assigned a small group of participants to review specific grant applications—similar to the way NIH uses three members of a study section to review a proposal. Then, the researchers brought the small groups together in groups of eight to 10 to discuss and score the proposals for funding—again, following the model used by NIH.
According to Carroll, the researchers found the study participants reached no agreement on the quality of any of the proposals, whether measured by intraclass correlation of their scores or by a separate measure, Krippendorff's alpha.
Further, according to Carroll, the researchers "found that scores for the same application were no more similar than scores for different applications" and "there wasn't even any difference between the scores for those funded immediately and those requiring resubmission."
Carroll acknowledges that those findings should not be viewed "as a death knell for the peer review process," because the study focused only on the exceptional proposals that received funding. But it does highlight the unreliability of the current grant system—and that unreliability, in turn, may hinder the careers of promising researchers.
According to Carroll, "the average researcher with an M.D. is 45 years old (for a Ph.D. it's 42 years old) before she or he obtains" a big NIH grant, which is required to receive promotions and tenure. As such, Carroll writes, if the current grant system fails, "the repercussions will be felt for decades."
Carroll also notes that fewer studies are being funded now than in prior years, as available funding has fallen by 23% in the last 12 years after adjusting for inflation. As such, "more promising researchers are washing out than ever before."
According to Carroll, the current system also "favors low-risk research," because "if you're going to fund only a small percentage of proposals, you tend to favor the ones most likely to show positive results."
Carroll also explains that the system "favors experienced researchers" with "thicker curriculum vitae, more preliminary data and name recognition" than newer researchers. Experienced researchers also have the benefit of knowing how to navigate the grant system.
Finally, Carroll notes that the current system is not blinded, which means it can be biased against women and minorities in ways that would exclude such researchers from the funding range. He writes, "If researchers are getting into the top 10% more than others based on such factors, especially with less and less money available, many great proposals—and many great researchers—are being sidelined inappropriately."
Carroll highlights several possible ways to improve the review system, including increasing NIH funding overall.
Another approach, proposed by John Ioannidis of Stanford University, would be to fund researchers instead of projects. Meanwhile, a group of informaticists from Indiana University have recommended all scientists have an opportunity to vote on at least a share of funding.
Other proposals are "more radical," Carroll writes. For instance, Ferric Fang and Arturo Casadevall, both editors of the journal mBIO, have proposed a system involving a modified lottery, which Carroll explains would assign funding by chance after reviewers conduct their initial review to determine which proposals clear a baseline bar for quality.
According to Carroll, "such a system could reduce bias and increase diversity among researchers" (Carroll, "The Upshot," New York Times, 6/18).
July 10 webconference: The new wave of health care innovation is here
There is a new wave of innovation in health care. This webconference will identify the trends of emerging innovations and evaluate the implications of these innovations for health systems.