Jeremy Berg–director of the National Institute of General Medical Sciences–has recently been posting some fascinating analyses of the recently modified NIH peer review process at his blog, including the relationship among overall impact scores, percentiles, and funding decision, as well as the correlations between the various criterion scores–significance, innovation, investigator, environment, approach–and overall impact scores. This is all fascinating stuff, and Jeremy and NIGMS are to be lauded for their openness with these data. Also, I am a big fan of the new application format, scoring system, and critique format.
However, my cynical theory is that the purpose of Enhancing Peer Review was not to make peer review “better”, because it already did its job perfectly fine: indentifying roughly the top quartile of applications in any given round of review. Rather one purpose of the Enhancing Peer Review effort was to placate the extramural community who was up in arms at the peer review outcomes that were being used to perform an intrinsically impossible task: identifying the top decile of applications. (This task is impossible, because there simply are no objective differences in “quality” within the top quartile that anyone can agree on.)
Since every investigator whose grant is judged in the top quartile is outraged at the indignity of not being judged in the top decile, something needed to be done to make these investigators feel that their concerns were valued and that the system would be made more “fair”: i.e., would judge all their grants in the top decile. Of course, this is mathematically impossible. The most interesting, but impossible, analysis of the new reviewing system is not post-hoc analyses of what reviewers are doing, but rather a direct comparison of assigned grant percentiles between the old and new systems. My guess is that the old system and new system would identify the same grants as in the top quartile, and almost all the same grants in the top decile (perhaps with some small differences: there may be some investigators whose grantsmanship styles are better suited to the old or new systems).
The other purpose of Enhancing Peer Review was to dramatically streamline the system, to make the peer review process faster and easier and more efficient on a per-grant basis. I think it actually did this job quite well, and I enjoy writing grants and reviewing them in the new system more than in the old. But the only metric that can tell us if peer review was “enhanced” in the sense of “improving outcomes” is whether there would be differences in the percentiling of particular grants in the old versus the new system.