- JoNova - http://joannenova.com.au -

Can Peer Review be fixed?

Posted By Joanne Nova On August 13, 2010 @ 3:34 am In Global Warming | Comments Disabled

The peer review system, so important to the bolstering the voice of the climate establishment and suppressing dissent, is broken. Not that it was perfect and somehow got wrecked, but that it was never stringent or transparent in the first place. As the force of money, power, and reputations was ramped up,  it was an eminently corruptible system, and thus it has become. Seriously, what other profession would call unpublished comments by two unpaid anonymous colleagues “rigorous”?

Dear IRS officer, my tax return was audited by two accounting friends I won’t name, and they say it’s right. OK?

Nigel Calder (a former editor of New Scientist) recently discussed the merits and failings of peer review and pointed at a couple of interesting articles in The Scientist. Not surprisingly, it’s not just climate science where peer review is up-the-creek. Other branches of science are subject to the same petty personality squabbles in a system where no one really gets much benefit from doing a proper honest analysis of their competitor (or compatriots) work.

I Hate Your Paper

Source: The Scientist. The story of how some journals are trying to fix peer review.

Suggestions include ways of allowing authors to carry reviews of a submission from one journal to another, posting reviewer comments alongside the published paper, or running the traditional peer review process at the same time as a public review (bring on the blogs).

Problem #1 Reviewers are biased by personal motives:
Solution?  Publish their names and their comments.

…several journals, including Biology Direct, …have decided to eliminate anonymity from the peer review process altogether. “Under the Biology Direct model, everything is transparent, and everything is in the open,” says Eugene Koonin, one of the journal’s editors-in-chief. Authors are responsible for choosing their own reviewers from the journal’s extensive editorial board of more than 200 scientists, and must find three willing reviewers for their manuscript to be considered. This process eliminates “the potential for irresponsibility in the anonymous approach to peer review,” Koonin says, adding that upon acceptance of a paper “the reviews themselves are published alongside the paper for everyone to read.”

I was surprised to hear that BMJ (British Medical Journal) also now have a fully transparent system. ( BMJ is one of the Mr Big publications; it has an impact rating of 14.) The other journals using transparent review apparently haven’t suffered either; their impact ratings have increased.
Problem #2: Peer review is too slow, affecting public health, grants, and credit for ideas
And see my post yesterday for a case study in concerted slowness…

Problem #3: Too many papers to review
Some journals publish the prepress version, allow comments and reviewers to add their thoughts, and then publish in the journal, but others point out that a multitude of prepress versions may be ignored by most people bar the competitors who may submit unfavorable comments.

Peer review isn’t perfect— meet 5 high-impact papers that should have ended up in bigger journals.

The Scientist: Breakthroughs from the Second Tier
Nature has an impact rating of 35, but a paper describing BLAST – (bioinformatics software for searching sequences of DNA or proteins) came out in 1990 in a minor journal (J Mol Biol, rating 3.9) and went on to gain not just dozens of citations, but nearly 30,000.

The article goes on to list 5 papers that each got more than 1000 citations but didn’t make it into big name journals (many were rejected by the big name editors, showing just how often seminal papers can be missed by people who’s specific job is to spot them). The papers are mostly from molecular biology and life science, but the process relates to climate science even more so (no one is saying the debate about genetics is settled).

The Editors, what about the editors?

I don’t know that we have to enforce anonymity or total transparency of reviewers. Ultimately the responsibility for the decision ought to rest with the editors. If other reviewers make nasty baseless comments, surely a dispassionate, well informed editor would see that and get another opinion, or recognize spurious arguments and publish the paper anyway. Editors have been wimping out of their responsibilities by pretending that the decision rests with the reviewers. Convenient excuse, eh?

But the real problem might be that Editors themselves are not free of personal motives or conflicting interests. When most of your subscribers, advertisers and supporters all prefer one scientific theory I guess it’s no surprise that editors prefer to publish stories they can all cheer about. Let’s face it, unfunded skeptics can’t afford to subscribe to many professional journals, so they’re not exactly a top target market for publishers. An impartial journal serves humanity better, but “humanity” is not subscribing — government funded libraries, institutions, departments and grant-seekers are.

Meanwhile, here in the free-for-all unfunded skeptic world, a peer-review system has spontaneously emerged. When people send me contributions I send them out to one or two trusted experts (and I can always use more, so if you don’t mind reading an article very occasionally and want to drop me an email or comment mentioning your specialties, it might come in handy sometime, I do appreciate advice).

PS: I’m awol this weekend, so quiet for a couple of days. Cheers :-)

PPS: If you are in Australia and want to make a difference at the Election on Sat Aug 21, I know people like Dennis Jensen (Perth), Steve Fielding (Melbourne) and Climate Sceptics (everywhere) would love any help they can get handing out How to Vote cards. These people have really put in a lot of work, and in some cases, taken very real risks to serve us all. This election promises to be very tight. Thanks…

VN:F [1.9.22_1171]
Rating: 5.5/10 (2 votes cast)

Article printed from JoNova: http://joannenova.com.au

URL to article: http://joannenova.com.au/2010/08/can-peer-review-be-fixed/

Copyright © 2008 JoNova. All rights reserved.