This example below shows the dangers of cherry picked and buried data. It shows how great news and joy can be reported from rancid results, and the only protection against this is open access. When the taxpayer funds research that is not fully and transparently public, and immediately available, the people are funding PR rather than science. “Peer review” does little to stop this, little to clean up the mess after it happens, and the truth can take years to be set free.
Ten percent of teenagers taking an anti-depressant harmed themselves or attempted suicide. This was ten times the rate of the teens on the placebo. The results of this clinical trial were published in 2001, but those alarming statistics were not reported. The drug went on to be widely used. A new reanalysis of the data, reported in the BMJ, revealed the dark and hidden dangers. The company that funded the research, Glaxo Smith Kline, has already faced record fines of $4.2 billion. The Journal of the American Academy of Child and Adolescent Psychiatry won’t retract the paper.
There are many ways to hide data. In this case, the results of the trial include 80,000 records which were [...]
Was that a half-truth or a lie by omission? Trick question…
Malcolm Kendrick reports on a new study that he says should “shake the foundations of medical research” but laments that it almost certainly won’t.
In the year 2000, the US National Heart Lung, and Blood Institute (NHLBI) insisted that all researchers register their “primary aim” and then later their “primary outcome” with clinicaltrials.gov. This one small change in the way medical studies were reported transformed the “success” rates in peer reviewed papers. Before 2000, fully 57% of studies found the success they said they were testing for, but after that, their success rate fell to to a dismal 8%. When people didn’t have to declare what their aim was, they could fish through their results to find some positive, perhaps tangential association, and report that as if they had been investigating that effect all along. The negative results became invisible. If a diet, drug or treatment showed no benefit at all, or turned up bad results, nobody had to know.
The world of peer reviewed climate research: like a universe of dark matter
It’s not like climate science suffers from unpublished “negative results” — no, it’s more [...]
Joy. It’s another profoundly unscientific “consensus” study. At least one person thought that the 97% PR figure was not enough, and that magic 99.9% would sway the crowds. As if there was even one fence-sitter sitting, waiting, saying, “97% was too low…”
For the herding type of human, “consensus” is magnetically convincing. Not so for the independent minds who have seen prediction after prediction fail. If a 97% consensus on a highly complex, immature science is difficult to believe, a 99.99% one is comic. More of the same unconvincing stuff will do nothing except set off the BS meter. This new study will sway no one. The supernatural purity of it will work against “The Cause”.
A consensus is the one and only argument of the unskeptical, and they are doing it to death.
One fan, James Powell, was so enthused he spent nine months reading titles and abstracts of 24,000 papers, and found only four scientists (4!) who didn’t agree with the consensus. Some 69,402 other scientists apparently endorse “the consensus” (whatever it is) because they used the terms “climate change”, or “global warming” and they didn’t also make a clear statement that it was false, or [...]
Richard Tol has an excellent summary of the state of the 97% claim by John Cook et al, published in The Australian today.
It becomes exhausting to just list the errors.
Don’t ask how bad a paper has to be to get retracted. Ask how bad it has to be to get published.
As Tol explains, the Cook et al paper used an unrepresentative sample, can’t be replicated, and leaves out many useful papers. The study was done by biased observers who disagreed with each other a third of the time, and disagree with the authors of those papers nearly two-thirds of the time. About 75% of the papers in the study were irrelevant in the first place, with nothing to say about the subject matter. Technically, we could call them “padding”. Cook himself has admitted data quality is low. He refused to release all his data, and even threatened legal action to hide it. (The university claimed it would breach a confidentiality agreement. But in reality, there was no agreement to breach.) As it happens, the data ended up being public anyhow. Tol refers to an “alleged hacker” but, my understanding is that no hack took place, and the [...]
How to separate creative genius from creative mistakes? Not with peer-review. It is a consensus filter.
Classical peer review is a form of scientific gatekeeping (it’s good to see that term recognized in official literature). Unpaid anonymous peer review is useful at filtering out some low quality papers, it is also effective at blocking the controversial ones which later go on to be accepted elsewhere and become cited many times, the paradigm changers.
And the more controversial the topic, presumably, the worse the bias is. What chance would anyone have of getting published if, hypothetically, they found a consequential mathematical error underlying the theory of man-made global warming? Which editors would be brave enough to even send it out for review and risk being called a “denier”? Humans are gregarious social beings, and being in with the herd affects your financial rewards, as well as your social standing. Even high ranking science journal editors are afraid of being called names.
Mark Peplow discusses a new PNAS paper in Nature:
Using subsequent citations as a proxy for quality, the team found that the journals were good at weeding out dross and publishing solid research. But they failed — quite spectacularly — [...]
If you suffer from an uncontrollable urge to claim that peer review is a part of The Scientific Method (that’s you Matthew Bailes, Pro VC of Swinburne), the bad news just keeps on coming. Now, we can add the terms “Peer Review Rigging” to “Peer-review tampering”, and “Citation Rings”.
Not only do personal biases and self-serving interests mean good papers are slowed for years and rejected for inane reasons, but gibberish gets published, and in some fields most results can’t be replicated. Now we find (is anyone surprised?) that some authors are even reviewing their own work. It’s called Peer-Review-Rigging. When the editor asks for suggestions of reviewers, you provide pseudonyms and bogus emails. The editor sends the review to a gmail type address, you pick it up, and voila, you can pretend to be an independent reviewer.
One researcher, Hyung-In Moon, was doing this to review his own submissions. He was caught because he sent the reviews back in less than 24 hours. Presumably if he’d waiting a week, no one would have noticed.
Nature reports: “THE PEER-REVIEW SCAM”
Authors: Cat Ferguson, Adam Marcus and Ivan Oransky are the staff writer and two co-founders, respectively, of Retraction Watch in [...]
An excellent article in The New Yorker: Is Social Psychology Biased Against Republicans?
It’s an article about the failings of peer review and research design in psychology due to the dominance of one particular political ideology (rather than having a spread more representative of the total population). You won’t be shocked to find there is a dominance of liberal left-leaning views in the profession. The paper it discusses is by Jonathan Haidt and co-authored by our friend Jose Duarte — the psychology PhD candidate and blogger who entertainingly and comprehensively dissected Lewandowsky on his blog: Do we hate our participants?
It will be no surprise that controversial psychology papers (which disagree with the reviewer’s world view) are usually treated harshly — no matter if the data is as strong. So, thinking of another field we know, what does it mean for research design and peer review when 97% of certified climate scientists hold one world view? (They not only agree on the scientific hypothesis but on the political action as well — and they boast about that?) What chance does a “controversial” paper have? Has anyone done a study on the political diversity of official climate scientists? There are plenty [...]
Most of the results reported in peer reviewed literature in medicine are mere artefacts of poor methodology, despite being done to more exacting standards than climate studies. There are calls in the medical literature for all data to be made public and for higher P values to be required. (Yes please say skeptics everywhere). Miller and Young recommend that observational studies don’t be taken at all seriously until they are replicated at least once. That would have ruled out the original HockeyStick two times over.
Even the absolute best medical papers are wrong 20% of the time, but mere observational studies (like climate research) failed 80 – 100% of the time. These studies of papers demonstrate why anyone who waves the “Peer Review” red flag is in denial of the evidence — “Peer Review” is not part of the scientific method. It’s a form of argument from authority. A fallacy of reasoning is still a fallacy, no matter how many times it is repeated. Those who claim it is essential or rigorous are not scientists, no matter what their government-given title says.
GEN, Genetic Engineering and Biotechnology News, May 1, 2014, Point of View
Are Medical Articles True on Health, [...]
Sir Isaac Newton
Peer review by anonymous unpaid reviewers is not a part of the Scientific Method.
Once upon a time the fate of a scientific paper was dependent on an Editor whose reputation depended on making sound decisions about what to publish. Modern science shifted responsibility from a single identifiable editor to an anonymous “committee”. What could possibly go wrong?
From Zocalo Public Square
Melinda Baldwin looked at the history of peer review:
I was incredibly surprised to learn that Nature published some papers without peer review up until 1973. In fact, many of the most influential texts in the history of science were never put through the peer review process, including Isaac Newton’s 1687 Principia Mathematica, Albert Einstein’s 1905 paper on relativity, and James Watson and Francis Crick’s 1953 Nature paper on the structure of DNA.
A revolution in science happened without formal “peer review”. Who would have thought?
Crucially, journals without refereeing processes were not seen as inferior or less “scientific” than those that used referees. Few scientists thought that two anonymous readers would better judge a paper than, say, the great physicist Max Planck (who was on the editorial board of the prominent German journal [...]
At least 120 computer generated nonsense papers have been reviewed and published in publications of the Institute of Electrical and Electronic Engineers (IEEE) and Springer, as well as conference proceedings. The fakes have just been discovered by a French researcher and are being withdrawn.
Cyril Labbé found a way to spot artificially-generated science papers, and published it his website and lo, the fakes turned up en masse. In the past, pretend papers have turned up in open access journals–this time the fake papers appeared in subscription based journals. But the man who caught the fakes says he cannot be sure he’s caught them all, because he couldn’t check all the papers behind paywalls.
According to Nature:
The publishers Springer and IEEE are removing more than 120 papers from their subscription services after a French researcher discovered that the works were computer-generated nonsense.
Over the past two years, computer scientist Cyril Labbé of Joseph Fourier University in Grenoble, France, has catalogued computer-generated papers that made it into more than 30 published conference proceedings between 2008 and 2013. Sixteen appeared in publications by Springer, which is headquartered in Heidelberg, Germany, and more than 100 were published by the Institute of Electrical and [...]
19 contributors have published
2045 posts that generated