Government Social Research
1/2/06: Is peer review broken? Article by Alison McCook in The Scientist, (20)(2),26 says that submissions are up, reviewers are overtaxed, and authors are lodging complaint after complaint about the process at top-tier journals. Cites Peter Lawrence, who has been publishing papers in academic journals for 40 years. His first 70 or so papers were "never rejected," he says, but that's all changed. Now, he has significantly more trouble getting articles into the first journal he submits them to.
Lawrence, based at the MRC Laboratory of Molecular Biology at Cambridge, UK, says his earlier papers were always published because he and his colleagues first submitted them to the journals they believed were most appropriate for the work. Now, because of the intense pressure to get into a handful of top journals, instead of sending less-than-ground-breaking work to second- or third-tier journals, more scientists are first sending their work to elite publications, where they often clearly don't belong. Lately, academia seems to place a higher value on the quality of the journals that accept researchers' data, rather than the quality of the data itself. In many countries, scientists are judged by how many papers they have published in top-tier journals; the more publications they rack up, the more funding they receive.
The recent discrediting of stem cell work by Woo-Suk Hwang at Seoul National University sparked media debates about the system's failure to detect fraud. Authors, meanwhile, are lodging a range of complaints: Reviewers sabotage papers that compete with their own, strong papers are sent to sister journals to boost their profiles, and editors at commercial journals are too young and invariably make mistakes about which papers to reject or accept.
The article says that, despite a lack of evidence that peer review works, most scientists appear to believe in peer review. Indeed, an abundance of data from a range of journals suggests peer review does little to improve papers. In one 1998 experiment designed to test what peer review uncovers, researchers intentionally introduced eight errors into a research paper. More than 200 reviewers identified an average of only two errors. That same year, a paper in the Annals of Emergency Medicine showed that reviewers couldn't spot two-thirds of the major errors in a fake manuscript. In July 2005, an article in JAMA showed that among recent clinical research articles published in major journals, 16% of the reports showing an intervention was effective were contradicted by later findings, suggesting reviewers may have missed major flaws.
Some critics argue that peer review is inherently biased, because reviewers favour studies with statistically significant results. Research also suggests that statistical results published in many top journals aren't even correct, again highlighting what reviewers often miss. Moreover, peer review can also err in the other direction, passing on promising work: Some of the most highly cited papers were rejected by the first journals to see them. There is also a suggestion that reviewers were less likely to reject a paper if it cited their work, although the trend was not statistically significant.
The article, with associated articles, can be viewed in full on The Scientist website.