This article deals with Merton's norm of Organised Skepticism according to which all scientific claims must be exposed to critical scrutiny before being accepted. This happens through the peer review process but as the paper suggests, there are inherent problems with the very process.
Here are some of the important arguments/questions in the paper:
Peer Review
- How do you define 'peer?'
- Anonymity in the review process assumes maximal effort and fair judgement. At the same time, it protects reviewers from retribution. But does it lead to laziness?
- Use of vague reasons such as 'my gut feeling is this will not work'
- Why not publish the reviews then?
- "In what direction and should the peer review process actually police scientific fraud and should the peer review itself be subjected to review and potential legal action if scientific fraud by the reviewer is suspected?"
- Tendency to accept only positive data
- Bias - free passage for well-known laboratories, country of origin also influences acceptance
- "If the peer review process is unfair, if the rights of the individual under review are not protected, if the “facts” presented during the review are inaccurate, and the result is a damaged reputation and loss of income then why shouldn’t you sue?"
- Conflict of interest can arise from differences between author, journals and funding sources, vested interests, ideologies, religions etc.
- Means of defining the impact of a scientist's research
- The number of citations for an article in a given year divided by the number of articles and reviews published in the same journal during the past two years
- “80:20” phenomenon - 20% of publications account for 80% of the citations
- But it is essential to evaluate the impact of the individual paper and take into account not only where it was published but, in particular, also how well it has been cited and by whom —a paper in a high impact journal does not necessarily equate with a high impact paper, it is the citation frequency that is more important
- Alternative method - Index of individual productivity, “h”, has been proposed by Hirsch (2005) where h is defined as the number of papers with citation number >h
- Difficulty - to determine with complete certainty that malicious intent and not interpretation error, or simply bad laboratory practice, was the cause.
- "Fraud in science, whether initially intended as hoaxes or planned with career and profit-making intentions, not only ruins the careers of the perpetrator, but also, potentially, their innocent colleagues, as well as tarnishing the reputation of the institution where the work was performed and reducing the confidence of the public in the value of scientific research"
- Self-plagiarism
- Cryptomnesia - unconscious plagiarism?
- Is reuse of descriptions of experimental methods (like A + B = C) plagiarism?
- "Scientists are no different from any other groups in society and, like many other analogous comparisons, a few rotten apples will always be found."
Recommendations
- The establishment of the equivalent of the Office of Research Integrity (ORI) in other countries
- Appropriate safeguards designed to protect both the whistleblower and the accused
- Processes whereby the apparent bias in peer review can be reduced are urgently required and should be evaluated
- Heightened awareness and education at all levels concerning the seriousness of scientific fraud in all of its manifestations
Feel free to comment.
- Dipali.
I guess the Sokal Affair was quite instrumental in introducing 'peer review' of journals.
ReplyDeleteAnother thing that would be interesting to review would be the objectivity of corporations' sponsorship of academic study. Eg. studies on Smoking in the early part of the 20th Century.
And here's an interesting one:
ReplyDeleteDouble-blind review favours increased representation of female authors
Abstract
Double-blind peer review, in which neither author nor reviewer identity are revealed, is rarely practised in ecology or evolution journals. However, in 2001, double-blind review was introduced by the journal Behavioral Ecology. Following this policy change, there was a significant increase in female first-authored papers, a pattern not observed in a very similar journal that provides reviewers with author information. No negative effects could be identified, suggesting that double-blind review should be considered by other journals.
Full article here: http://www.sciencedirect.com/science/article/pii/S0169534707002704
Uttara - The authors say that in the last 20 years, there's been an increase in interest in academic work by industry due to vested interests and intellectual property.
ReplyDeleteHere's an interesting example: The Journal of Occupational and Environmental Medicine had to retract a paper that it published in 1997 that claimed no association between cancer and hexavalent chromium. This conclusion contradicted a previous paper by the same authors. Apparently 1997 paper was “conceived, drafted and edited” by consultants for the PG & E Corporation (Pacific Gas and Electric Corporation) that was at the time involved in litigation over the very same issue. (This litigation is the basis of the film, 'Erin Brockovich') Interestingly though, the retraction was based solely on "violations of editorial policy," and said there was "no evidence that suggests scientific fraud."
Here's a link to an article on the issue: http://classic.the-scientist.com/news/display/23590/