Throughout my career as research scientist I read hundreds of scientific journal articles. They all had been through the peer-review process and all had been deemed suitable for publication in recognized world-class journals. The quality of those articles, however, was not always as high as one might expect.
Paraphrasing a quote probably inaccurately attributed to Winston Churchill: “Democracy is the worst form of government, except for all the others.” I think the same can be said of the peer review process. In my experience it is somewhat of a lucky draw regarding the quality of the review, which could significantly affect the quality of the article that is published. I will be spending a few blogs on the Ides of Science by assessing some of the common mistakes that I have found scientists in my old field making. And, since research articles are the most common means by which scientists communicate their work to others, I thought that it would be useful to introduce this by discussing the means by which journal articles are published and the potential problems that the process presents.
Firstly, when we say “peer review” we are talking about the process where a scientific article (we usually call it a “paper”) is scrutinized by others in the same field as the author(s) who are knowledgeable in the area addressed by the paper. Those peer reviewers (we usually call them “referees”) assess the quality of the paper, point out errors or inconsistencies, and issue a recommendation to the editor of the journal as to whether the paper is suitable for publication. The process by which a paper is published in peer-reviewed journals is as follows:
1. An author or group of authors submit the paper that they wish to publish in the journal.
2. The editor of the journal sends that paper out to other scientists (referees) in the same field that the paper addresses. So, for example, if I wanted to submit a paper on coronal mass ejections (CMEs) the editor would likely send the paper to other experts on CMEs. Some journals (like the Journal of Geophysical Research) submit it to two referees while others (like the Astrophysical Journal) send it to just one. Additional referees can be sought by the editor in cases where there is a major disagreement somewhere or for some other reason.
3. The referee(s) assess the paper for validity of results, clarity of the narrative, applicability of the methodology applied, whether the results represent a suitable advancement of the field, and whether the paper is suitable for publication in the journal to which it was submitted. The referee(s) then issue(s) a recommendation to the editor broadly under the following three categories:
(A) Recommended for publication with no changes (this is rare);
(B) Recommended for publication after some changes have been made (these can be major or minor, many or few);
(C) Not recommended for publication (or more precisely but rarely actually stated: recommended not for publication).
4. The editor decides whether to accept the paper, send it back to the author(s) for revision, or to reject it. They are not obliged to accept all (or even any) of the referees’ suggestions, but they usually do.
5. Normally, after a resubmission or two the paper will either be suitable to the referee(s) or not and the editor will choose to publish the paper or reject it.
I have personally peer-reviewed dozens of papers and have provided all of these recommendations to the editor at one time or another. I have refereed papers that I thought were practically flawless and immediately recommended them for publication, and I have also refereed ones that I thought were terrible and recommended that they not be published. Most commonly I would recommend that revisions be made; some major, some minor. And, on rare occasion, the editor has overruled my recommendation.
You would think that after such serious scrutiny the quality of papers that actually make it to publication is very high. While it is true that the overall quality of papers is improved as it passes through the peer-review process, the end product sometimes still leaves a lot to be desired. When I worked at the Southwest Research Institute, we had a weekly meeting called “Solar Journal Club”. In this, a member of our group would choose a published paper and we would closely scrutinize it (as a side note I recommend a practice like this to any research group that consists of students or early-career researchers). We studied well over a hundred papers and I could count on one hand the number of papers that I would have regarded as excellent.
Why doesn’t Peer Review Produce Higher Quality Papers?
Keep in mind that scientists are people and like all people they bring their own prejudices and preferences to the referee table. They are supposed to be trained to overlook this kind of thing, but it is next to impossible to approach a paper without some kind of bias. Here are some factors that I have witnessed that likely influenced the quality of the peer-review process.
1. Quality of the Referee.
Peer-review is only as good as the peers themselves. Sometimes a referee simply does not have the level of knowledge required to perform a review with the needed level of detail. This is quite common in cases where a paper addresses more than one topic; often a referee would be well-versed in one of the topics that is being discussed but not the others, and would be more likely to overlook important details in the parts of paper that are outside their area of expertise.
Sometimes a referee takes objection to a paper for political reasons. Maybe they are an advocate of a model or idea that is being challenged in the paper, or maybe they are an author of an earlier study that is being discredited with the submitted paper. I have even seen a case where I suspect a paper was deliberately held up by the referee because the referee had already submitted a paper on the same subject and wanted theirs to be published first! Many scientists are one-trick ponies and if someone performs a study that discredits their one trick they are going to take it personally. This is especially the case when it may adversely affect their prospects of winning research grants in the future.
2a. The “You Didn’t Talk About -My- Work Enough” Tactic.
This is a neat little trick, where the referee (who has likely published papers in the same area as the submitted paper) will require the authors to discuss more of the referee’s work before they recommend it for publication. Since peer-review is an anonymous process, referees are sometimes clever, and recommend papers that are not always authored by them directly, but they always tie back to work that the referee has done (and some referees are much more shameless). This is designed to expose the referee’s work to more readers and to legitimize that work because “hey look it’s being cited in all these papers!” Keep in mind that the quality of a paper is measured by the number of papers that cite it that are not connected with its authors. Papers that are published by authors who are closely associated with the paper’s authors are still counted as self-citations. In other words, the quality of your paper increases if it is cited in papers that are published by authors not directly connected to you. The “you didn’t talk about -my- work enough” tactic, then, is a cynical attempt to artificially increase the quality of a paper and is, of course, an abuse of power by the referee, as they essentially hold the author’s paper to ransom until they address the referee’s work to a large enough extent.
3. Personal Grievances.
Sometimes the referee simply does not like one of the authors. They may have formed a personal opinion about them in prior meetings or from some other personal interactions, or feel threatened by the author because they represent a danger to the referee’s livelihood. While it’s sad to say, and although in my experience it is very rare, sometimes a paper is recommended for rejection by a referee outright regardless of its content.
4. Poor Training Overall.
This probably ties in with point 1 above, but I have noticed that many (probably most) scientists are simply not well trained in all aspects of the scientific method. From recognizing logical fallacies to the appropriate application of statistics, the scientific method seems to be a topic that skips the notice of many scientists as they climb the greasy pole. I will be spending the next few blogs discussing failings in the scientific method, particularly as it applies to professional scientists.
I guess what I am saying is that peer review is no guarantee of quality when it comes to scientific publications. While it is unquestionable that the peer review process has improved the quality of papers from when they were originally submitted, and that it is a good process to assess the work that is being submitted for publication, they are still only as good as the peers themselves. And those peers are biased, often not fully trained, and may have a political agenda.
Take those peer-reviewed papers with a grain of salt. And always perform your own assessment before assigning any value to a publication. Even if it is a peer-reviewed one.