In my experience, doing peer review is a rather private affair.”— Neil Stoker

In my experience, doing peer review is a rather private affair. I would get an email asking me if I’d review a particular paper, and if I agreed, a private agreement was reached between myself and the editor. I then privately adjusted my work plan, and at some point in the next week or two, I read, thought about it, wrote a report, sent it off and hopefully never heard back again, except perhaps for a polite note from the editor a few weeks later as to whether it had been accepted or not. I might completely forget it ever happened. A paper appeared some time later that looked familiar, like I remembered it from a dream…

That I’d done this, was something I would probably mention to no-one, partly because it was a confidential review.  And while I and colleagues might mention in passing that we did reviews,  why we did them, or how we wrote them, or how many we did, was something that I discussed with no-one.

Yet these days people are talking very publicly about peer review. Newspapers tell us that “peer review, it appears, is less reliable than it sounds”, and Nature currently has a collection of 22 opinion articles on various aspects, and academic papers describe its “fundamental flaws”.

Partly perhaps because of its essentially private nature, it might be unsurprising that I disagree with much that I read.  After all, I’ve developed my own belief system in the total absence of discussion.  Maybe it’s good that our ideas are aired and tested.

A jury of one’s peers

It seems to me that peer review is like the jury system, in which I see two separate aspects.  The first is a functional one: it acts as a filter, through which hopefully the guilty pass and the innocent do not. Similarly, the world of science uses peer review as a process that filters good science from bad. I like the metaphor of a filter partly because filters are never perfect.  Their aim is to separate, but I can think of no filter that is, or can be, perfect.  Diagnostic tests, for example, are given cut-offs that optimise accuracy, but false negative and positive results are unavoidable (and the cut-offs may be situation-dependent).  Peer review is no different.

Note that the roles of juries and peer review are merely to give an opinion as to whether the case – given the evidence presented – appears to be valid. Just as it’s not up to the jury system to determine if the evidence provided has been faked, so it is not up to peer review to decide if data are fraudulent.  Yet articles (such as the ones linked to above) frequently assert that increasing scientific fraud is a sign of peer review failure. Similarly, just it’s not up to the jury system to evaluate what is and is not legal, it’s not up to peer review to work out how negative clinical trial data are made public – those are wider structural issues.

Note also that people’s opinions will vary, and just as two juries might give different verdicts, so different peer reviewers might well do the same.  Certainly when I got reviews back, I could see that reviewers frequently had completely different evaluations and points of criticism.

For me, the point about this is that there is no pretence or intention that this is ‘perfect’; it’s a filter by scientists, for other scientists, who don’t for a second take publication as indication of ‘truth’. Rather it’s an affirmation that the work is competent.

A significant problem here is that other communities (politicians, the press, the public) interpret things differently – and admittedly scientists when speaking for themselves as opposed to through their publications, are often to blame.

Trust

The second aspect is an emotional one; I trust the system, in that I believe it is generally fair, and is right more often than not. Again, that does not equate with an sense of absolute truth, but with an idea of the issues having been taken seriously, through a system that strives for truth that it will never attain, and may never know whether it has succeeded or not.

This comparison between law and science  helps keep me aware that we are dealing with ‘best guesses at the moment’, rather that ‘truth’.  It also reminds me that we are dealing with a social structure that we need to critique, but alter with great care.

Recognition

So Peer Review Week, which started in 2015, seems an excellent innovation, partly because it provides a time to focus on this important topic, and partly because it allows all sides to be involved in that discussion.  Perhaps even the shy retiring reviewers themselves can be encouraged to briefly come out of the shadows.

This year, the theme is “Recognition for Review”.  The private process I described above is generally ignored by managers in performance reviews, where publications and journal impact factors completely overwhelm any other metric. Yet scientists give significant amounts of their most precious commodity – focused attention – to this activity that aids every part of the scientific process.

How might reviewers get recognition?   I worry about suggestions of financial remuneration, which can backfire when applied to social structures.  I think it’s nice to see journals formally and publicly thank us. But I’m particularly interested in the idea of linking reviews to our academic record through ORCID. One of the many exciting innovations since I stopped full-time science, I’m a big fan of ORCID, in giving scientists, junior and senior, identities that transcend where we happen to be working at the moment. This allows my line manager to see what I do in a way that has authority; it also allows me to stay aware of what I have done, and perhaps use that to judge whether I do too much or too little.  I might even prefer to do a review for a journal that actively does this linking.

I did reviews for a number of reasons.  Some were selfish: I got to be aware of what was going on in my field ahead of publication, and the experience of working within the system helped me write my own papers, and deal with responses to my own papers more effectively.  But I also felt that it was part of good citizenship, it was fair that I give up a significant amount of time and energy to the wider scientific cause.  Scientific publishing, funding and assessment are all undergoing enormous upheavals at the moment, and it is important to keep as much of the sense of shared and common purpose, and mutual respect, as is possible, and peer review is a vital part of that.

Neil Stoker (ORCID ID) spent 30 years as a full-time scientist until 2009. He is currently completing an MSc in Science Communication at Imperial College London, and has been working on a placement with FEMS.