Interview: Jon Tennant, author of ‘The State of the Art of Peer Review’
11-09-18
Jon Tennant is author of The State of the Art of Peer Review, published in FEMS Microbiology Letters. In this piece of research he looks into how peer review works today, what is happening with recent waves of innovation in peer review associated with the ‘open science’ movement, and sets out his thoughts on the future of peer review.
To tie into Peer Review Week 2018, we at FEMS interviewed Jon to gain further insights into this academic piece of writing, and his thoughts on this year’s theme: diversity and inclusion in peer review.
Were there any particular factors that lead you decide to write The State of the Art of Peer Review?
Well, the main one was that David Nicholas, Editor of this special volume, sent me an invitation to write a balanced review of the state of art in peer review, with early career researchers especially in mind. This was based I think largely on a recent paper that myself and more than 30 awesome people from around the world openly collaborated on to brainstorm on how new social technologies could be applied to peer review in some way.
I saw this as a way to extend this paper and push forward the ideas, both in terms of my thinking about it, but also in public discussion of these things. I’m finding peer review to a really cool beast to try and wrap my head around, so this was a great opportunity to try and open up my thinking in this space. My only condition was that it would be Open Access, which FEMS kindly obliged to for free – thanks!
What is the peer review system like in your scientific field, palaeontology? Are there any unique traits that would surprise people working in other fields?
I suspect that it is largely the same as many other fields. I’ve published a range of different types of papers, to those describing new specimens, to full on analytical and data-driven ones. Typically, the peer review style for these differs; for example, for those describing new specimens/species, it really requires having an in-depth knowledge of those specimens, or at least related ones, preferably based on first hand experience.
In some cases where the fossil groups are poorly known or researched, this can make finding referees difficult, especially ones who you aren’t working with already! For more analytical ones though, one thing I found surprising was that I don’t think any referee has ever looked at the code/data used, or tried to see if they can test or reproduce the results.
To me, this is one of the core functions of peer review, making sure that things are actually verifiable, and it seems odd to have this largely absent. Go figure.
I think peer review is becoming more difficult in our field in some respects though, as it’s becoming more and more interdisciplinary all the time.
For example, it now includes everything from high energy physics and microbiology through to biomechanics and biochemistry. This makes finding reviewers capable of peering through different lenses into research more challenging, especially when some of this research is really pushing boundaries of what we know forward.
Generally though, the pre-publication peer review process for me has always been productive, civil, and ultimately improved the research in one way or another. I think part of this is that the Palaeontology community is quite small and close, and aware that there could be repercussions for behaving in a way that isn’t proper.
However, I also know that I have been quite lucky in this respect, and know others in the field who have been reduced to tears from the review process, or left academia altogether feeling shocked and bewildered. Of course, these are the stories we often don’t hear, as the stories are only told by those who tend to persevere within the present system.
In your view, what is the biggest problem with the current state of peer review?
I have to pick one? 😉
I would have to say it’s the overall secrecy, and the problems and questions this creates.
For example, there is little way to detect or penalise abusive or bullying behaviour, little accountability for decision makers, little insight into whether it’s actually doing what we think and hope it’s doing either in theory or practice, and instead we just sort of have to trust that it is actually doing what we want. And a lot of evidence suggests that it’s not, or at least not in an optimal way.
But also it seems that many researchers seem generally apathetic towards this state, and even sometimes aggressive to those who voice concerns against the present review system. Which seems strange to me that we have sort of just abandoned a system to fate and often ceded control to commercial entities, which ultimately controls the flow of scientific knowledge and the structure of our academies, as well as researcher careers.
The (main?) reason for this is that this traditionally closed system has helped to create the present status quo in academia and scholarly communication, and any criticisms towards peer review can trigger a sense that they are undermining or challenging the very foundations of scholarship or the legitimacy of research communities.
So, for some, the secrecy creates a place of comfort that protects the established way of doing things, while at the same time making it very difficult to ask any questions about how peer review truly operates at different levels. This makes any sort of innovation or change pretty difficult.
Looking from a more positive angle, has anything been improving in recent years?
Goodness yes!
Just this week we saw the amazing collective effort from publishers, spearheaded by ASAPbio, to commit to sharing their review reports. For me this is a huge signal that the present system is being disrupted in a way that is more transparent and beneficial to the wider research community, but also in a way that has buy in from across stakeholder groups.
I think efforts like this from the publisher side of things are important. Peer review is a key part of what defines the legitimacy of their brands, and thus their overall business stability, and therefore making shifts like this is brave, but also a good indicator that system-wide shifts towards being more ‘open’ are happening. Huge kudos to Jessica Polka and the ASAPbio community for driving this.
But as well as that, we are seeing increased cross-stakeholder discussions, with more questions being asked and novel solutions and ideas popping up all the time. There are some wickedly smart and passionate in this space, and it’s great seeing what the next steps they might conjure up are.
That we now have editors, journalists, students, researchers, librarians, funders, learned societies and publishers all engaged in wider discussions about peer review and the scholarly communication system to me is quite incredible, and a great space to be in at the moment. I think that with this we are going to start seeing increased efforts decoupling peer review from traditional journal or article-based processes in the future.
This year, the focus of Peer Review Week is “Diversity & Inclusion” – does the current state of peer review have a problem with diversity and inclusion?
Ask me again after PRW has finished and we look at the people who are engaged in the discussions.
I mean, to me the answer is an almost unequivocal yes, in many respects. However, I think this is more broadly reflective of the fact that we have insurmountable diversity and inclusion issues that perpetuate academia and impact virtually every part of research from the granting process all the way to research evaluation. Hopefully those participating in PRW will be mindful of this, and really help to ask and answer some difficult questions of ourselves and the system we operate within.
What is the most important reason for increasing diversity & inclusion in peer review?
When we embrace different viewpoints from across the spectrum, including those that are markedly different from our own, that is when beautiful things in research happen. Really the same is true of virtually any aspect of our lives, in that if we only hear the same things from the same people over and over again, we get a ‘broken record’ effect and basically stagnate.
So for me, making sure that we are including diverse voices in an inclusive space is firstly the right thing to do, but also creates a better process for us all (as the opposite is stagnation and exclusion).
What one thing would most improve diversity and inclusion in peer review?
Not ask the middle-class hetero European white dude how to answer that question 😉
Seriously though, it is people like me who should be shutting up and listening to those voices who are systematically marginalised by the present way of doing things. If we want to promote inclusivity and diversity, we first have to recognise the symptoms that promote exclusivity and homogeneity.
So, my usual recommended solution is not to have too much of a public view on these things, and to actively invite those who we wish to ‘include’ to give their views instead, while also creating an environment that helps to empower those individuals to do so.
That being said, I do believe that there is so much more, and some specific things that we, as I guess someone who represents the majority demographic in this space, can do too. For example, being mindful and aware that systemic biases exist, and doing what we can each do to improve them.
This means not inviting the same speakers to our PRW panels/events that we always do, making conscious efforts to increase participation from voices who are usually marginalised (including in agenda setting), declining invitations and recommending other voices instead, and simply shutting up some times or saying no. If you are in control of a microphone, pay attention to who you are giving it to.
As an extension to this, we need to make sure that we aren’t just paying ‘inclusivity’ lip service, and only applying it in a superficial manner.
For example, simply getting a diverse group of people to participate in something like peer review for the sake of a box-ticking exercise. If we want to truly promote inclusivity, we need to recognise that often the rules are still made by the same people, but others don’t know how to play the game.
We want to create an environment were diverse voices are actually helping to set the rules of engagement, not just being invite to conform to them. We should make sure that we aren’t just inviting people for the sake of representation, but with a full awareness of what that representation brings, and how to foster the environment that promotes and sustains this.
An Indonesian colleague here told me recently “I am not just an object to be included.”, when we were talking about these things, and that resonated with me quite hard.
One thing I suggest in the paper, on a more practical level, that could improve this state is to provide sufficient training and support, particularly for more inexperienced or at-risk reviewers, as well as risk-mitigation strategies, that would enable researchers to be comfortable experimenting with peer review, including innovative forms of it.
But really, if we want to address these things in peer review, we have to recognise that there are wider systemic issues that we need to fix first. And to help fix these things, we have to embed inclusivity in our principles right from the start, and make sure that the voices we need to change these things are being heard all the way to the top.
Not blockchain.
Don’t forget, you can read his published article The State of the Art of Peer Review here.