This guest blog post is written by Dr Alexandra Freeman, Director, Octopus Publishing Community Interest Company (CIC)
On the one hand peer review is lauded as the ‘gold standard’ mark of quality – in theory, work that has been thoroughly read and checked by a group of several independent experts in the field should be something considered to have a greater degree of trustworthiness than something simply posted online by an individual (like a blog….) But on the other hand, not only can it be glacially slow, but also very dependent on the time, attention, expertise, opinions and ethics of the reviewers selected.
It was only when I heard a variety of commentators with deep and diverse expertise critiquing potentially ‘high impact’ papers for the UK’s Science Media Centre that I realised what true peer review could be like. These commentators, with their specific, specialist knowledge, so often picked up on methodological issues, interpretation issues, statistical issues or shed light on the way that this new paper fitted in with the politics of the topic at that point in time. Only occasionally were these issues of the kind that led someone to cry ‘this should never have passed peer review’ – much more often they were critiques that just gave the necessary nuance to how much weight could be put on results, or how generalisable they might be, or what the authors were really trying to say.
It struck me that whilst journalists were (thankfully) being offered this privileged service of open and trustworthy peer review from a range of genuine experts, everyone should be able to see and benefit from these comments. Imagine if every statistical analysis was accompanied by comments from a professional statistician; if every method described had been critiqued by a methodological expert; if interpretations could be published of the same analysis from a wide variety of people with different training, backgrounds or experience. How much richer would the research record be? How much more useful than each of us only passing our own personal judgement on each article, bounded by our own inevitably narrow experience, and unshared with others?
This was all part of my thinking when coming up with Octopus, the platform that is designed to be the new, digital-first primary research record for scientific work.
In Octopus, the unit of publication is different from a traditional, narrative ‘paper’. Each part of the scientific process has its own publication type: a defined research problem, a hypothesis or theoretical rational, a protocol or method, data or results, analysis, interpretation, real world implementation… and peer review. Reviews can be linked to any of the other publication types, and because these publication types are much more focussed and specialist than a ‘paper’, it means that the reviews will specifically critique that part of the work: analyses can be critiqued by professionals specialising in analysis, methods by methodological specialists etc. Although in Octopus review is done post-publication, it will be immediately obvious to readers if a publication has been reviewed or not (since they can read any reviews linked to it) – and by whom.
Octopus will use ORCID as its log-in mechanism, and all publications (including reviews, which are treated like any other kind of publication) will be linked to the ORCID iD of their author. Within Octopus, every author will have a personal page that will list their publications, highlighting of what type they were. This will hopefully mean that good reviewing is incentivised: firstly because the writing of reviews will add to the publications on the author’s personal page, and secondly because reviews will be subject to reader ratings on pre-defined criteria, just like any other kind of publication within Octopus. These criteria allow us as a scientific community to set what ‘good’ means for each type of publication, giving a framework of what is expected of them to new authors, as well as providing a means of rewarding good reviewing and disincentivising poor work.
I know from my own experience that writing an open review – one that has your name on it as the author – is a different experience from writing directly to an editor. But that different feeling and approach is no bad thing. As researchers we have to learn humility – our work will be limited by factors outside our control as well as our own mistakes and narrowness of experience: reviews are a way of learning from the valuable experience and knowledge of others. We also have to unlearn the hierarchical bounds that we might be used to – a review is not the same as a teacher marking the homework of a schoolchild. It is the addition of richness to work through adding the thoughts of others with different experiences. Those others may be earlier in their career, or from a completely different discipline – they may have similar views, or completely opposite opinions: it all adds to the depth and diversity of the research record.
The theme of this year’s Peer Review Week is ‘identity in peer review’. In Octopus, ‘identity’ has a complex position. On the one hand we aim to discourage the use of identity as a means of judgement – removing cues that can cause bias, such as photos or first names. On the other hand we aim to enforce transparency and lack of anonymity as a means of fostering a culture of professionalism and removing cloaks behind which anyone can hide bad behaviour. What we’re really aiming for is a culture in which people have a ‘professional identity’ to which all their professional works (publications, ratings, paid and unpaid positions) are unavoidably linked – but that that identity is not so easily linked to personal characteristics that should not provide any basis for professional judgements, such as their gender or appearance. This may feel uncomfortably counter to the culture in which lines between professional and personal are increasingly blurred, but the hope is that it provides an environment where researchers can eventually feel more valued for their professional contributions, regardless of their personal characteristics or circumstances.
Moving to a system as different as Octopus will undoubtedly be a strange experience, and may take time to settle as authors get used to the standards expected of them when writing identifiable, post-publication reviews. I do, though, think that a change of culture is needed. I think we need to incentivise everyone to read and critically appraise the work of others; to think of scientific research as a worldwide, collaborative endeavour with the outcome being building knowledge, rather than a ‘heads down’ approach in which only writing our own papers is rewarded, and where journal editors and reviewers act as gatekeepers to career advancement, rather than the peers they are meant to be.
Octopus is due to launch in spring 2022 – so come this time next year, will Peer Review Week look the same? Or might we instead be celebrating ‘Write a Peer Review’ week, where everyone is encouraged to write and publish a review within Octopus: open, transparent, and useful to the community (as well as, of course, gaining credit for the review author if they do a good job).