The Peer Review Process - A Complete Guide

This article will look at all things peer review.

From what it is, its history, what the peer review process contains, what to think about when deciding which peer review software to select, and much more.

This guide will be your one-stop shop for all things peer review.

So let's discover all you need to know about peer reviews!

What is a peer review?

The academic peer review process is used in research and academic conferences (as well as academic journals). It is the evaluation of an abstract or paper - by qualified individuals (peers) to assess if it's worthy of publication in a journal or to be presented at an academic conference.

In layman's terms - a selection process!

During peer review, the significance, credibility and veracity of the research and its findings will be accounted for, along with its overall intentions for its readers and potential impact on the sector.

This will help those who read the entry in a journal or listen to the presentation to affirm that the information is legitimate, has been through a rigorous process, and is indeed plausible.

And that is why the peer review process is important!

The history of peer review

There have been various ways of reviewing one's work over the centuries, from gatekeepers, referees, and society secretaries to the now well-known and rounded art of peer reviewing.

The scientific societies were the forerunners in creating something along the lines of a 'peer review process" in the 18th century.

Though it was more of a "pre-publication review" of work to ensure the society publishing the work could be accredited to it as a whole and that the work was not viewed as an elaborate story but as genuine, scientific research.

This paved the way for the peer review process that encompasses a more in-depth critique and validation of work.

What is the peer review process?

Over the years, the process for peer review has been refined many times to the exact way it is performed today. So what exactly is the peer review process?

Ultimately, it's a group of peers from within your sector reviewing your work to attain its accreditation. It is used to accept work to be presented at conferences and for publication in academic journals.

Call for submission of abstracts

The first step to peer reviewing is a call for abstracts (for an academic conference) or receiving submissions from hopefuls wanting their work published in an academic journal.

This call will typically have been displayed on a website, via email and, in some instances, social media too.

The submitter will then write their abstract and submit it for peer reviewing in the hope they will be selected to speak at the conference or have their work published in the academic journal.

Reviewal of submissions

Even before the call for submissions was released, the conference or journal organisers would have already chosen who would review the submissions.

Once the abstracts have been collected, these will be allocated to reviewers. It is down to the organisers to choose how many submissions will be allocated to each reviewer.

Some organisations may have already done an "initial screening" of the submissions to ensure they have met any entry requirements set, but this isn't always the case.

Reviewers are usually provided with guidelines to follow to review submissions. This can be anything from suggesting a general appraisal approach to specific marking criteria to follow while completing the review. For example, originality or advancement within the field.

Reviewers will typically enter a grade and be asked to leave comments about the submission.

Submitters will typically have access to the submission guidelines to understand what the reviewers will be looking for and to help them write their best abstract tailored to this specific submission.

The committee's decision

Once all the submissions have been reviewed, the next stage of the process involves the committee's decision (or editor if for publication in a journal).

The reviewers' opinions of the submitted submissions will be considered when deciding whether to accept or reject the abstracts.

Typically, before a final decision is made, the submissions that have comments from the peer reviewers on how to improve their abstract (for example) will often be revised by the submitter to give them a chance to improve their work based on the changes suggested.

Once complete, the updated submission will be analysed by the reviewers again and then sent to the committee for a final decision.

Accepted abstracts will mean the submitter can attend and present their paper at the conference or see their work published in a well-respected journal.

If you manage a peer review process, we've provided you with a step-by-step guide on what you need to do later in the article.

How many peer review types are there?

There are three peer review options in which a peer review can be administered - single-blind, double-blind and open.

• Single-blind

The most popular peer review option, the single-blind peer review process, involves having the submitters' names visible to the reviewer, but the reviewers' names are hidden from the submitter.

This process allows the reviewer to have that safety net of being fully honest with their review without having to deal with any positive or negative reaction from the submitter.

The single-blind peer review process also lets the reviewer know which institution the submitter belongs to and enables them to reflect on the submitter's previous work.

However, this does leave room for bias to creep in. The reviewer can allow, whether intentionally or unintentionally, the submitter's previous work to overshadow the current submission being reviewed.

• Double-blind

The double-blind process means the submitters' names are hidden from the reviewers, and the reviewers' names are hidden from the submitters'.

The main reason to use a double-blind review is to remove bias at all levels from the review altogether, and it is for this reason that this type of peer review is becoming popular.

The flip side to this is if the reviewer has more information on the submitter, such as their previous work (as they would with a single-blind peer review), the reviewer can use this knowledge for context when reviewing the current submission.

However, as we said above, this scenario can leave room for previous work to overshadow the current submission.

But the end result of the review could mean a more well-rounded review.

• Open

The open peer review process is when the names of both the submitters and reviewers are visible to each other.

A relatively new approach to peer review, open peer reviews bring one hundred per cent clarity to both parties, which can, of course, be both good and bad.

Negatively, it can publicly show which reviews are too harsh and weaker in context. But knowing that your review will be 'public" may reinforce a tighter, more strict approach. Or it could also stop the reviewer from providing a genuine, honest review.

So, in the end, the reviews provided might not say exactly what the reviewer wanted to say.

How to write a peer review

If you've been chosen to peer review, there are a few pointers to take onboard before you dive in.

Check the review format

First, make sure you know which review format you are to follow. Depending upon the software you are using for your peer review, there will be specific ways for you to add your review.

For example, do you only need to add a grade, or do you need to add comments… specific or general comments, and are there word limits to your comments?

So ensure you are au fait with how to work the software and securely upload your review to the specifications required.

Prepare a checklist

Most of the time; your institution will provide a guidance document for you to follow when performing your peer review. Alongside this, prepare a checklist for using your own criteria. Things you can consider can include:

• Relevance of the submission

• Advancement within the field

• Originality and quality

Ensure you schedule enough time for you to complete each review fully, leaving no stone unturned. Plus, this can help provide peace of mind that you have taken the time, care and consideration required to submit a balanced, helpful and comprehensive review.

Read everything TWICE!

Having a thorough read of the abstract you're reviewing from start to finish, pausing, and making notes of your initial overall impression are the stable building blocks for writing a comprehensive review.

Analyse not only the text but also if they have included any illustrations, graphs or tables (if allowed with the submission).

• Do these flow well within the work?

• Do they align with what is being written about?

• Are they clear and relevant?

• If any statistics are included, check that these are accurate and applicable to the abstract.

Tip: It's always a good idea to type your notes on a platform such as Google Docs so you can easily access them and make any quick necessary changes.

From these pointers, you can then begin to curate your peer review.

Draft your review

With your notes, you can create your first draft of your peer review.

Start with:

• A brief, generic summary of your review

• Then go into detail with the following points:

  1. Include strengths and weaknesses

  2. Major and minor issues

These pointers will help structure your review, but they must be professional, constructive, and neutral.

When providing your feedback, write in a calm, friendly but objective tone, providing detailed comments with any criticism you provide backed up with facts and references.

Check and check again

We don't just mean grammatically check, although this is important.

No, we mean to read your review and ensure it covers all bases and provides suitable feedback clearly, professionally and concisely.

• Does your review provide good-quality feedback?

• Does it sound unbiased and professional?

• Are your comments easy for the submitter to understand, and can they quickly implement any necessary changes you have recommended?

How to manage the peer review process

The entire peer review process can seem overwhelming when you're responsible for it, but it can be a doddle with the right software.

Gone are the days of creating a spreadsheet and having your email inbox flooded with hundreds of emails relating to peer review alone.

Finding the right peer review software may take time, but it will save you a lot of time and organisation in the long run.

The great thing about peer review software is that you can control everything:

• Who reviews what submissions

• You can see which submissions have been reviewed/awaiting review

• Reviewers can easily mark the submissions and see how many reviews they have

• Send emails, whether templated or ad-hoc, to reviewers and submitters with ease

• Easily find the reviews that have passed the peer review process

So, where to start?

A step-by-step guide to managing a peer review process

1. Decide who will be a peer reviewer

Start by deciding who will be peer reviewers and let them know.

If there are any instructions they need to follow when performing their review, then make sure the reviewers are aware and have the information to hand.

2. Call for submissions

If you haven't already, you need to put out a call for submissions and start collecting them.

3. Collection of submissions

Once the submissions start to be collected, you can either start assigning them to reviewers as they are submitted, or you can wait until you have closed the call for submissions and then assign them to reviewers.

This usually depends on whether the submissions go through an initial "vet" to ensure they are correct for your conference/journal before being correctly reviewed.

4. Assign submissions to reviewers

Assign submissions to your reviewers and wait for the reviews to be conducted and scored.

Some peer review software allows reviewers to leave comments to help submitters address any areas needing to change before a final review, allowing them a better chance of success.

5. Committee members approval

Once everything has been finalised and reviewed, in most cases, the ones with the best marks are then sent to the committee members to decide who has been successful.

6. Contact submitters

When these steps are completed, you can contact the submitters, informing them if they have been unsuccessful or successful and what the next steps are.

Questions to ask when looking for peer-review software

Once a laborious admin-heavy task, the academic peer review process is now usually handled by abstract management software.

The abstract collection is the first part of the process, and whilst assessing the software on its ability to handle this process, you should give equal weight to looking at the features and tools offered throughout the reviewing process.

Data protection and privacy are a given, but when looking for software to manage your academic peer review process, we recommend you give some thought to the stages below and ask yourself and peer review software providers the following questions:

Does the reviewing form take care of your peer-reviewing requirements?

Before you decide on which academic peer-reviewing software to purchase, ensure you know exactly what information you will need from your reviewers. Aside from the usual grades and comments type of questions, do you have additional requirements you need to collect?

Take a look at the form templates, questions and setups and think of the following:

• Does the system handle single and double-blind reviews?

• What about conflicts of interest?

• How are they declared and managed?

• Do you need the reviewer to upload anything - e.g. a bio or certification?

• Can you share reviews directly with submitters, if required?

• What if you require a two-stage review process - for example, a first review suggesting amendments, followed by a second review of the amended submission?

Academic peer review software suppliers should be able to talk you through your requirements for collecting peer reviews and illustrate how their system will be a time-saving, flexible and robust solution.

Is the assigning process simple and user-friendly?

Once your abstracts are in, the next task will be to assign abstracts to reviewers. Good academic peer review software will simplify this process, saving you lots of time and stress.

A few questions to consider and ask include:

• Will you need to do many repetitive tasks - e.g., assigning the same submissions to dozens of reviewers? If so, does the academic peer review software allow for bulk assignments or ways to automate this process?

• Will you need to assign submissions based on reviewers' areas of expertise?

• Will you be assigning submissions to reviewers or the other way around?

• Does the system allow for both?

• Can you quickly reassign reviews if a conflict of interest has been declared or identified?

There are lots to consider when thinking about the assigning process, and you should spend time identifying your must-haves.

Are the reviewers going to be happy?

Reviewers have a job to do, and making it as easy as possible for them will make your life a lot easier too. When assessing academic peer review software, ask the following questions:

• What does the user interface look like? Is it simple and user-friendly?

• Can reviewers access the system and their assignments in a minimum of clicks?

• Does the interface reduce the requirement to switch screens - e.g., can the reviewing form and abstracts be viewed side-side?

• Can reviewers access summaries and reports of their reviews easily and quickly?

• Do the reviews autosave, which reduces the fear caused by potential power outages and unstable connections?

Keeping reviewers happy is crucial for a successful academic peer review process. They are more likely to complete their reviews and less likely to contact you with questions and problems.

How do I keep tabs on reviewers' progress?

Assigning reviews is one thing; keeping track of who has and still needs to complete their reviews is another task altogether.

Having all the data you need to hand is crucial to managing this process. For this area, think of the following questions when researching for your peer review software.

• Can you instantly see who has completed their reviews?

• Is there a robust and effective chase-up process built into the system?

• Can you send reminders and track whether these have been delivered and opened?

• Can reviewers amend their reviews, if necessary, subject to deadlines?

• What if you, as an admin, need to amend a review?

• Are timestamps collected in case issues arise?

Incomplete and late reviews can stall the abstract management process and sometimes jeopardise the research conference itself, so it's imperative that the academic peer review software you decide on can deliver on tracking reviewers' progress.

There are many academic peer review software providers that can deal with your abstract management and peer review process. Oxford Abstracts has spent 20 years talking to conference organisers, and we're proud to say we can say a firm 'yes' to all the above questions and a lot more.

Whether you're submitting work for peer review, being a peer reviewer yourself, or organising a peer review, we hope this article has helped demystify and provided helpful tips for you to do what is necessary within the peer review process.

And of course, finally…. Good luck!