Could your next peer review be done by ... a robot?

Peer review is an integral part of scientific publishing.

It is a process whereby experts from a particular field evaluate the work done by a researcher in the same discipline.

Peer review was established to ensure scientific accuracy, credibility of the claims and research methods, to maintain standards, and enhance the quality of manuscripts.

Another important role of peer review is to ensure the manuscript is suitable for the intended journal and identify new pathways for future research.

In its early days, it took the form of a friendly scientific face-to-face or mail exchange. In time, it evolved into an official and rigorous process that evaluates scientific knowledge.

Unfortunately, peer review also became a nightmare for many scientists.

While the role of peer review is undoubtedly important for governing scientific research, the process itself has become a point of contention.

The reviewer is expected to examine the study and give unbiased, high-quality feedback to the editor, indicating all the strong points and weaknesses, and to make suggestions for improvements.

However, today’s process has to a large extent become influenced by politics.

Reviewers are selected by editors and remain known only to them; while, in most cases, authors are known to the reviewers.

Imagine what could happen if your professional rival or competitor were a reviewer.

Since reviewers are not disclosed, there is a tendency for them to discredit research work without any consequences.

Envy is an important factor that is killing the process. Many reviewers are envious of colleagues who proffer novel and potentially impactful ideas in their field of expertise, and often find arguments to reject such manuscripts.

There have been established cases in the past in which reviewers have rejected a manuscript only to pursue the idea and put it forward themselves.

Unfortunately, editors tend to favour reviewers rather than authors.

Scientific fields have become so broad that an expert in one field may not have sufficient knowledge or expertise to effectively evaluate a research paper.

Scientists tend to be busy, and the peer review process can be demanding, both in terms of time and knowledge; this makes the quality of some peer review questionable.

There have been different attempts to solve issues related to peer review. These include the double-blind peer-review process, where both reviewers and authors are known only to editors; or the introduction of at least two to three independent reviewers in order to eliminate bias. These have improved the process, but not eliminated all the challenges associated with scientific review.

A recent study by the prestigious scientific journal Nature shows a serious crisis in scientific reproducibility that can be linked to poor peer-reviewing.

According to Nature’s survey, over 70% of respondents claim they have not been able to reproduce scientific data produced by their colleagues, and more than half have failed to reproduce their own experiments.

A few months ago, I encountered the power and capabilities of IBM Watson, an artificial intelligence (AI) developed by IBM.

At a Policy Hackathon event organized by Global Shapers in London, a team demonstrated an IBM Watson-based system capable of watching YouTube videos of politicians giving talks.

The system analyzed their speech and body language, then searched the internet for additional information related to the politician’s claims and the context of discussion in order to ascertain whether the speaker is telling the truth or not.

The prototype was developed over a period of only two full days!

So can a similar system bring more objectivity and better governance to scientific knowledge by introducing artificial peer reviewers?

Yes, I believe it can.

Artificial peer-review process can eliminate all the issues plaguing the present peer-review process by introducing more objectivity, broader expertise (by referring and comparison to previous studies), and shortening the overall time of peer review (which, in some cases, can take up to two months, or even longer) to become almost instantaneous.

All required data could be derived from a data-mining process from previously published studies.

I know what you’re thinking: what if the previous data is wrong?

Well, AI peer reviewers could solve this issue too, by performing “inverse peer review" to evaluate the correctness and legitimacy of previously published studies.

But the reliability of existing data isn’t the only issue AI peer review would raise.

There’s an accountability issue, too. Who takes responsibility for incorrect outcomes?

Or, what if the AI was sabotaged? We have seen what the public has done to Tay, Microsoft’s AI chatbot.

Is scientific knowledge accurate enough to provide legit learning datasets for artificial peer reviewers? If you believe Nature’s study, it is not.

Should we entirely eliminate the human factor, or should we treat AI peer review as a supporting or preliminary source in the decision-making process?

These and more questions require in-depth thought.

Moreover, before we can see AI peer review in action, many legal issues have to be sorted.

While AI can solve all current issues in the peer review system, it may also bring new challenges to the process.

Another question that readily comes to mind is this: if AI can review papers, when will it start writing them?

Provide enough answers to those issues, and the next generation of reviewers just might be made of silicon.