AI | Can AI Assist in Peer Review?

The peer review process plays a crucial role in maintaining rigor and quality standards in academic research. However, the constant growth in research output has placed tremendous strain on this system as the number of papers requiring expert evaluation increases each year. Reviewers find themselves spending more and more hours combing through submissions, and submission backlogs continue lengthening. With research productivity suffering as a result, scientists have been searching for ways to streamline peer review without compromising its usefulness. Could artificial intelligence (AI) offer a solution by assisting reviewers or automating certain tasks?  

A new study from researchers in Italy and the UK explores this question by developing an AI system to analyze past peer review decisions and predict outcomes for new papers based on superficial characteristics like word choice, readability, and formatting. Their findings suggest AI may indeed play a supportive role, potentially reducing reviewers’ workload without replacing human judgment. However, the researchers also raise important considerations around biases the approach could inadvertently introduce if not properly overseen.

The team collected over 3300 conference papers along with reviewer scores and acceptance decisions from previous years. They focused on two major AI conferences through the open review platform OpenReview.net as well as a wireless communications conference. For each paper, they extracted shallow features like word frequencies, readability metrics, and formatting details rather than delving into content. They trained a neural network on this data to infer rules connecting features to review outcomes.

To test the model, they examined how well it could predict Human reviewers’ recommendations for unseen papers based solely on surface attributes. Surprisingly, accuracy was often high, suggesting a strong correlation between superficial qualities and overall evaluation results. Papers with clearer writing and proper formatting, for instance, tended to receive more positive reviews on the whole.

This correlation indicates such superficial cues may serve as credible proxies for general quality according to the researchers. Well-presented papers are less likely to contain deeper flaws, so assessing submissions partly through readability and appearance could help streamline screening. AI could flag issues for early reworks without full reviews, potentially avoiding biases superficial problems introduce. It may also advise inexperienced reviewers by signaling expected quality levels.

Of course, the study has limitations. It focused narrowly and did not attempt full replication of peer review’s complex, expert judgment. Further, the approach risks propagating existing biases if unjust oversight allows, as models inherit tendencies from their training data.

The researchers suggest several applications to reduce such risks and benefit the process overall. AI explanations could uncover implicit motives behind decisions, helping address biases. Authors may gain insight on issues unconsciously swaying reviewers to improve future chances of success. Editors may extract general rules for better management too.

Controlled experiments integrating AI recommendations into live review are needed to understand interaction effects fully. The models should also analyze full review texts and feedback over keyword frequencies alone for a richer perspective. Accounting for disciplinary norms remains important too, as norms differ across fields in research presentation.

If built and applied carefully with oversight, the researchers argue AI could directly aid editors in screening submissions more rapidly. Beyond catching formatting and stylistic glitches flagged for remediation without further assessment, AI may also better match papers and reviewers by expertise. Reducing redundant reviews through more informed desk rejects could likewise save untold reviewer hours each year.

At the same time, transparency must be prioritized so decision rationales are clearly explained and verifiably impartial. Opaque “black box” AI risks losing user trust and propagating unintended bias. Vigilant testing is also needed to prevent emergent discrimination against historically marginalized researcher groups over time as systems are dynamically refined. Overall project design and review practices informed by fairness principles can help maximize AI’s benefits and avoid harms.

While far from replacing experts, AI shows promise supporting peer review through explainable recommendations on common yet time-consuming tasks. If managed conscientiously with oversight, it may alleviate growing strains on the system without compromising core quality assurance functions. Careful studies integrating AI assistance into real review workflows going forward should reveal how close to realization such an approach may be for easing peer review’s important yet increasingly taxing responsibilities.

Reference(s)

1. Checco, A., Bracciale, L., Loreti, P. et al. AI-assisted peer review. Humanit Soc Sci Commun 8, 25 (2021). https://doi.org/10.1057/s41599-020-00703-8

Click TAGS to see related articles :

AI | PEER REVIEW | RESEARCH | SOCIETY