Investigating the effect of publication text similarity between reviewers and authors on the rigor of peer review: An intellectual proximity perspective

Abstract

The involvement of experienced peers as reviewers plays a crucial role in manuscript evaluation during the peer review process. Nonetheless, concerns have arisen regarding potential cognitive bias when reviewers assess research that is outside their areas of expertise. Despite these concerns, quantitative analysis of this issue remains limited. This study aims to empirically investigate whether submissions reviewed by peers with academic backgrounds similar to the authors' research areas correlate with more rigorous comments during the peer review process. Utilizing a dataset of 2,147 papers published in the journal eLife, along with their publicly available peer review reports and reviewers' publication records, we employed natural language processing techniques to measure the publication text similarity of reviewers to that of the manuscript's authors, representing a minuscule part of intellectual proximity. We then used a linear regression model to examine whether such similarity was associated with review rigor, quantified by the frequency of statistical terms from two well-known glossaries. We observed no statistically significant differences in the rigor of comments made by peers with varying levels of publication text similarity in the constructed dataset and setting. The findings remained consistent across several robustness checks and alternative specifications. This suggests that no discernible cognitive bias is introduced by the reviewers' academic background during the peer review process, enriching the extant literature and offering important insights into understanding the role of reviewers in maintaining fairness.