Effectiveness of Anonymization in Double-Blind Review
By C. Le Goues, Y. Brun, S. Apel, E. Berger, S. Khurshid, Y. Smaragdakis
Communications of the ACM,
Vol. 61 No. 6, Pages 30-33
Peer review is a cornerstone of the academic publication process but can be subject to the flaws of the humans who perform it. Evidence suggests subconscious biases influence one's ability to objectively evaluate work: In a controlled experiment with two disjoint program committees, the ACM International Conference on Web Search and Data Mining (WSDM'17) found that reviewers with author information were 1.76x more likely to recommend acceptance of papers from famous authors, and 1.67x more likely to recommend acceptance of papers from top institutions.6 A study of three years of the Evolution of Languages conference (2012, 2014, and 2016) found that, when reviewers knew author identities, review scores for papers with male-first authors were 19% higher, and for papers with female-first authors 4% lower.4 In a medical discipline, U.S. reviewers were more likely to recommend acceptance of papers from U.S.-based institutions.2
These biases can affect anyone, regardless of the evaluator's race and gender.3 Luckily, double-blind review can mitigate these effects1,2,6 and reduce the perception of bias,5 making it a constructive step toward a review system that objectively evaluates papers based strictly on the quality of the work.
Very interesting! It seems then that manual reviewers are no better at guessing than using a machine learning-based approach based on the number of times authors appear in the references, see https://peerj.com/preprints/1757v1.pdf.
Interesting. In a recent double-blind review process we came across one issue: multiple entries by the same author(s), involving self-plagiarisation. The double blind process made this very hard to detect; theoretically a CRP system could detect and warn without revealing author identities, but ours at least did not do so.
The results reported here are interesting. However, since some of the figures cited in the introductory paragraph are being repeated in online discussions, it seems worth pointing out the following:
1) The 1.76x and 1.67x odds multipliers quoted from Tomkins et al. for the effects of author fame and institution reputation are from an early version of that paper. The numbers from the latest version (v6) on arXiv are smaller (1.63 and 1.58, respectively). This most recent version was probably not available when this Viewpoint was submitted or initially drafted. Version 5, posted May 2017 has factors of 1.66 and 1.61, respectively. Of course, these are still large effects.
2. The confidence intervals for these odds multipliers from Tomkins et al. are quite large: the bottom end of the range for the "author fame" correlation coefficient corresponds to an odds multiplier of 1.05. In light of this, and the changes from different version of the paper, I think one should be careful about placing too much emphasis on their exact size.
3. I think it is worth noting some negative findings from the cited papers, particularly when they contradict the other mentioned positive findings. For instance, the Viewpoint mentions that in a certain medical field, earlier work found a bias from American reviewers in favor of American authors. On the other hand, Tomkins et al. find no significant bias based on shared nationality of reviewers and authors. Additionally, Tomkins et al. do not find a statistically significant bias based on author gender (though they argue that when placed in context of a larger meta-analysis of other studies, these results may be considered significant).
Displaying all 3 comments
Log in to Read the Full Article
Purchase the Article
Create a Web Account
If you are an ACM member, Communications subscriber, Digital Library subscriber, or use your institution's subscription, please set up a web account to access premium content and site
features. If you are a SIG member or member of the general public, you may set up a web account to comment on free articles and sign up for email alerts.