Published peer review reports have higher informative content than unpublished reports

Although publishing peer review reports increases editorial transparency, little is known about the differences in terms of information content, readability and similarity between open and unpublished peer review reports across journals. We compared 140,844 published and 117,250 unpublished peer review reports from 233 medical journals published by Elsevier and Springer Nature between 2016 and 2021 using natural language processing. Our results showed that published peer review reports were longer and had more informative content, with the greatest difference found in the number of “suggestion and solution” sentences. Published peer review reports were also more readable and more similar to each other in terms of content structure. Reports by women had higher information scores and were more readable than reports by men, while reports by reviewers from non-Western institutions had lower information scores and were less readable than reports by reviewers from Western institutions. Our results suggest that increasing the transparency of review reports could lead to more detailed reports focusing on suggestions for improving manuscripts.