Metrics and peer review agreement at the institutional level

Abstract In recent decades, many countries have started funding academic institutions based on the evaluation of their scientific performance. Postpublication peer review is often used in this context. Bibliometric indicators have been suggested as an alternative to peer review. A recurrent question is whether peer review and metrics yield similar outcomes. In this paper, we study this question based on a sample of publications submitted to the national Italian research assessment exercise (2011–2014). In particular, we study the agreement between peer review and metrics at the institutional level, and compare this to the internal agreement of peer review. We base our analysis on a hierarchical Bayesian model using cross-validation. We find that the level of agreement is higher at the institutional level than at the publication level. Overall, the agreement between metrics and peer review is low, but on par with the similarly low internal agreement among two reviewers for certain fields of science. The low agreement between metrics and peer review is no reason for rejecting the use of metrics for some fields in the Italian national research assessment exercise. Our results provide input to the broader discussion of research evaluation, in which other factors also play an important role.