Three ways to improve peer review

Many organizations rely on prosocial behaviours – choices that benefit others but have a personal cost – to achieve their objectives. For instance, foundations rely on charitable contributions for funding, governments partly rely on voluntary compliance for tax revenue, and employers rely on voluntary referrals for hiring. Because such prosocial behaviours have positive externalities by definition, increasing such behaviour can improve welfare. What are the most effective policies to encourage prosocial behaviour?

While there is a large body of evidence from the lab on the determinants of prosocial behaviour and altruism (e.g. Ledyard 1995, Vesterlund 2014), evidence from the field remains more limited.

In Chetty et al. (2014), we study this question by focusing on a setting familiar to academic researchers – the peer review process. Peer review is a classic example of prosocial behaviour – the personal rewards from submitting a high-quality referee report quickly are typically small, but the gains to the authors of the paper and to society from the knowledge produced are potentially large (Ellison 2002).

We evaluate the impacts of economic and social incentives on peer review using an experiment with 1,500 referees at the Journal of Public Economics. We randomly assign referees to four groups: a control group with a six-week (45-day) deadline to submit a referee report, a group with a four-week (28-day) deadline, a ‘cash incentive’ group rewarded with $100 for meeting a four-week deadline, and a ‘social incentive’ group in which referees were told that their turnaround times would be publicly posted.

Figure 1. Review times by treatment group

Notes: This figure shows the distribution of review times by treatment group during the experimental period. Each survival curve plots the percentage of reports still pending vs. the number of days elapsed since the referee received the invitation. The solid vertical lines depict the six week deadline (45 days) and the four week deadline (28 days). The dashed vertical lines depict the reminders sent one week before each deadline.

Figure 1 shows the impacts of the treatments on review times. It plots ‘survival curves’ by treatment group – that is, the fraction of reports that were submitted by the day shown on the horizontal axis. Based on the evidence in this figure, as well as other related analyses of referee performance, we obtain four sets of results.

First, shortening the deadline from six weeks to four weeks reduces median review times from 48 days to 36 days. Because missing the deadline has no direct consequence, we believe the shorter deadline acts as a ‘nudge’ (Thaler and Sunstein 2008) that changes the default date at which referees submit reports. Consistent with this interpretation, most of the increase in referees’ speed occurs immediately after they receive an email reminding them that their report is due in one week.

Second, providing a $100 cash incentive for submitting a report within four weeks reduces median review times by an additional eight days. Prior work has debated whether extrinsic incentives such as cash payments are effective in increasing prosocial behaviour because they may crowd out intrinsic motivation (Titmuss 1971). The fact that monetary incentives work implies that the positive effect of the price subsidy predicted by traditional economic models dominates any crowd-out of intrinsic motivation in the context of peer review.

We further assess whether cash incentives crowd out intrinsic motivation by testing whether referees who received cash incentives become slower than those in the four-week deadline group after the cash incentives end. We find no such evidence – again indicating that crowd-out of intrinsic motivation is not large in this context.

Third, we find that the social incentive treatment reduces median review times by approximately 2.5 days. This effect is much smaller than the impacts of the other treatments, but the degree of social pressure applied here is relatively light. The fact that even this treatment has an effect suggests that more direct forms of social pressure – such as personalized emails from editors – may have powerful impacts on referee behaviour. Moreover, social incentives complement the other interventions by influencing individuals who are less responsive to other incentives. In particular, we find that tenured professors are less sensitive to deadlines and cash incentives than untenured referees, perhaps because they are busier or wealthier. Social incentives, in contrast, have much larger effects on tenured professors, as shown in Figure 2.

Figure 2. Impacts of treatments on tenured vs. untenured referees

Notes: This figure shows the distribution of review times for tenured referees (Panel A) and untenured referees (Panel B). Each survival curve plots the percentage of reports still pending vs. the number of days elapsed since the referee received the invitation.

Finally, we evaluate whether the treatments have an impact on other outcomes besides review time. Economic models of multi-tasking (e.g. Holmstrom and Milgrom 1991) predict that referees will prioritize the incentivized task (i.e. submitting a report quickly) at the expense of other aspects of performance (e.g. the quality of reviews). We find that the shorter deadline has no effect on the quality of the reports that referees submit, as measured by whether the editor follows their recommendation, or the length of referee reports, as shown in Figure 3. The cash and social incentives induce referees to write slightly shorter referee reports, but do not affect the probability that the editor follows the referee’s advice. We also find little evidence of negative spillovers across journals – the treatments have no detectable effects on referees’ willingness to review manuscripts and review times at other Elsevier journals.

Figure 3. Impact of treatments on review quality

Notes: This figure shows the effects of the treatments on review quality, as measured by the percentage of cases in which the editor’s decision to accept or reject the manuscript matches the referee’s recommendation.

Lessons for the peer review process

Our findings offer three lessons for improving the peer review process.

1. Shorter deadlines are extremely effective in improving the speed of the review process. Moreover, shorter deadlines generate little adverse effect on referees’ agreement rates, the quality of referee reports, or performance at other journals. Indeed, based on the results of the experiment, the Journal of Public Economics now uses a four-week deadline for all referees.

2. Cash incentives can generate significant improvements in review times and also increase referees’ willingness to submit reviews. However, it is important to pair cash incentives with reminders shortly before the deadline. Some journals, such as the American Economic Review, have been offering cash incentives without providing referees reminders about the incentives. In this situation, sending reminders would improve referee performance at little additional cost.

3. Social incentives can also improve referee performance, especially among subgroups such as tenured professors who are less responsive to deadlines and cash payments. Light social incentives, such as the Journal of Financial Economics’ policy of posting referee times by referee name, have small effects on review times. Stronger forms of social pressure – such as active management by editors during the review process in the form of personalized letters and reminders – could potentially be highly effective in improving efficiency.

More generally, our results reject the view that the review process in economics is much slower than in other fields, such as the natural sciences, purely because economics papers are more complex or difficult to review. Instead, our findings show that small changes in journals’ policies can substantially improve the peer review process at little cost.

Lessons for increasing prosocial behaviour

Beyond the peer review process, our results also offer some insights into the determinants of prosocial behaviour more broadly.

1. Attention matters – reminders and deadlines have significant impacts on behaviour. Nudges that bring the behaviour of interest to the top of individuals’ minds are a low-cost way to increase prosocial behaviour, consistent with a large literature in behavioural economics (Thaler and Sunstein 2008).

2. Monetary incentives can be effective in increasing some forms of prosocial behaviour. We find no evidence that intrinsic motivation is crowded out by financial incentives in the case of peer review, mirroring the results of Lacetera et al. (2013) in the case of blood donations. While crowd-out of intrinsic motivation could be larger in other settings, these results show that one should not dismiss corrective taxes or subsidies as a policy instrument simply because the behaviour one seeks to change has an important prosocial element.

3. Finally, social incentives can be effective even when other policy instruments are ineffective. This result echoes findings in other settings – such as voting (Gerber et al. 2008), campaign contributions (Perez-Truglia and Cruces 2013), and energy conservation (Allcott 2011) – and suggests that social incentives are a useful complement to price incentives and behavioural nudges.

Published in collaboration with Vox

Authors: Raj Chetty, Professor of Economics, Harvard University, Emmanuel Saez, Professor of Economics and Director of the Center for Equitable Growth, University of California Berkeley and László Sándor, PhD Candidate in Economics, Harvard University

Image: Edwy Plenel, French journalist and Mediapart chief, looks at the Mediapart website on a computer screen in the newsroom at the offices of French investigative news website “Mediapart” in Paris. REUTERS/Charles Platiau