How machine learning can be effectively used in policy targeting

The impact of a public policy partly depends on how effective it is in selecting its targets. Machine learning can help by exploiting increasingly available amounts of information. Using data from Italy, this column presents two examples of how to employ machine learning to target those groups that could plausibly gain more from the policy. It illustrates the benefits of machine-learning targeting when compared to the standard practice of employing coarse policy assignment rules based on a few arbitrarily chosen characteristics.

In order to inform policymaking, we need answers to both causal and predictive questions. Kleinberg et al. (2015) provide an example that helps us draw a distinction between the two. A policymaker taking action against a drought might need to know whether a rain dance will produce the intended effects. This is a purely causal question. But other rain-related decisions, such as whether to distribute umbrellas, only require a good prediction for the likelihood of bad weather, because the gain of holding an umbrella is known, at least with some degree of approximation. This is what Kleinberg et al. (2015) call a ‘prediction policy problem’.

Most of the policy evaluation literature has focused on causal questions. The ‘credibility revolution’ in econometrics (Angrist and Pischke 2010) has provided us with tools for the ex post estimation of the causal effects of public programmes. Nevertheless, the impact of a policy also depends on how effective it is in selecting its targets. Predicting which target may garner a larger return can be crucial when resources are scarce. This is where machine learning comes in handy. Machine learning algorithms developed in statistics and computer science (Hastie et al. 2009) have proven particularly powerful for predictive tasks (Mullainathan and Spiess 2017).1

We present two examples of how to employ machine learning to target those groups that could plausibly gain more from the policy.2 The examples illustrate the benefits of machine-learning targeting when compared to the standard practice of implementing coarse assignment rules based on arbitrarily chosen observable characteristics. Only ex post evaluation methods, however, will eventually tell us whether targeting-on-prediction delivers the intended gains. The column also discusses some of the issues — transparency, manipulation, etc. — that arise when machine learning algorithms are used for public policy.

Examples of machine learning targeting

In Andini et al. (2018a), we consider a tax rebate scheme introduced in Italy in 2014 with the purpose of boosting household consumption. The Italian government opted for a coarse targeting rule and provided the rebate only to employees with annual income between €8,145 and €26,000. Given the policy objective, an alternative could have been to target consumption-constrained households, who are supposed to consume more out of the bonus.

Surveys often contain self-reported measures that can be used as proxies of consumption constraints. We use the 2010 and 2012 waves of the Bank of Italy’s Survey on Household Income and Wealth (SHIW), which asked households about their ability to make ends meet. We employ off-the-shelf machine learning methods to predict this condition based on a large set of variables that could be potentially used for targeting. Our preferred algorithm is the decision tree, which leads to the assignment mechanism shown in Figure 1. It selects only a few variables, essentially referring to household income and wealth. For instance, those households that have financial assets below €13,255 and household disposable income lower than €36,040 are predicted to be consumption-constrained. The other groups that are predicted to be so can be easily assessed by looking at the figure.

Figure 1. Decision-tree prediction model for consumption-constrained status

Even if we have reasons to believe that households predicted to be consumption-constrained are more likely to consume out of the rebate, nothing ensures that this is the case (Athey 2017). To lessen these concerns, we separately estimate, on 2014 SHIW data, the effect of the bonus for households predicted to be consumption-constrained and for those that are not. To do this, we go back to standard econometrics, assuming selection on observables and running regressions.

The estimated effect is positive and significant only for the predicted consumption-constrained households. The distribution of tax-rebate recipients suggests that great gains could be achieved by using machine learning for targeting (Figure 2, bordered in red): around one-third of the actual beneficiaries are predicted to be not consumption-constrained, while about two-thirds of non-recipients are predicted to be constrained.

Figure 2. Fraction of predicted consumption-constrained by tax-rebate recipient status

In the second application (Andini et al. 2018b), we focus on the ‘prediction policy problem’ of assigning public credit guarantees to firms. These schemes aim at supporting firms’ access to bank credit by providing publicly funded collateral. In principle, guarantee schemes should target firms that are both creditworthy — to ensure financialsustainability of the scheme — and rationed — because other firms would likely receive credit anyway (World Bank 2015). In practice, existing guarantee schemes are usually based on naïve rules that exclude borrowers with low creditworthiness. We propose an alternative assignment mechanism based on machine-learning predictions, in which both creditworthiness and credit rationing are explicitly addressed.

The investigation refers to Italy, where a public Guarantee Fund has been in operation since 2000. Using micro-level data from the credit register, which includes measures for both credit rationing3 and the insurgence of bad loans, we identify the hypothetical target firms and estimate a machine-learning model to predict this status. In this case, our favouritemachine-learning algorithm is the random forest, which is essentially the average of many decision trees.

To assess whether targeting these firms will improve the scheme’s impact on credit availability, we first consider the set of firms that received the guarantee and split them intotwo groups, based on whether machine learning predicts them to be a target or not. A simple comparison of the growth rate of disbursed bank loans in the years following the provision of the guarantee shows that the machine-learning-targeted group performed better (Figure 3). The result is confirmed by applying a more rigorous regression discontinuity design (as in de Blasio et al. 2018), which shows a larger impact of the guarantee on credit availability for the machine-learning-target group.

Figure 3. Performance of Guarantee Fund beneficiary firms by machine learning-target status

Issues with machine learning targeting

Applying machine-learning methods to policy targeting requires dealing with some controversial topics. One concern arises with the interpretability of the decision rule, which may come at the expense of precision. In some applications, simple algorithms are sufficiently accurate, as with the decision tree in the tax rebate example. In other cases, the preferable machine learning targeting rule is like a black box, as the above-mentioned random forest for the public guarantee scheme. The lack of interpretability of black-box rules can be seen as a threat to transparency, because excluded agents may not easily understand on which grounds they have not been targeted (Athey 2017). The trade-off between precision and transparency depends on the specific application.

However, there is a dimension of transparency, which may be referred to as substantive transparency, that concerns the accountability of the policymaker to accomplish her mission, i.e. using public money in an effective way. In this respect, machine-learning targeting clearly trumps coarse assignment rules.

Another concern is that machine learning-based targeting rules are built by looking at a specific payoff (e.g. improve firm access to credit). To reach this goal, the targeting rule selects specific groups (the targets) and not others. This selection may hurt other policy goals (‘omitted payoffs’; see Kleinberg et al. 2018). For instance, elaborating on the Guarantee Fund example, the public guarantee was also strongly advocated as a counter-measure for the recession. One may wonder whether our machine-learning targeting rule ends up excluding firms located in southern regions, which were more strongly hit by the crisis. This is not the case: the machine learning-based rule prioritises the South, while the Guarantee Fund’s actual rule favours the North.

The last concern is that, when a targeting rule is known, agents may manipulate their observable characteristics to benefit from the policy. This is, however, true for any kind of targeting. Comparing, for instance, the decision tree with the actual rule for the tax rebate, one could see that, in order to get the bonus, the potential recipient should manipulate multiple variables (say, both financial assets and income in the simplest case), while the actual measure is only based on income.