Do better roads really improve lives?

How can improved roads change peoples’ lives? How much do people benefit from road projects? Answering these seemingly simple questions is, in fact, much trickier than it appears.

We recently concluded an impact evaluation to measure the socio-economic impacts of World Bank-financed municipal road improvements on poor rural households in the state of Tocantins, Brazil. After 10 years of study, what were the results and lessons learned? And how did we go about conducting the evaluation?

The study followed a methodology traditionally used in impact evaluations in the social sector and was based on a precedent in Vietnam. Throughout the state, one of the least-developed and least-populated in Brazil, most municipal roads are unpaved with inadequate maintenance. The World Bank’s municipal roads project helped construct 700 concrete bridges and 2,100 culverts crossing rivers and streams, providing year-round access to remote populations that once couldn’t access municipal centers during rainy season.

The anticipated result chain of the project was as follows: improvement of physical accessibility would contribute to increase travel demand to markets, schools and health services. This would, in turn, contribute to improved education, better health and increased business opportunities. Finally, it would result in long-term household income growth.

Our study aimed at measuring these impacts through a “difference in differences with matching,” a method that compares a treatment group (population benefiting from the interventions) and a control group (population that does not), while ensuring similar socio-economic characteristics (or comparability) between groups. An “instrumental variables estimator” was then used to confirm the robustness of the results.

The results show positive socio-economic impacts to rural residents, as well as provides for several policy implications:

• Transport quality improved with more choices in modes of transport, individual car ownership increased as well as bicycle acquisition;
• There was an increased number of girls attending school (which is expected to contribute to increasing job opportunities for women on the long run); and
• The project helped increase the number of agricultural jobs and household income.

What were the main challenges the evaluation faced? Can we draw lessons for the future?

First, some of the expected long term impact may require longer time to materialize than was possible under the study, where surveys were conducted within a few years of works completion. Conversely, longer-term surveys may compromise comparability of the two groups, which might change due to external factors that may influence the characteristics of both groups in different ways.

Second, the sampling can be easily biased, as the beneficiaries of transport projects are generally not randomly assigned but rather targeted. To maintain comparability, we recommend performing a preliminary site survey to identify groups with sufficiently similar characteristics.

Third, the questionnaire used for an impact evaluation needs to be fine-tuned to the specific impacts that the projects are expected to bring. For example, our study intended to measure changes in distances whereas the frequency of impediment to travel would have been more relevant.

And finally, we need a good dose of pragmatism to address unexpected operational issues during the implementation of the surveys. In the present case, the local government decided not to interview the control group to avoid creating unnecessary expectations for those who didn’t benefit from the project. To resolve the absence of a formal control group, we conducted a second survey before the end of the project in some communities of the beneficiary group, which created a de facto control group with, in turn, reduced the risk of sampling bias.

This kind of longstanding study, while challenging, remains extremely valuable in measuring the impacts of projects and policies.

This article appeared on The World Bank’s Transport for Development Blog. Publication does not imply endorsement of views by the World Economic Forum.

To keep up with the Agenda subscribe to our weekly newsletter.

Author: Eric Lancelot is a Senior Transport Engineer for the Transport Unit, Latin America and the Caribbean Region at The World Bank.

Image: Miniature cars move along the elevated freeway. REUTERS/David McNew.