Measuring Results of the El Salvador Connectivity Project

The MCC Compact with El Salvador was a five-year investment (2007-2012) of $449.6 million. The $270 million Connectivity Project is the subject of an independent impact evaluation summarized here.

  • The overall program logic of the Connectivity Project was to connect northern El Salvador with the rest of the country helping to create opportunities for the region’s residents through increased access to markets. The project constructed, improved, or rehabilitated 223.3 km of the Northern Transnational Highway, to reduce transportation costs within the Northern Zone, to the rest of the country, and to neighboring countries to facilitate access to markets, increase productive use of  land, and attract new investments. Increases in accessibility were also thought to improve utilization of health and education services. Together these effects were expected to cause an overall improvement in household welfare.
  • Using both regression discontinuity and continuous treatment methodologies, it was found that the Northern Transnational Highway (NTH) modestly reduced travel time to a household’s nearest market as well as the travel time to various services, and lowered the cost of accessing them. Yet there were no significant changes in agricultural sales, harvests, land values, income or expenditure. There were delays in the construction of the NTH which resulted in most segments only being competed in the later stages of the project. Thus, the evaluation captures short-run effects and longer term benefits may take longer to materialize.
  • Using a computable general equilibrium model, the simulations found that the improvement of the NTH reduces transportation costs by ten percent for all agricultural commodities produced in the northern region. However, reducing transport costs without an increase in productivity associated with it does not appear to have a very big payoff.
  • The following lessons were learned from this evaluation: (1) road infrastructure improvements can reduce travel times very quickly, but changes in people’s behavior, such as agriculture production decisions, may take more than one to two years, (2) MCC should explore the use of relatively low- cost methods of monitoring key intermediate outcomes to better inform the time-pattern and composition of road impacts, and (3) road evaluations should include an assessment of reduced vehicle operating costs because that is a key benefit stream in the economic analysis that is typically used to justify investments in road improvements.
  • This evaluation is complete. However, MCC is planning on contracting an independent evaluator to (i) determine the Post Compact ERR using HDM-4 analysis, (ii) assess the road maintenance regime, (iii) analyze the composition of road users, and (iv) assess the transportation market structure. Information from this second evaluation will be used to determine if MCC will undertake any additional household data collection and analysis to evaluate longer-term household outcomes.

In Context

The MCC Compact with El Salvador was a five-year investment (2007-2012) of $449.6 million in three projects: Connectivity, Human Development, and Productive Development. The Connectivity Project included two major activities: the Northern Transnational Highway (NTH) and the Network of Connecting Roads (NCR). However, in 2009 the Connectivity Project was restructured and the NCR activity was removed. The $270 million NTH component is the subject of an independent impact evaluation released by MCC in April 2017, the results of which are summarized here. This component represents 60 percent of the total compact. Other components of the Compact are the subject of already published or forthcoming independent evaluations.

Pie Chart: El Salvador Compact Spending

Program Logic

The NTH Activity was designed to improve the lives of Salvadorans in the Northern Zone by connecting northern El Salvador with the rest of the country helping to create opportunities for the region’s residents through increased access to markets through the east-west highway. The Activity consisted of the construction of the Northern Transnational Highway, a two lane paved road which serves as a transport artery within the Northern Zone and connects with roads to southern El Salvador, to the new Pacific Ocean port at La Union in the eastern El Salvador, and to the Caribbean ports of Puerto Barrios in Guatemala and Puerto Cortez in Honduras. It constructed, improved, or rehabilitated 223.3 km of the NTH, along with an additional 43.7 km rehabilitated by GOES, allowing the highway to provide contiguous and reliable access to communities in the Northern Zone, as well as to main transportation corridors. Reduction of the transportation costs within the Northern Zone, to the rest of the country, and to neighboring countries was intended to facilitate access to markets, increase productive use of land, and attract new investments. Increases in accessibility could also improve utilization of health and education services. Together these effects were expected to cause an overall improvement in household welfare.

Flow Chart: Program Logic

There were several key assumptions underlying the NTH program logic during the design of the investment:

  • Travel time and cost were major impediments to accessing markets and public services.
  • Traffic would increase substantially after road improvements, partially due to diverted traffic from alternative routes.

·         Road improvements would be properly maintained.

For the complete program logic of the NTH component, please refer to pages 3 and 4 of the final evaluation report.

 

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot not reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

 

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

Indicators

Level

Baseline (2007)

Actual Achieved (Dec 2012)

Target

Percent Complete

Kilometers of roads completed

Output

0

223.32

195.67

114%

Source: Closeout ITT from Dec 2012, which includes data through the end of the compact, based on reporting from the construction contractors and FOMILENIO

 

The average completion rate of output targets is 114 percent and targets were met or exceeded for the only output indicator. 1

 

Evaluation Questions

The evaluation was designed to answer the following questions:

Travel and Coping Costs: Access to Markets and Public Services

  • Does access to the improved NTH increase access to markets in the northern zone? Through reductions in time and/or costs?
  • Does access to the improved NTH increase the use of public services? Specifically, health and education services?

Agricultural Transportation Costs and Incomes

  • Does access to the improved NTH reduce agricultural transportation cost? Specifically, does it affect input transportation costs and freight costs?
  • Does access to the improved NTH improve market participation by increasing the likelihood of going to the market and/or the volume sold in the market?

·         Does access to the improved NTH increase income from agricultural sources?

Off-farm Income, Income Diversification, and Time Allocation

  • Does access to the improved NTH increase the availability of non-farm employment?
  • Does access to the improved NTH promote the creation of non-farm enterprises?
  • Does access to the improved NTH increase income from non-farm sources?
  • How  does  access  to  the  improved  NTH  affect  the  time  allocation  across  labor  and  leisure activities? How does it change the labor allocation between farm and non-farm activities?

Gender and Social Exclusion

  • Do the effects on farm and off-farm activities and income differ by gender or by expenditure levels (initial conditions)?
  • How are the effects on health and education access different for men and women? How are they different for the extremely poor versus the relatively poor?
  • What factors (use of time behavior, sources of income, etc.) might explain the impact (or lack of impact) in a specific subpopulation?

Income and Consumption Patterns

  • If access to the improved NTH increases income, how does this reflect in the consumption patterns of households? Is there an effect on food consumption and/or non-food consumption? How are they different?

Land Values

·         Does access to the improved NTH increase land investments and land values in the northern zone?

Macro-regional Impacts

  • What is the impact of the NTH in the entire economy of the northern zone of El Salvador? And across other regions of the country?

 

The evaluation did not cover the following benefit stream that was modeled in the economic analysis of the program. More detail on this topic can be found in the Evaluation Design Report here: https://data.mcc.gov/evaluations/index.php/catalog/135

  • Vehicle operating cost savings

 

Evaluation Results

The NTH Activity impact evaluation was based on three empirical strategies to identify the effects of a transport artery within the Northern Zone on the outcomes of interest: (1) Regression Discontinuity, (2) Continuous Treatment, and (3) Dynamic Regional Computable General Equilibrium.  The first approach was to exploit regression discontinuities. As the NTH was built in different segments, this created adjacent areas that were differentially exposed to road rehabilitation. Thus, households were compared in the neighboring treatment and control areas to capture the effect of the project. The second  approach exploited the continuous variation in travel times to relevant locations (i.e. markets) resulting from the NTH. The third approach incorporated regional disaggregated sectors for agriculture (North, South, and Central) to measure the effects of the changes in accessibility on the northern region with respect to all other regions of the country. The results from each of these methodologies are presented below; however the regression discontinuity and the continuous treatments results are combined in one table because the results were similar.

 

 

Evaluator

Social Impact with the International Food Policy Research Institute

Impact or Performance?

Impact

Methodology

Regression Discontinuity and Continuous Treatment

Evaluation Period

Program implementation: May 2009 to Nov 2012 with different segments being completed at different times

Baseline data collection: 2009

Interim data collection: 2010, 2011, 2012 Final data collection: 2013

Exposure period after final data collection: One to two years

Outcomes

Travel and Coping Costs: Access to Markets and Public Services

  • Reduced travel time to a household’s nearest market by 3 to 18 minutes.
  • Modest reductions in self-reported travel time to various services and lower costs of accessing those services.

 

Land

  • No impact found on land ownership or land values.

Objective-level Outcomes

Agricultural Transportation Costs and Incomes

  • There was no impact on the probability of cultivating cash crops nor the probability of selling agricultural production.
  • A significant increase in overall self-consumption of agricultural output was found (though this impact varies significantly across regression discontinuity comparison groups and specifications).
  • Results were seen in patterns of agricultural input use. For example, the NTH increased fertilizer use.

Off-farm Income, Income Diversification, and Time Allocation (including results for Gender and Social Exclusion)

  • For women, there were increases in non-agricultural hours of work and decreases in agricultural hours of work. There was some evidence that increases in time allocated to non- agricultural activities were driven by reductions of leisure time and less so from reallocation from agricultural work. This impact is not robust to all of the specifications and is restricted to some of the regression discontinuity groups (i.e. this effect only took place in groups that had longer exposure to the highway).
  • For children, few significant changes were found in the time they spent on leisure, agricultural labor, or on nonagricultural labor.

Effect on household income attributable to MCC

Income and Consumption Patterns

  • No evidence was found to suggest that there were changes in expenditure and income. As most segments were only constructed in the later stages of the project, the results capture short-run effects. Road benefits may take some time to materialize if households have to adapt their behavior.

 

 

 

Evaluator

Social Impact with the International Food Policy Research Institute

Impact or Performance?

Performance

Methodology

Computable General Equilibrium model

Evaluation Period

Program implementation: May 2009 to Nov 2012 with different segments being completed at different times

Baseline data collection: 2009

Interim data collection: 2010, 2011, 2012 Final data collection: 2013

Exposure period after final data collection: One to two years

Outcomes

Macro-regional Impacts

  • The simulations found that an increase in productivity in the agricultural sector in the Northern region of the country has a much larger impact on the Gross Domestic Product (GDP) growth rate than an investment strategy of improving the transportation network. The GDP growth rate rises slightly, but the rate of growth of household consumption actually falls. The additional growth in production does not offset the increase in investment necessary to produce it, given that the investment is financed domestically. The improvement on the NTH reduces the transportation costs by ten percent for all agricultural commodities produced in the north region. Reducing transport costs without an increase in productivity associated with it does not appear to have a very big payoff

Objective-level Outcomes

Not applicable

Effect on household income attributable to MCC

Not applicable

 

Lessons Learned

  • Road infrastructure improvements can reduce travel times very quickly, but changes in people’s behavior, such as agriculture production decisions, may take more than one to two years. The evaluation found no evidence that farmers were increasing agriculture production, changing crops, or selling more as result of the road improvements; however, since the final data collection took place only one to two years after the roads were completed, it’s possible that those behaviors just take longer to manifest.

 

  • MCC should explore the use of relatively low-cost methods of monitoring key intermediate outcomes, such as continued regular traffic counts, to better inform the time-pattern and composition of road impacts. While few if any detectable welfare/income benefits accrued over the time period studied, longer-term impacts cannot yet be ruled out. MCC, along with other donors, have learned that the time pattern of rural road investment benefits can be slower than was projected in this study. Since these findings do not distinguish between small total benefits, or slow realization of benefits, the magnitude of the full eventual benefits remains uncertain. More frequent monitoring of key intermediate outcomes could inform the appropriate time to conduct a follow-up household survey where significant household impacts are expected.

 

  • Road evaluations should include an assessment of reduced vehicle operating costs because that is a key benefit stream in the economic analysis that is typically used to justify investments in road improvements. This evaluation provided valuable information about outcomes for people who live within 30 minutes of the improved road; however it does not assess the traffic patterns of the road and cannot be used to update the HDM-IV model that calculated the ex-ante and close-out Economic Rates of Return.

 

MCC has learned from these lessons, as well as other road evaluations, and is working on incorporating them into how compacts are designed, implemented, and evaluated in these ways:

  • Impact evaluations of roads projects will be pursued sparingly, and only when the model is well identifiable and informative effects reasonably detectable.
  • The focus of data collection will shift away from populations surrounding the roads to road users.  The key beneficiaries of road improvements may not be those residing closest to the road and could instead be the full set of households or businesses gaining improved access to markets via the new or improved roads.
  • Every road evaluation will incorporate the estimation of a post-compact or ex-post ERR based on approved and verifiable measurement standards.

 

Next Steps

This evaluation is complete. However, MCC is planning on contracting an independent evaluator to (i) determine the Post Compact ERR using HDM-4 analysis, (ii) assess the road maintenance regime, (iii) analyze the composition of road users, and (iv) assess the transportation market structure. Information from this second evaluation will be used to determine if MCC will undertake any additional household data collection and analysis to evaluate longer-term household outcomes.

Footnotes
  • 1. These figures are calculated using all non-evaluation indicators with targets in the Northern Transnational Highway