A (Brief) History of MCC and Roads
MCC’s guiding principles of country ownership, accountability, and a focus on results have led the agency, together with its country partners, to develop projects totaling nearly $3 billion in the transportation sector. Since MCC’s creation in 2004, roughly 30 percent of its total compact investment portfolio has funded projects focused on roads, making MCC one of the leading bilateral donors in the transportation sector.MCC’s investments in roads have been far-reaching. The agency has built or rehabilitated 3,400 kilometers of roads in 16 countries around the world, roughly the equivalent distance from New York City to Phoenix, Arizona. These road works have consisted of reconstruction of pavement, drainage, and bridges, with the majority of the projects producing paved roads and a small share applying gravel or another type of treatment. Almost all road activities have incorporated policy and institutional reforms that required partner countries to finance their road maintenance funds as a condition to receiving MCC funding.
Why Roads?
Road investments are attractive to partner governments and donors alike because of their potential to improve access to goods, services, markets, and information. Better roads can reduce both transportation costs and travel times for producers and consumers, saving road users money and allowing more time for productive and leisure activities. Both time and cost savings also allow for greater access to vital services, like schools and health centers, facilitating long-term improvements in health and education when other necessary conditions are in place. Finally, roads can reduce isolation, allowing better flows of information, goods, and services, and improving connectedness.Challenges related to poor road infrastructure and high transportation costs comprise one of the most common barriers to economic growth as identified through MCC’s analyses of constraints to economic growth. Of the 30 constraints analyses that MCC has undertaken with partner countries since 2007, 16 have identified the transport sector as a binding constraint to growth.
Differentiating Economic Rates of Return (ERRs): Original ERRs are forecasts of the likely economic benefits and costs of roads investments, conducted prior to the start of a compact. Closeout ERRs update original ERRs and are produced by MCC after compacts close; they are still forecasts, as some benefits do not start until after a compact is completed. Evaluation-based ERRs are developed as part of MCC’s independent evaluation process, better capturing costs and benefits not seen at initial compact closeout.
MCC’s Review of Road Investments
To assess the success of MCC’s road investments, each one undergoes an independent evaluation to measure whether targeted outcomes were realized. These evaluations also produce learning that will help the agency and interested stakeholders improve future road projects. All completed evaluations are publicly available on MCC’s Evaluation Catalog.As MCC’s early roads projects and their evaluations came to a close, the agency began a review to assess the portfolio’s practices and results and identify areas for improvement. The review was three-fold — examining MCC’s operations, analytics, and results (as captured by its independent evaluations) — with the aim of integrating evaluation results with lessons learned from operational and analytical practices to allow MCC to improve the quality of its work.
The Seven Lessons from MCC’s Roads Review
MCC’s review of its prior experience investing in roads yielded the following seven lessons.Lesson 1: Understand the specific problem that a road investment aims to resolve and let that problem articulation inform project development.
MCC’s road evaluations did not detect significant impacts on targeted outcomes related to agricultural production, tourism, incomes, or consumer prices. This result raised two important questions: did MCC correctly understand the likely impacts of the selected road investments in their particular contexts; and were the evaluations appropriately designed to capture those impacts? Indeed, investigation into the theories of change underlying early road investments revealed that there was little documented data or analysis to support the linkage between road improvements and expected long-term outcomes, like agricultural production or tourism, for many road projects.
With these evaluation findings, MCC better recognizes that the ability of a project to achieve and demonstrate its results requires a clear understanding of the problem being addressed by the investment and an evidence-supported theory of change. This understanding is critical when infrastructure is the chosen vehicle to achieve economic growth and poverty reduction. A well-articulated theory of change should serve as the foundation for project design and guide teams throughout implementation.
Lesson 2: Prioritize and select projects based on a road network analysis.
For MCC’s roads investments to produce the highest return for the largest number of beneficiaries, they should be identified through an analysis of the relevant road network that assesses key criteria, such as current traffic volume and road roughness. This type of information, which would be verified by MCC in the field, would allow economists to estimate expected returns across a wide geographic area and enable MCC to prioritize investments that would be the most economically viable. This process of in-depth investigation before specific road investments have been identified would allow MCC to better understand the completion risk associated with potential roads and avoid those that might not be suited to MCC’s model.
Lesson 3: Address policy and institutional issues in the transport sector up front to ensure sustainability of road investments.
Though many MCC road projects incorporated maintenance-related interventions, these activities were generally small and lower priority than the construction work. Compacts usually addressed maintenance by incorporating it as a condition precedent that made project funding disbursements contingent on the partner country financing its road maintenance fund. However, these measures did not directly ensure that maintenance funds were used each year. While the structural capacity of MCC roads was engineered to last approximately 20 years under a specific maintenance regime, the reality may be that assumptions about partners’ maintenance practices might not hold, leading to a shorter road lifespan.
Looking ahead, timely and appropriate road maintenance procedures are critical to the sustainability of road investments and need to be a key area of focus during compact development. As part of ongoing efforts in policy and institutional reform, MCC will assess partner governments’ needs, plan activities to achieve feasible and measurable institutional improvements, and evaluate those achievements. MCC’s engagement with partner countries should prioritize these issues so that partners are fully committed to implementing sound maintenance practices and allocating sufficient funding to support the sustainability of their existing road network, before MCC commits to capital-intensive road investments.
Lesson 4: Develop guidelines to promote consistent application of economic analysis tools across road projects.
Cost-benefit analysis (CBA) of road projects at MCC is conducted using one of two economic modeling tools, depending on the type of roads being analyzed. However, MCC’s review found that the assumptions used to build each of the models were not always consistently derived and were based on data of varying quality. As a result, MCC is developing a standard set of guidelines for economic analysis of the transport sector, the use of which MCC anticipates will systematize CBA methodology and will yield CBAs of road projects that reflect similar levels of rigor, data completeness, and data quality. Further, scenarios produced by these CBAs will allow MCC to make investment decisions with a better understanding of what is uncertain in the expected returns and what outcomes are needed to make the investment worthwhile.
Lesson 5: Require enhanced design review for road investments throughout the project lifecycle to better manage completion risk and improve investment value.
Costs for roads investments have frequently escalated above estimates produced during compact development, often resulting in a reduced scope for the project or a reallocation of compact resources to increase project funding, both of which decrease the benefits relative to costs of the investment. Unanticipated risks related to resettlement and contractor performance also affected implementation of road projects, suggesting that better risk identification and mitigation plans were needed.
Therefore, MCC will implement a more collaborative and technically driven design review, starting from the project development stage through closeout. The envisioned design review process will formalize how MCC collects and checks fundamental design input data to ensure the viability of assumptions going into the initial economic analysis. Jointly with a partner country, MCC will conduct a value engineering review during the design phase that uses CBA as a tool to determine the most cost-effective and highest-value design solution(s). Once an engineering design solution is selected, MCC will continue its oversight of the construction process through random verification of the design requirements to not only ensure conformity to the design but also to identify further design improvement opportunities that would maximize economic benefits to the population. The final stage of the review process will be the validation by all stakeholders of the completed road, allowing for a more informative closeout economic rate of return (ERR).
Lesson 6: Standardize the content and quality of road data collection across road projects.
MCC has recognized the need to take a more active role in securing accurate and complete data to feed into CBA and project budgeting. In the past, MCC has made investment decisions based on incomplete or unverified information, which resulted in design and implementation challenges. MCC’s review confirmed this issue, finding that data collection approaches for common measures such as traffic counts, travel times, and vehicle operating cost inputs were often poorly documented or unreliable.
MCC will, therefore, standardize the types of data collected for prospective and completed roads projects in such a way that meets the needs of project design and implementation, CBA, and monitoring and evaluation, allowing the agency to make better-informed investment decisions. To this end, MCC has developed guidelines for a standard package of data collection and is revising the standard set of indicators reported by road projects to ensure that monitoring data is collected in a consistent manner throughout the various stages of implementation.
Lesson 7: Better balance cost and the potential for learning when designing road project evaluations.
An assessment of independent evaluations of roads investments to-date identified critical challenges of road project evaluation, including:
- Road projects do not lend themselves easily to rigorous evaluation approaches;
- A road investment’s impact on economic growth often relies on a variety of factors external to the road investment, making the theory of change complex and challenging to validate through an evaluation;
- The size of potential impact complicates road evaluations and can make them more expensive; and
- The timing of data collection is critical but not an exact science.
MCC’s road evaluation approach has also shifted the focus of data collection. Previously, evaluations focused on measuring changes in outcomes for populations residing near the road. However, depending on the type of road, the main beneficiaries may not be those living close to it, but rather those accessing markets that the road now connects. MCC expects that focusing on road users will produce more useful results that provide an indication of how the road is changing behavior. Furthermore, evaluations will focus more on better measuring immediate and intermediate outcomes such as road roughness, travel times, and transportation prices, rather than prioritizing higher-level outcomes. Lastly, every evaluation will now estimate a post-compact ERR using an accepted roads economic model, which will provide a more accurate reflection of returns to the road investment.