2016 Report on Closeout ERRs

Closeout economic rates of return (ERRs) are estimates of project rates of return calculated at project completion, when costs and any change of project scope are known, but benefits are largely still unknown. This report documents facts on Economic Rates of Return (ERRs) on MCC supported projects to serve as a starting point for policy discussion. Key findings from this report are that (i) closeout ERRs have been produced for the majority of projects and MCC funds, (ii) closeout ERRs remain on average above MCC’s 10 percent required threshold, (iii) more than one-third of closed projects have closeout ERRs below MCC’s threshold and (iv) roughly three-fourths of projects have ERRs that decrease from the time the compact Enters Into Force (EIF) to closeout.

Authors

Sandra Ospina (Economic Analysis, Senior Director), Ospinasp@mcc.gov

Marissa Block (Economic Analysis, Research Assistant)

Introduction

This is MCC’s third annual report on Economic Rates of Return (ERRs). The main purpose of the report is to take stock of projected ERRs for compacts that have closed (“closeout ERRs”) and compare them with initial ex-ante ERRs (“original ERRs”).  The report also presents facts on original ERRs for compacts under implementation (Open Compacts).

ERRs play a central role at MCC.  Original ERRs represent projections of project costs and benefits over time, based on data available at the beginning of a compact. A project’s original ERR is calculated during compact development and is used to determine whether MCC will fund the project. Closeout ERRs are updated estimates of original ERRs done shortly after the compact ends. While the costs and final scope of a project at closeout are known, the benefits are still largely unknown. Since benefits extend over 20 years after the completion of a project, closeout estimates are necessarily still projections.  Therefore, closeout ERRs should not be interpreted as the final ERRs of MCC projects.  It is useful to track how ERRs may vary or change in implementation by several characteristics such as country, year, sector or size.  It would also be useful to derive and test hypotheses based upon any patterns documented in this report.  However, this report does not attempt to do so.  Nor does it attempt to test causal relationships or provide policy recommendations.  Nonetheless, it is useful for learning and accountability purposes to compare them with the original estimates.  Further analysis may be pursued in a subsequent report.

The key takeaways from this report are that (i) closeout ERRs have been produced for the majority of projects (63%) and MCC funds (66%), (ii) closeout ERRs are on average above MCC’s 10 percent threshold (13.2%), (iii) more than one-third of closed projects have closeout ERRs below MCC’s threshold and (iv) roughly three-fourths of projects exhibit ERRs that decrease from the time the compact Enters Into Force (EIF) to close.

The report is organized as follows. Section II describes coverage. Sections III, IV and V describe aggregate and cumulative information on ERRs. This information includes facts on ERRs by sector, year, project size and compact among other useful ways to tabulate the data.

Coverage

Table 1 provides the number of closed projects to date, funds committed and funds disbursed. It disaggregates these numbers according to projects with ERRs, with pending ERRs, projects for which ERRs are not planned and projects that were cancelled. The Economic Analysis (EA) division started updating ERRs upon compact closeout in 2011. At end-2016, EA has estimated and published closeout ERRs for 86 projects on MCC’s external website. 1 As Table 1 shows, MCC has completed 137 projects with a disbursed value at compact closeout of approximately US$6.5 Billion. These projects span 21 compacts. Once the remaining seven closeout ERRs are updated, 77 percent of all funds disbursed will have a closeout ERR (68 percent of all projects).

There may be various reasons why some projects may not have a closeout ERR. We tabulated the stated reasons provided by country economists assigned to each compact for the lack of a closeout ERR in Table 6 in Annex II. 2 The three broad explanations given were compact termination and cancelled projects, insufficient data at closeout, or small project size. Specifically, five projects were cancelled due to termination of the Mali and Madagascar compacts.  In several cases, there was insufficient data at closeout to update the ERR. Some projects were considered too small (i.e. less than $10 million) to justify the estimation of a closeout ERR. 3

Table 1
Overall Description of Data – Closed Compacts
Number Funds Committed Funds Disbursed
Total Closed Projects 137 6,560.47 6,503.98
Projects with Closeout ERR 86 3,888.59 4,260.64
Projects without Closeout ERR 51 2,671.88 2,243.34
Closeout ERR pending 7 926.23 733.7
Projects for which no ERR Will Be Calculated 44 1,745.65 1509.64
Completed, but no ERR Planned 28 895.15 888.52
Terminated or Cancelled Projects 16 850.5 621.12

Characteristics

We note that in principle ERRs are estimated at the lowest level of disaggregation – where any activity or cluster of sub-activities can be separated in terms of delivering defined benefit streams and therefore for which a decision on investment can be made separately. The lowest possible level of logical disaggregation may occur at the sub-activity level. When reporting summaries for compact statistics—as we do here—ERRs are sometimes aggregated by project or by activity using funds as weights. The majority of reported ERRs, 64 out of 85 closeout ERRs, are calculated at the activity level while the remaining 21 are calculated at the project-level.

Table 2 shows weighted and unweighted ERR averages for all closed projects with an ERR and by selected categories. It also shows the number of projects included in the respective average calculation and the funds disbursed. The table shows the overall average, averages by sector, by closing year, by three different project sizes, and by compact. The compact category is organized according to GDP per capita in the year he country was selected.

The top line entry in Table 2 shows the average ERR for the 86 projects. The weighted average (by disbursed funds) at compact closeout is 13.2 percent. The ERRs reported in Table 2 range from -5.5 to 34.7%.

While the reported averages per sector and per project size usually do not vary dramatically, the averages by year and by country compact show high variability. The average closeout ERR for compacts that closed in 2015 is lower than other years. The compacts that closed in 2015 are Moldova and Senegal. In the compact category the weighted average ERRs range from -5.5 to 20.3%. Moldova’s ERR at -5.5 percent is the lowest overall followed by Nicaragua, Burkina Faso, Tanzania, Senegal, and Lesotho. El Salvador, Morocco, Ghana and Namibia’s weighted closeout ERR stand out at around 20 percent. We note that the number of compacts reporting all ERRs planned is 17. 4

Figure 1 depicts all projects with an original and a closeout ERR. The interactive feature of this graph shows specific project information. The projects are presented from left to right in ascending order according to the original ERR. The red solid line represents the original ERRs and the blue solid line represents the closeout ERRs. The solid black line is drawn at MCC’s current 10% ERR threshold. All 30 projects with a closeout ERR below this threshold, nearly 34 percent of all projects, are labeled.

Figure 1 shows that the majority of the original project ERRs are above their corresponding closeout ERRs. Future work should investigate the proximate reasons for why this is so. More than one third of closed projects have ERRs below MCC’s threshold and roughly three-fourths of projects exhibit ERRs that decrease from EIF to close. Of the 80 projects for which there are original and closeout ERRs, the weighted average for original ERRs is 17.9 percent compared to the 12.8 percent for corresponding closeout ERRs. Figure 1 also reveals substantial variation across project closeout ERRs. They range from -8 percent to 109 percent.

Figure 2 shows the difference in percentage points between closeout ERRs and original ERRs. The figure is organized from lowest to highest difference (closeout ERR – original ERR). The red bars depict the difference in ERRs for projects with ERRs that fell below the threshold while the green bars depict this difference for projects with a closeout ERR above the threshold. Figure 2 shows that for most projects the ERRs decreased, sometimes exceeding a fall of 20 percentage points. The right hand tail in Figure 2 shows the 17 projects for which the closeout ERR increased relative to their original ERR (21% of all projects with original and closeout ERRs). The average of the absolute value of the gap is 5.9 percentage points.

Table 2
All Projects with Closeout Data
Sector Weighted Closeout ERR Unweighted Closeout ERR Total Projects Funds Disbursed (US$M)
Agriculture and Irrigation 14.5% 15.6% 21 1031.20
Cap Bldg & Inst Dev 34.7% 34.7% 1 54.90
Energy 15.1% 13.7% 4 101.60
Finance, Invest., Trade 11.5% 11.0% 5 94.33
Health, Ed. & Community 15.1% 21.7% 14 540.94
Land 12.1% 13.8% 6 190.90
Other* 14.2% 15.3% 3 66.10
Transport 11.6% 9.8% 18 1763.83
WASH 11.7% 12.7% 14 416.85
Closeout Year Weighted ERR Unweighted ERR Total Projects Funds Disbursed (US$M)
2011 11.5% 9.9% 10 748.22
2012 20.2% 14.5% 15 665.51
2013 15.1% 16.0% 38 1472.91
2014 9.6% 16.4% 16 543.90
2015 5.0% 4.4% 5 655.20
2016 19.5% 23.7% 2 174.90
Size at Closeout (US$) Weighted ERR Unweighted ERR Total Projects Funds Disbursed (US$M)
< 50M 15.5% 15.5% 58 1241.15
50M – 100M 13.2% 13.9% 16 1023.25
> 100M 11.7% 11.3% 12 1996.25
Country Weighted ERR Unweighted ERR Total Projects Funds Disbursed (US$M)
Mozambique 15.2% 20.7% 8 310.80
Burkina Faso 3.4% 2.9% 5 332.00
Tanzania 10.0% 6.8% 3 77.20
Benin 12.8% 11.2% 3 278.13
Senegal 7.0% 6.9% 4 553.6
Lesotho 8.7% 9.8% 6 297.72
Ghana 20.1% 13.9% 7 248.41
Nicaragua 2.1% 2.1% 1 57.80
Moldova -5.5% -5.5% 1 101.60
Georgia 12.2% 8.0% 3 309.70
Mongolia 12.4% 13.0% 7 223.39
Philippines 19.5% 23.7% 2 174.90
Armenia 12.3% 14.6% 2 36.90
Morocco 20.2% 19.8% 14 563.80
Vanuatu 10.3% 10.3% 1 65.69
El Salvador 20.3% 15.1% 8 417.10
Namibia 19.2% 22.5% 11 211.90
All Projects 15.2% 15.7% 76 3232.71

*This includes Namibia’s tourism projects.
Note: The total number of projects in Table 2 excludes projects for which no ERR will be calculated and projects with pending ERRs. Cabo Verde and Honduras are excluded from the table as their projects ERRs aren’t estimated.

Chart 1

Chart 2

Projects with closeout ERRs below the threshold

Table 3 disaggregates projects with closeout ERRs that fall below 10 percent in different categories. The table shows the number of projects per sector, year, project size and country. The top line entry for the table shows that the average ERR for the 30 projects is 3.2 percent corresponding to US$1.6 billion or 25 percent of disbursed funds. The table shows that more than half (57 percent) of the projects are small projects (<US$50M). However, only a quarter of all small projects fall below the threshold, whereas half of all projects in the US$50-$100M category fall below the threshold  Finally, one third of the disbursed funds correspond to three high value projects that sum up to US$533.75M. The three projects are Senegal’s RN6 ($262.7M), Mozambique’s Rehabilitation/Construction of Roads Project (US$127.7M), and Lesotho’s Health Sector project (US$128.5M).

For Nicaragua, Moldova, and Burkina Faso, all projects fall below the threshold. Almost all sectors have at least one project with an ERR below the threshold. The transport sector has the largest number of projects below the ERR threshold amounting to 56 percent of total transport projects and 46 percent of the value of the sector’s portfolio. Water, Sanitation and Hygiene (WASH) projects are similarly high – 43 percent of projects and 38 percent of the value of the sector’s portfolio. Land projects that fall below the threshold account for 55 percent of the value of the sector’s portfolio, the highest of all sectors. Two of the four energy projects fall below the threshold and account for 36 percent of the value of energy’s portfolio. The three health, education, and community projects account for 50 percent of the sector’s total (US$432.6M), and one of these projects, the Lesotho project on health, accounts for 33 percent of the sector’s total. On the other hand, less than 30 percent of agriculture and irrigation projects fall below the threshold, which accounts for only one quarter of the value of the sector’s disbursed funds.

Table 3
Closed Projects with Closeout ERR below 10% Threshold
Sector Weighted ERR Unweighted ERR Total Projects Funds Disbursed (US$M)
Agriculture and Irrigation -0.1% 1.1% 6 258.30
Cap Bldg & Inst Dev 0 0.00
Energy 7.7% 8.2% 2 36.90
Finance, Invest., Trade -0.9% -0.9% 1 15.10
Health, Ed. & Community 6.5% 6.4% 3 218.75
Land 5.8% 5.8% 2 105.40
Other 0 0.00
Transport 3.5% 3.3% 10 818.50
WASH -0.2% -0.3% 6 160.20
Closeout Year Weighted ERR Unweighted ERR Total Projects Funds Disbursed (US$M)
2011 3.6% 4.1% 4 183.30
2012 1.0% 1.2% 3 73.70
2013 4.8% 2.2% 11 469.95
2014 4.5% 4.3% 8 427.30
2015 0.4% 1.5% 4 458.90
2016 0 0.00
Size at Closeout (US$) Weighted ERR Unweighted ERR Total Projects Funds Disbursed (US$M)
< 50M 1.7% 1.9% 17 337.30
50M – 100M 5.5% 5.1% 8 539.50
> 100M 2.1% 2.2% 5 736.35
Country Weighted ERR Unweighted ERR Total Projects Funds Disbursed (US$M)
Mozambique 5.4% 1.7% 3 171.70
Burkina Faso 3.4% 2.9% 5 332.00
Tanzania 2.9% 2.7% 2 31.50
Benin 5.8% 5.8% 1 39.20
Senegal 2.1% 3.8% 3 357.30
Lesotho 3.1% -1.4% 2 171.55
Ghana 0.0% 0.0% 2 53.90
Nicaragua 2.1% 2.1% 1 57.80
Moldova -5.5% -5.5% 1 101.6
Georgia 3.6% 4.2% 2 86.30
Mongolia 9.4% 9.0% 2 76.50
Philippines 0 0.00
Armenia 0 0.00
Morocco -0.9% -1.0% 2 18.70
Vanuatu 0 0.00
El Salvador 3.6% 3.6% 1 19.80
Namibia 8.3% 6.7% 3 95.30
All Projects 3.2% 2.8% 30 1613.15

According to staff economists the main reasons closeout ERRs fell below the threshold are increased costs and reduction in scope of the project. Tables 8a and 8b in Annex II list reasons stated by compact-assigned economists per project for the decrease in ERRs below the threshold. 5 The increased costs may have led in some instances to the reduction in scope. Other reasons that staff economists have listed include poor data at EIF and poor project design.

Original ERRs for Compacts under Implementation

Table 4 shows the number of projects under implementation as well as the funds committed. The table disaggregates the number of projects according to projects with original ERRs, with pending ERRs and projects for which no ERR will be calculated. The table shows that 80 percent of the 25 open projects have an original ERR. They correspond to 83 percent of committed funds. The majority of the funds for which ERRs are not estimated correspond to the Indonesia compact (US$M 383). 6 The pending ERR is for Liberia.

Table 4
Overall Description of Data – Open Projects
Number Funds Committed
Total Open Projects* 25 2,406.56
Projects with Original ERR 20 1986.46
Projects without Original ERR 5 420.1
Original ERR pending 1 21.1
Projects for which no ERR Will Be Calculated 4 399

Total open projects is defined as all projects of compacts which have entered into force. Additionally, Jordan is included as an open compact as the closeout ERRs are due in July 2017.

Figure 3 shows compact original ERR averages weighted by committed funds for all compacts that have Entered Into Force (EIF) when the five year time limit for completing project disbursements begins. The figure organizes the compacts according to EIF year starting with the earliest compact to EIF. The green bars denote the closed compact original ERRs and the yellow bars the open compacts’ original ERRs The figure shows that for open compacts, except Malawi, the average of the original ERRs appears to be lower than the average of the ERRs for closed compacts. The dotted trending line has been added for emphasis on the apparent downward trend on original ERRs.

Table 5 shows the number of projects under implementation as well as the funds committed. The table shows the number of projects and average ERRs per sector, year, project size and country. The table includes the Jordan compact for which the closeout process is underway. As with closed compacts, countries are ordered based on GDP per capita in the year selected. The weighted average ERR for all 20 projects is 21 percent. If Malawi’s original ERR is replaced with 18.7 percent—the ERR revised one year later—the weighted average ERR falls to 16.3 percent.

Chart 3

Table 5
All Open Projects with Original ERRs
Sector Weighted ERR Unweighted ERR Total Projects Funds Committed (US$M)
Agriculture and Irrigation 0 0.00
Cap Bldg & Inst Dev 0 0.00
Energy 30.3% 25.8% 6 739.8
Finance, Invest., Trade 18.6% 18.6% 1 42.4
Health, Ed. & Community 12.5% 15.3% 6 348.3
Land 22.0% 22.0% 1 18.75
Other 0 0.00
Transport 20.3% 20.3% 1 109.6
WASH 16.4% 16.0% 4 672.61
EIF Year Weighted ERR Unweighted ERR Total Projects Funds Committed (US$M)
2011 16.0% 16.5% 2 275.01
2012 43.2% 28.0% 3 393.75
2013 15.9% 15.0% 2 486.3
2014 12.7% 16.8% 4 116.1
2015 16.5% 16.8% 3 252.7
2016 15.7% 20.3% 5 407.6
Size at EIF (US$) Weighted ERR Unweighted ERR Total Projects Funds Committed (US$M)
< 50M 19.0% 20.6% 9 204.55
50M – 100M 8.0% 8.0% 1 56.5
> 100M 22.1% 19.2% 9 1670.41
Country Weighted ERR Unweighted ERR Total Projects Funds Committed (US$M)
Liberia 11.0% 11.0% 1 201.6
Malawi 48.1% 48.1% 1 332.2
Zambia 17.0% 17.0% 1 354.8
Ghana II 19.7% 21.8% 4 261
Indonesia 13.0% 13.0% 1 131.5
Georgia II 12.7% 16.8% 5 116.1
Cabo Verde II 16.4% 18.0% 2 61.55
El Salvador II 16.5% 16.8% 3 252.7
Jordan 16.0% 16.5% 2 275.01
All Projects 21.2% 19.1% 20 1986.46

Figure 4 plots committed project funds and their corresponding original ERRs for open compacts, including Jordan. The solid black line is drawn at the 10% threshold level. Georgia II’s Education Infrastructure project falls just below the threshold with an ERR of 8 percent. Figures 3 and Figure 4 are updated on the basis of closeout ERRs completed in 2016 to present trends in project’s economic rates of return for reporting purposes.

Chart 4

Annex I

2016 Updates to the Closeout ERR Report

The following list details the changes and additions of projects from the dataset used to calculate the statistics found in the 2016 Report on Closeout ERRs. Overall, the total number of closed projects increased from 122 to 137 and twelve new closeout ERRs were added. The total number of open projects decreased from 35 to 25.

Updates

  • The funds at closeout for Ghana’s Trunk Roads is $29.9 million, not $33.2 million. As a result, Ghana’s closeout ERR is slightly higher.
  • One of Morocco’s Fruit Tree Productivity activities was double counted. At EIF, there were two separate original ERRs for the intensification and rehabilitation and the intensification and expansion activities, but at closeout these activities were combined into one ERR.
  • Benin’s Access to Markets project was changed from the Finance, Investment, and Trade sector to the Transport sector.
  • Three activities were added to Burkina Faso’s Roads Project, all of which do not have closeout ERRs.
  • El Salvador’s Community Infrastructure Activity was changed from the Health, Education, and Community sector to the Transport sector. The activity rehabilitated a small number of rural roads and bridges.
  • Three projects for Moldova were moved from open to closed. The original ERR for the Access to Agricultural Finance was removed as this ERR was never officially cleared for public release. Additionally, two activities within the Transition to High Value-Added Agriculture Project were added. Costs for both of the activities are included in the ERR for the Rehabilitation of the Centralized Irrigation System Activity.

Additions

  • Four closeout ERRs were added for Burkina Faso, two for Philippines, four for Senegal, and one for Moldova.
  • The original ERR for Liberia’s Energy Project is now included. The original ERR for the Roads project is pending.
  • Original ERRs for the three projects in El Salvador II are now included.
  • Six projects for Ghana II are included in the open compact data and there are three original ERRs. Funds for the Regulatory Strengthening and Capacity Building Project are accounted for in the ERRs.

Annex II

Table 6 – Stated Reasons for Projects that lack an ERR*

Projects without Closeout ERR: 50
Projects for which no ERR will be Calculated: 44

Completed, but no ERR planned: 28
Country Project Activity Reason
Armenia Irrigated Agriculture Irrigation Infrastructure A revised ERR was calculated in the third year of the compact (2008). At closeout, it was intended that the independent evaluation would include a post-compact ERR. However, this did not occur.
Benin Access to Justice N/A Insufficient data on benefits at closeout to calculate an updated ERR. The original model’s assumptions could not be validated. As the model did not provide specific information to estimate benefits, a simple update on costs ay closeout is misleading.
Burkina Faso Agricultural Development Diversified Agriculture The compact was approved without an ERR for this activity. At closeout, there was insufficient data and information to produce an ERR.
Burkina Faso Bright 2 Project N/A Insufficient data and information to quantify benefits at closeout. A post-compact ERR was estimated by the independent evaluator two years post-compact, with the estimate ranging from 3% to 8%.
Burkina Faso Roads Project Development of Rural Roads Insufficient data and information. The compact was approved without an ERR for this activity. At closeout no data was available on traffic volume, road roughness, or other data required for a cost-benefit analysis.
Burkina Faso Roads Project Capacity Building and Technical Assistance for Road Maintenance Insufficient data and information. The compact was approved without an ERR for this activity. This was a $5.1 million technical assistance activity with unclear outputs.
Burkina Faso Roads Project Incentive Matching Fund for Periodic Road Maintenance Insufficient data and information. The compact was approved without an ERR for this activity. At closeout no data was available on traffic volume, road roughness, or other data required for a cost-benefit analysis.
Cabo Verde Infrastructure Port of Praia An independent evaluation is being finalized. A post-compact ERR was expected to be included in the evaluation. However, a post-compact ERR was not included in the 2017 summary report presented to MCC.
Cabo Verde Infrastructure Roads and Bridges Insufficient data and information; initial estimates at the time of closeout indicate that the ERR would be below the 10% threshold; however, an ERR is expected to be calculated as part of an ex-post evaluation.
Cabo Verde Watershed Management & Agricultural Support  N/A Insufficient data and information at closeout. An evaluation team is assessing the use of the irrigation structures five years after the closing of the first compact.
Georgia Enterprise Development Agribusiness Development Insufficient data and information.
Georgia Enterprise Development Georgia Regional Development Fund Insufficient data and information.
Ghana Agricultural Development Farmer and Enterprise Training, Credit & Land Tenure There is little evidence of expansion and intensification of farming and significant restructuring of project components weaken the capacity to calculate a relevant closeout ERR.
Ghana Agricultural Development Post-Harvest Handling Insufficient data and information at closeout to calculate an updated ERR and significant redesign of the project make it difficult to calculate a relevant closeout ERR.
Ghana Rural Development Community Services: Electrification The project was significantly de-scoped and at closeout the budget was too small to calculate an ERR.
Ghana Rural Development Financial Services Insufficient data and information.
Ghana Rural Development Procurement Activity was too small to calculate an ERR ($1.5M).
Ghana Transportation Rehabilitation of Ferry Services Insufficient data and information to calculate an updated ERR at the time of closeout. After re-evaluation, an ERR is planned.
Honduras Rural Development N/A Original ERR was dependent on assumptions that could not be backed up with data.
Honduras Transportation N/A Insufficient data and information at closeout to calculate an updated ERR. A post-compact ERR was requested to be incorporated in the impact evaluation.
Lesotho Water Sector Wetlands and Restoration Conservation Activity was too small to calculate an ERR ($4M).
Lesotho Private Sector Development Gender Equality in Economic Rights Activity was too small to calculate an ERR ($1.04M).
Moldova Transition to High Value-Added Agriculture Access to Agricultural Finance Same as reason given for no original ERR; Data needed to calculate an ERR was never made available by the MCA.
Namibia Education Regional Study and Resource Centers Same as reason given for no original ERR; There is no evidence that libraries can be linked to student outcomes.
Nicaragua Rural Business Development N/A Original ERR was dependent on assumptions that could not be backed up with data.
Tanzania Energy Kigoma Solar Power Insufficient data and information
Cancelled Projects: 16
Country Project Activity Reason
Burkina Faso Agricultural Development Access to Rural Finance The activity was terminated after MCC determined it had not made sufficient progress in achieving project targets.
Cabo Verde Private Sector Development Financial Sector Reform N/A
Lesotho Private Sector Development Credit Bureau and National ID The GoL officially requested removing the National ID card aspects of the National ID and Credit Bureau Activity from the compact, to align the activity with its own national e-passport initiative.
Lesotho Private Sector Development Payment and Settlement Systems N/A
Madagascar Agricultural Business Investment N/A Compact was terminated in 2009 due to the military coup.
Madagascar Financial Sector Reform N/A Compact was terminated in 2009 due to the military coup.
Madagascar Land Tenure N/A Compact was terminated in 2009 due to the military coup.
Mali Airport Improvement N/A Compact was terminated in 2012 due to the military coup.
Mali Alatona Irrigation N/A Compact was terminated in 2012 due to the military coup.
Mali Industrial Park N/A Compact was terminated in 2012 due to the military coup.
Mongolia Rail N/A The Government of Mongolia requested to withdraw the Rail Project from the compact due to an inability to fulfill certain conditions for transparency by the Mongolian rail company’s Russian partners.
Mongolia Roads Bayanzurkh Bridge Project was cancelled due to two failed procurements and insufficient time remaining within the compact to initiate a third procurement.
Mongolia Roads UB-Nalaikh Road Project was cancelled due to two failed procurements and insufficient time remaining within the compact to initiate a third procurement.
Mozambique Water & Sanitation Nacala Urban Water Supply Cancelled due to under-performing contractor.
Nicaragua Property Regularization N/A Compact was partially terminated in response to actions by the Nicaraguan government inconsistent with MCC’s eligibility criteria; as a result, funding for this activity stopped.
Tanzania Energy Malagarasi Hydro (Energy) Cancelled due to environmental considerations.

Projects without Original ERR: 29
Open Projects without Original ERR: 5

Projects for which No ERR will be Calculated: 4
Country Project Activity Reason Closeout ERR Status
Georgia II Improving General Education Quality Education Assessment Support There is no literature or standard practice on methods to quantify the benefits of testing. It is too early in the compact to determine.
Ghana II Access Project N/A At the time of the investment decision, the project was not fully designed. It is too early in the compact to determine.
Indonesia Green Prosperity N/A As two of the four activities with the project are grant facilities, the size and composition of the individual grants at EIF were unknown and it was not possible to calculate an original ERR. The two remaining activities are complementary to the grant facilities. As a grant facility, there will likely be no project-level closeout ERR. However, ERRs for individual grants will be calculated.
Indonesia Procurement Modernization N/A Insufficient information and data to quantify benefits. A closeout ERR is planned.
Projects Included in Other ERRs: 1
Country Project Activity Reason
Ghana II Regulatory Strengthening and Capacity Building Project N/A The costs of this project are included in the total costs of the ECG Project and the Demand Side Management Project.
Closed Projects without Original ERR: 22
Projects for which no ERR will be calculated: 17
Country Project Activity Reason
Burkina Faso Agricultural Development Access to Rural Finance Compact approved without a cost-benefit analysis for this activity.
Burkina Faso Agricultural Development Diversified Agriculture Compact approved without a cost-benefit analysis for this activity.
Burkina Faso Rural Land Governance Compact approved without a cost-benefit analysis for this activity. In 2011 an ERR was calculated for the phasing decision at 10.8%.
Burkina Faso BRIGHT 2 Project Compact approved without a cost-benefit analysis for this activity. This project built on MCC’s 2005 threshold program.
Burkina Faso Roads Project Development of Rural Roads N/A
Burkina Faso Roads Project Capacity Building and Technical Assistance for Road Maintenance N/A
Burkina Faso Roads Project Incentive Matching Fund for Periodic Road Maintenance N/A
Cabo Verde Private Sector Development Partnership to Mobilize Investment Activity was considered too small to construct and ERR ($5M).
El Salvador Productive Development Finance and Investment Support This was a grant facility, so there was no data ex-ante to construct an original ERR.
Ghana Rural Development Financial Services Activity was considered too small to construct an ERR ($24M).
Ghana Rural Development Procurement Activity was considered too small to construct an ERR ($2M).
Lesotho Water Sector Wetlands and Restoration Conservation Activity was considered too small to construct an ERR ($5M).
Lesotho Private Sector Development Payment and Settlement Systems Activity was considered too small to construct an ERR ($1.7M).
Lesotho Private Sector Development Gender Equality in Economic Rights Activity was considered too small to construct an ERR ($1.04M).
Madagascar Agricultural Business Investment N/A
Mali Industrial Park N/A
Moldova Transition to High Value-Added Agriculture Access to Agricultural Finance Data needed to calculate an ERR was never made available by the MCA.
Namibia Education Regional Study and Resource Centers There is no evidence that libraries can be linked to student outcomes.
Projects were added after EIF: 5
Country Project Activity Reason
Mongolia Energy & Environment MC Energy Investment Fund This project is a result of funds being reallocated from the cancelled Rail project.
Morocco Artisan and Fez Medina Literacy & Vocational Training This project is the result of a reallocation of funds away from other projects.
Morocco Fruit Tree Productivity Catalyst Fund The activity was added after a re-scoping decision. The funds for the activity are included in the other Fruit Tree Productivity activity closeout ERRs.
Namibia Education Vocational and Skills Training, VGTF The activity was added to complement the National Training Fund.
Tanzania Energy Kigoma Solar Power This was added at the re-scoping decision following the cancellation of the Malagarasi Hydro Activity.

*Note: Reasons provided by economists on country teams. The stated reasons have not been systematically cross-checked or validated.

Table 7
Compact-Level Summary of Projects without a Closeout ERR
Country Cancelled Projects Projects with No ERR Planned Pending ERRs As a % of Total Projects As a % of Total Funds Disbursed
Armenia 1 33.3% 76.7%
Benin 1 25% 6.5%
Burkina Faso 1 5 1 58.3% 35.2%
Cabo Verde 1 4 100% 100%
El Salvador 0% 0%
Georgia 2 40% 14.4%
Ghana 6 46.2% 41.5%
Honduras 2 100% 100%
Lesotho 2 2 40% 2.3%
Madagascar 3 100% 100%
Mali 3 100% 100%
Moldova 1 1 40%** 55%**
Mongolia 3 30% 0.20%
Morocco 0% 0%
Mozambique 1 11.1% 16%
Namibia 1 8.3% 8.2%
Nicaragua 1 1 66.7% 40.8%
Philippines 1 33%*** 55.6%***
Senegal 0% 0%
Tanzania 1 1 4 66.7%**** 85.9%****
Vanuatu 0% 0%

*This will become 50% of total projects and 28.3% of total funds disbursed when the pending ERR becomes available.
**This will become 20% of total projects and 7.5% of total funds disbursed when the pending ERR becomes available.
***This will become 100% of total projects and total funds disbursed when the pending ERR becomes available.
****This will become 22.2% of total projects and 18.2% of total funds disbursed when the four pending ERRs become available.

Table 8A
Major Causes of Closeout ERRs That Fell Below the Threshold
Country Project/Activity Reduced Scope Increased Costs Reduced Beneficiaries Project Design Poor Data at EIF Low Carbon Credit Prices Government/Third-Party Progress
Benin Access to Land X X
Burkina Faso** Rural Land Governance X
Burkina Faso Roads
Burkina Faso Di Irrigation
El Salvador* WASH X X X
Georgia Regional Infrastructure Dev. X X
Georgia* Energy Rehabilitation: North/South Gas Pipeline X X
Ghana Irrigation X
Ghana Trunk Roads X X
Lesotho Health Sector X X X
Lesotho Rural Water Supply X
Moldova Rehabilitation of CIS X X
Mongolia Choir-Sainshand Road X
Mongolia Wind Park X X
Morocco* Enterprise Support X X X
Morocco* Artisan Production X
Mozambique Quelimane WASH X X
Mozambique Roads X X
Mozambique Mocuba WASH X
Namibia National Training Fund X
Namibia Improving the Quality of General Education X
Namibia Land Access and Management Activity and Livestock Activity X
Nicaragua* Transportation X
Senegal Podor Irrigation
Senegal RN2 X
Senegal RN6 X
Tanzania Morogoro Water Supply X
Tanzania Mafia Airport X

*Projects with completed evaluations.
**No original ERR.

Table 8B
Major Causes of Closeout ERRs that Fell Below the Threshold
Benin Access to Land While the target number of land titles were surveyed, only a quarter of them were expected to be issued, greatly reducing project scope. The project also experienced moderate cost increases.
Burkina Faso** Rural Land Governance
Burkina Faso Roads Approved at the time of the investment decision with ERRs below the hurdle rate
Burkina Faso Di Irrigation Approved at the time of the investment decision with ERRs below the hurdle rate
El Salvador* WASH The original model assumed water-borne disease would be completely eliminated by provision of clean water, which was not the case.
Georgia Regional Infrastructure Dev. Inadequate Water Supply
Georgia* Energy Rehabilitation: North/South Gas Pipeline Gas losses, the main anticipated benefit, were substantially reduced before the rehabilitation work began due to maintenance that was completed by the government after original economic analysis, but prior to the start of MCC-funded projects.
Ghana Irrigation Initial assessments indicate that two components of this project are not adequately maintained and that the third has deteriorated considerably.  Given the current state of the infrastructure, it is unlikely that the project will have a positive return.
Ghana Trunk Roads
Lesotho Health Sector The ERR reduced after factoring in increased costs, borne by the Government of Lesotho, decreased benefits due to incorrect baseline assumptions, as well as adding other donor costs for the benefit streams not attributable to MCC funding alone.
Lesotho Rural Water Supply Limited economic potential in target community
Moldova Rehabilitation of CIS The closeout ERR is significantly lower than the original estimates due to higher construction costs than expected, higher costs due to an unexpected exchange rate depreciation, and less optimistic assumptions about transition rates of land to HVA (due in part to limited access to redit and limited access to foreign markets).
Mongolia Choir-Sainshand Road Lower-than-expected traffic
Mongolia Wind Park Government inaction; construction delays
Morocco* Enterprise Support Costs were greater than originally expected, which was the primary reason the decision was made not to scale up. Additionally impacts on value-added growth and business survival rates are difficult to detect and measure after a single year.
Morocco* Artisan Production Artisans were both reluctant to adopt the new kilns or unable to obtain financing for their 20% contribution to the purchase of the kiln. As a result, at the time of the evaluation, only two gas kilns were operational compared to the target of 144 kilns.
Mozambique Quelimane WASH The original model was estimated based on old data and the project was put on hold.  Once reliable data was obtained, project was restarted with leftover funds, but the scope was greatly reduced.
Mozambique Roads The project was reduced from four road segments totaling 491 km to two segments totaling 253 km.
Mozambique Mocuba WASH The project was significantly de-scoped in order to focus on an emergency activity targeted at a hospital.
Namibia National Training Fund There were delays in launching the fund.
Namibia Improving the Quality of General Education
Namibia Land Access and Management Activity and Livestock Activity
Nicaragua* Transportation The project was partially terminated and funding ceased to activities not already contracted. Of those activities that were funded, costs increased.
Senegal Podor Irrigation Exogenous factors (i.e. weather) delayed harvest season, subsequently delaying realization of benefits
Senegal RN2 Incorrect baseline assumptions of IRI; re-measured mid-compact by MCC independent engineer
Senegal RN6 Incorrect baseline assumptions of traffic counts
Tanzania Morogoro Water Supply Due to construction delays, the treatment plants were completed after the compact by the Tanzanians, impacting the percentage of O&M costs recovered and the volume of water produced by the plants at compact end.
Tanzania Mafia Airport

*Projects with completed evaluations.
**No original ERR.

In Tables 8A and 8B above, twelve evaluations are underway, and seven evaluations are planned. Three projects currently do not have any evaluations planned.

Annex III

Excerpts from MCC’s Summary of Findings

I. Measuring Results of the Morocco Enterprise Support Project

Economic Rate of Return (ERR) Estimates and Decision Not to Scale-up the Enterprise Support Pilot

As there was insufficient empirical evidence on the extent to which post-creation business training would increase revenues, productivity and survival rates of new enterprises, the Enterprise Support Project was structured as a two-year pilot program subject to a rigorous impact evaluation using a randomized design. Scale-up of the project was made contingent upon the results of the pilot program producing an estimated ERR above 10 percent for the scale-up project. The ERR prior to the implementation of the pilot program was estimated at 13.5 percent. Upon results one year after implementation, the overall ERR including scale-up phase projections was estimated at 1.21 percent while the ERR of the scale-up phase alone was estimated at 7.21 percent. 7 MCC decided not to proceed with scale-up of the Enterprise Support Project as the revised ERR estimate did not pass the hurdle-rate of 10 percent and was not estimated to generate sufficient benefits to justify the costs of the project.

ERR at Closeout (Project-wide, not exclusive to Support to Small and Medium Enterprises (SMEs) Activity)

At closeout, the results of the impact evaluation became available and it was possible to revisit some of the benefit assumptions that could not be updated at the time the scale up decision was made. The impact evaluation found SME failure rates were reduced by 30%, rather than the 20% reduction assumed by the original ERR. However, this has essentially no significant impact on the ERR (an interesting problem of itself that we have yet to resolve). The only other significant impact the impact evaluation found was an improvement in value added for the AGR-INDH sub-activity of 30%, vs. 5% as was assumed in the original ERR. Making this change to the parameter increased the ERR for the AGR-INDH sub-activity from -3.5% to 4.4%, a significant improvement but still not higher than the 10% threshold. The overall closeout ERR was -0.9%.”

II. Measuring Results of the Artisan and Fez Medina Project Artisan Production and Promotion and Fez Medina Activities

“Even though potters were convinced of the relevance of replacing traditional kilns with less polluting gas kilns, this activity faced serious reluctance for many reasons: (i) the 20% contribution of potters to the imported gas kiln was beyond the financial capacity of potters; (ii) potters were reluctant to contract loans from banks for various reasons, among which the reimbursement conditions and (iii) Banks were not willing to finance the majority of potters operating informal, unregistered businesses; (iii) the maintenance cost of imported kilns were perceived as a long-term burden. The project design did not take into account the potential of the local gas kiln manufacturers and favored imported kilns from the start. The project should have considered developing a local kiln based on local usage and adapted to the needs and capacities of Moroccan potters.”

III. Measuring Results of the Nicaragua Transportation Project

“The calculation of the ex-post ERR utilized the Roads Economic Decision model (RED) developed by the World Bank. This approach captures the project’s flow of net benefits to road users defined as benefits minus costs which are discounted across time to yield a single value, using pre- and post-construction traffic counts and estimates of future traffic counts. The ex-post ERR estimated by the independent evaluator showed that the roads failed to meet the 10 percent hurdle rate. The average for the project, as a whole, was estimated to be 2.1 percent. Actual capital costs were, on average, 2.2 times greater than those estimated in the feasibility studies.”

IV. MCG Gas Pipeline Rehabilitation: Project Summary and ERR Model Revised

“Before the rehabilitation work was undertaken, the ERR was estimated to be 11.7% based on a forecast of a reduction in the amount of gas leakage from the pipeline and a assumed corresponding sale of carbon credits due to the reduction in methane emissions to the atmosphere. However, the high gas leakage rate was reduced before the rehabilitation work was undertaken, and it was not possible to sell carbon credits.”

V. Measuring Results of the El Salvador Water and Sanitation Sub-Activity

“High costs per beneficiary can kill even the best projects. The economic rate of return for the El Salvador Water and Sanitation Sub-Activity was estimated as 3.6% at the close of the Compact, due largely to low population densities and high costs. While the execution of water, sanitation, and hygiene projects can be improved using the above lessons, it must be kept in mind that if we target expensive or low density regions, the benefits are unlikely to justify the project.”

“‘Higher than expected costs per borehole reduced the number built during the compact, while low population densities reduced the number of households using each borehole. Combined, these led to a measurable drop in beneficiaries.”

Bibliography

Cronshaw, Mark, and Gustavson Associates. MCG Gas Pipeline Rehabilitation: Project Summary and ERR Model Revised. Rep. 23 June 2010. Web.

Millennium Challenge Corporation “Measuring Results of the Morocco Enterprise Support Project”, MCC, Summary of Findings, 2015

Millennium Challenge Corporation “Measuring Results of the Artisan and Fez Medina Project Artisan Production and Promotion and Fez Medina Activities”, MCC, Summary of Findings, 2015

Millennium Challenge Corporation “Measuring Results of the Nicaragua Transportation Project”, MCC, Summary of Findings, 2014.

Millennium Challenge Corporation “Measuring Results of the El Salvador Water and Sanitation Sub-Activity”, MCC, Summary of Findings, 2017.

Millennium Challenge Corporation. “Economic Rates of Return: Country Spreadsheets.” Millennium Challenge Corporation. Web.

Ospina, Sandra, Aaron Mitchell, and Aaron Szott. 2014 Report on Closeout ERRs. March 2015.

Ospina, Sandra and Marissa Block. 2015 Report on Closeout ERRs. June 2016.

Footnotes
  • 1. Jordan’s compact ended by end-2016 but closeout ERRs are due July 2017.
  • 2. Table 6 also tabulates stated reasons for the lack of original ERRs. When projects lack both the original ERR and the closeout ERR, the stated reasons usually coincide.
  • 3. A caveat on the stated reasons in Table 6 is that they are survey-based. As such, they may not represent true underlying causes.
  • 4. Note that these differences across countries, years, and sectors in Table 2 should not be interpreted as statistically significant or indicative of meaningful comparative differences in performance across countries due to the very small sample size.
  • 5. The reasons for four projects are taken from the “Summary of Findings” of their corresponding evaluation reports or from the reports themselves. Annex III provides excerpts of the available “Summary of Findings” for projects with evaluations.
  • 6. See Annex II for further explanation as to why an ERR was not originally calculated.
  • 7. Given the nature of post-creation business support, impacts on value-added growth and business survival rates are difficult to detect/measure after a single year. However, costs were greater than originally expected, which was the primary reason the decision was made not to scale up. While the results of the impact evaluation as summarized in this report further supports MCC’s final decision not to scale-up (as well as the fact that costs at closeout were even higher than estimated at the time of the scale up decision), previous interventions in post-creation business support would suggest that some benefits from support may accrue several years after implementation, longer than the total of two years given to the pilot program to be evaluated. Unfortunately, due to concerns about attrition rates (e.g. difficulty in following up with businesses) and possible provision of support services to control group businesses, further impact evaluation of this Project is not feasible.