Measuring Results of the Georgia Regional Development Fund

The first MCC compact with Georgia was a five-year investment (2006-2011) of $387.2 million. The $32 million Georgia Regional Development Fund (GRDF) component is the subject of an independent performance evaluation summarized here.

  • The Georgia Regional Development Fund was designed to catalyze SME development in the regions outside of Tbilisi, primarily in the agriculture and tourism sectors, thereby boosting regional economic growth and employment.
  • Of the fourteen GRDF investees, four were successful in boosting economic growth and employment in the regions outside of Tbilisi, while another four eventually became insolvent. The remaining six showed mixed performance. Those that were successful were so in large part due to the fact that they were able to attract outside sources of private financing.
  • GRDF was a new twist on traditional investment funds in two ways: it targeted small and primarily rural companies in a country with little experience in investment funds, and it put greater emphasis on “developmental return” than financial return. As a result, the fund took on a riskier investment portfolio that was ultimately not financially sustainable.
  • This evaluation is complete and there are no planned next steps.

In Context

The MCC compact with Georgia was a five-year investment (2006-2011) of $387.2 million in two projects:  Enterprise Development and Regional Infrastructure Rehabilitation. The Enterprise Development Project included two major activities, Georgia Regional Development Fund (GRDF) and Agribusiness Development Activity (ADA). The $32 million GRDF component is the subject of an independent performance evaluation released by MCC in May 2017, the results of which are summarized here.  This component represents 8 percent of the total compact. Other components of the compact are the subject of previously released independent evaluations.

Pie Chart: Georgia Compact Spending

*These figures are based on MCC obligations as of April 2017.

 

Program Logic

The Georgia Regional Development Fund was designed to catalyze Small and Medium Enterprises (SME) development in the regions outside of Tbilisi, primarily in the agriculture and tourism sectors, thereby boosting regional economic growth and employment.

Program Logic Flow Chart

A key assumption underlying the GRDF program logic during the design of the investment was that by demonstrating successful investments in SMEs outside of Tbilisi, there would be an increase in additional private investment, thereby further promoting regional development.

 

For a more detailed version of the program logic, please refer to Section 1.2 of the Final Evaluation Report.

 

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources.  Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program.  However, monitoring data is limited in that it cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention.  The limitations of monitoring data are a key reason why MCC invests in independent impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone.  Where estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

 

Indicators Level Baseline (2006) Actual Achieved (April 2011) Target Percent Complete
Increase in gross revenues of portfolio companies Outcome 0 16,880,669 22,200,000 76%
Increase in portfolio company employees Outcome 0 208 1,892 11%
Increase in local suppliers to the portfolio companies Outcome 0 514 2,508 20%
Increase in wages paid to the portfolio company employees Outcome 0 1,787,378 3,118,000 57%
Increase in locally sourced goods and services purchased by the portfolio companies Outcome 0 3,943,260 6,332,000 62%
Debt investments into portfolio companies Output 0 24,308,251 15,750,000 154%
Equity investments into portfolio companies Output 0 2,571,752 6,250,000 41%
Portfolio companies receiving investment Output 0 12 20 60%
Businesses receiving technical assistance Output 0 10 27 37%

Source: Closeout ITT from June 2011, which includes data through the end of the compact.

 

The average completion rate of output and outcome targets is 72 percent; and in 4 of the 16 monitoring indicators, targets were met or exceeded. 1

 

Evaluation Questions

The evaluation was designed to answer the following questions:

  • Did GRDF meet its stated objectives? Were GRDF’s stated objectives clear and actionable? Was the concept of “development impacts/returns” implementable?
  • What factors explain the success of the relatively more successful/profitable firms (e.g. internal competencies, industry/market factors, GRDF technical/financial support, etc.)?
  • What barriers/challenges explain any underperformance noted in GRDF portfolio firms (e.g. internal problems, changes in market forces, government interventions/changes, weak entrepreneurial skills, weak accounting practices, etc.)?
  • What were some indirect effects of GRDF investments? For example, did GRDF investments allow the beneficiaries to more easily access other forms of financing? Was GRDF debt leveraged into more senior debt? Has GRDF created any positive externalities in the Georgian economy?
  • How were technical assistance funds employed by the fund manager, Small Enterprise Assistance Funds (SEAF)? Did these funds allow for efficiency/profitability/other gains in portfolio SMEs’ operations?
  • To what extent has the GRDF investment been essential for the SMEs’ development, and for their access to finance?
  • Did GRDF provide financing that wouldn’t have been accessible otherwise? Did GRDF provide better terms to portfolio firms (e.g. rates, collateral requirements, etc.) than they would have been able to acquire elsewhere?

 

Evaluation Results

The GRDF Evaluation built on the standard due diligence approach for private equity funds and analyzed the GRDF’s interventions around three dimensions: Project Design, Institutional Framework, and Outcomes. The evaluation involved a document review, in-depth financial analysis, stakeholder interviews, and in-depth case studies. The companies selected for the case studies represented a cross-section of the portfolio in terms of financial and development performance.

 

 
Evaluator             A2F Consulting
Impact or Performance? Performance
Methodology    Private Equity Fund Due Diligence Approach
Evaluation Period 2016, aligned with the closing of the Fund 2 , ten years after the creation of GRDF and five years after the end of the Compact.
Outcomes –          Financial performance of GRDF portfolio companies was poor with an overall net Internal Rate of Return (IRR) estimated to be approximately -14.22%. Twelve out of the 14 GRDF investees ran into financial difficulties and encountered trouble in servicing their payment obligations.
Objective-level Outcomes –         Performance with regards to development returns 3  was also rather mixed, but largely mirrored financial performance. Seven out of the 14 companies had positive returns, while seven were negative or zero.

–          Of the fourteen GRDF investees, four were successful in boosting economic growth and employment in the regions outside of Tbilisi, while another four eventually became insolvent. The remaining six showed mixed performance.  Those that were successful were so in large part due to the fact that they were able to attract outside sources of private financing.

Effect on household income attributable to MCC N/A

 

Lessons Learned

GRDF was a new twist on traditional investment funds in two ways:  it targeted small and primarily rural companies in a country with little experience in investment funds, and it put greater emphasis on “developmental return” than financial return. Focusing on small and rural companies required long leads times in initiating and exiting investments, thereby contributing to relatively high management costs.  The emphasis on developmental returns, at least as implemented in GRDF, proved to be cumbersome and complicated communications regarding outcomes. Both of these aspects of the fund contributed to perhaps overly-restrictive investment criteria and increased the risk of the investments.

 

The measurement and attribution of non-financial outcomes of an investment fund proved to be difficult.  The “developmental return” had several shortcomings, particularly because it was based on annual percentage changes of four measures (i.e., wages paid, local purchases, taxes paid, and investee revenues), which inherently favored recently created and unproven SMEs with the most room to grow. Thus, using the development return as a basis for determining a portion of the fund manager’s compensation proved to be problematic, as it incentivized short-term performance over long-term value creation and ultimately led the fund manager to take riskier investments that were not necessarily financially sustainable.

 

Limitations imposed on the length of MCC compacts will continue to be an obstacle to supporting investment funds in the future. The five-year time limit on MCC Compacts incentivizes short term gains, rather than long-term value creation. GRDF was a unique case in that it is MCC’s only experience with a stand-alone Compact activity that remained on-going after the close of a Compact. However, even with the life of GRDF extending beyond the compact period to ten years, the GRDF encountered difficulties in liquidating its portfolio in a timely manner. Additionally, the inability to intervene in the post-Compact period left MCC exposed to reputational and perhaps other risks.

                                                                                                                                                                     

Potential synergies between Compact components could have been more adequately exploited in order to increase the probability of success. Significant lead time is required to prepare small and medium-sized companies for eligibility for investment; therefore, expanding the role and size of the Technical Assistance Facility (a $2 million component of GRDF) may have helped to improve business viability and build SME capacity prior to and during investment. Additionally, investments in the priority sectors of agribusiness and tourism were among the worst performing.  Had the Agribusiness Development Activity been given more significant lead time to first address the primary agriculture market, GRDF investments in the secondary agriculture market may have been more successful.

 

Next Steps

This evaluation is complete and there are no planned next steps.

Footnotes
  • 1. These figures are calculated using all non-evaluation indicators with targets in the Georgia Regional Development Fund Activity.
  • 2. GRDF was initially meant to close in April 2016. However, due to more complicated than anticipated exits, GRDF was still in the process of winding down as of April 2017, the date of this report.
  • 3. Development return was calculated as the weighted average of annual changes in the wages paid, revenues generated, taxes paid, and local purchases of supply goods by GRDF investees.