Measuring Results of Namibia Conservancy Support Evaluation

The MCC compact with Namibia was a five-year investment (2009-2014) of $304.5 million. The $18.3 million Conservancy Support activity is the subject of an independent performance evaluation summarized here.

  • This activity aimed to build the business opportunities and financial sustainability of 31 conservancies in order to benefit households through increased employment opportunities, distributions of conservancy revenue, and incomes.
  • Over 2,700 game were translocated to supported conservancies and 15 joint venture enterprises were established.
  • Conservancy joint ventures are positively correlated with full-time employment in some model specifications but not others, whereas grants to create small and medium enterprises are not correlated with increased employment.
  • The evaluation is complete and there are no planned next steps.

In Context

The MCC compact with Namibia was a five-year investment (2009-2014) of $304.5 million in three projects:  the Education Project, the Tourism Project, and the Agriculture Project.  The Tourism Project included three major activities, Improved management of and infrastructure for Etosha National Park, Marketing Namibia as a tourist destination, and Ecotourism development in conservancies (“Conservancy Support”). The Conservancy Support activity consisted of three components:  a needs assessment of conservancies, technical assistance and capacity building, and grant funds for joint-venture enterprises. The $18.3 million Conservancy Support activity is the subject of an independent performance evaluation released by MCC in 6/13/2017, the results of which are summarized here. This component represents 6 percent of the total Namibia compact. Other components of the compact are the subject of independent evaluations still underway when this summary was published.

These figures are based on MCC obligations as of August 2015.

Program Logic

The Conservancy Support activity within the Tourism Project was designed to build the capacity of 31 conservancies with high-tourism potential in order to protect their natural resources, attract investment, increase their business focus, achieve financial sustainability, and ultimately benefit households through increased employment opportunities, larger distributions of conservancy revenues, and increased incomes. In addition, the Compact provided grant funding to promote joint-venture tourism enterprises between conservancies and the private sector in 15 of the 31 conservancies.

Program Logic: Conservancy Support

There were several key assumptions underlying the Conservancy Support activity program logic during the design of the investment:

  • There is sufficient private sector interest in joint ventures with conservancies to respond to the joint venture funds and generate new businesses.
  • Wildlife translocation to conservancies will improve their viability as tourist destinations.
  • Conflicts emanating from increased wildlife in conservancies can be mitigated by preventive measures and will not discourage conservancy members from supporting tourism as a livelihood option.
  • Tourists will continue to seek out lodges and campsites that are slightly off the beaten track and carry a “community‐friendly” label.
  • Conservancies are viable models to manage or benefit from tourism enterprises and financial benefits can be distributed to members in an equitable fashion.
  • Training and technical assistance to conservancy committees and managers result in sustainable improvements in governance, tourism management and business skills development for the wider conservancy; conservancies will have a transition plan in the event of a change in conservancy leadership.
  • Women members of conservancies will be included in conservancy management.
  • Increases in demand for tourism products on conservancies can be managed in an environmentally sustainable manner.
  • Improved management and private sector investment within conservancies will lead to increased revenue to conservancies, and will translate into welfare improvements of conservancy member households.

For a more detailed version of the program logic, please refer to page 9 of Annex 4 of the Namibia M&E Plan, which can be found here: https://assets.mcc.gov/documents/ME_Plan_-_NAM_-_V7_-_Jul14.pdf.

 

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources.  Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program.  However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention.  The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone.  When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance in terms of output and outcome indicators specific to the evaluated program.

 

Indicators Level Baseline (2009) Actual Achieved (09/2014) Target Percent Complete
Game translocated to conservancies with MCA-N support (#) Output 0 2,758 2,135 129%
Rare game translocated to conservancies with MCA-N support (#) Output 0 732 651 112%
Amount of new private sector investment secured by MCA-N assisted conservancies(a) (Namibian $) Outcome 0 136,371,696 40,000,000 341%
New joint venture lodges or tented camps (#) Outcome 0 15 12 125%
New small tourism and natural resources enterprises within conservancies (#) Outcome 0 13(b) 10 130%
New jobs in tourism created in conservancies (#) Outcome 0 224 120 (c)
Annual general meetings (AGMs) with financial reports submitted & benefit distribution plans discussed (#) Outcome 0 26 25 104%
MCA-N grants aimed at mitigating human wildlife conflicts (#) Outcome 0 9 8 113%
Annual gross revenue to conservancies receiving MCA assistance (Namibian $) Outcome 30,833,276 51,759,310 33,608,271 754%
Share of conservancy revenue paid out in dividends and/or spent on community services (%) Outcome 21 32 24 (d)

Source: (e.g. September 2014 ITT, based on reporting from the Namibia Ministry of Environment and Tourism, Namibian Association of Community-Based Natural Resource Management Support Organizations, Conservancy Support implementer, and the Conservancy Support/Indigenous Natural Product household survey)

(a) Includes current and committed investment; target exceeded due to large number of JV grants provided and higher-than-expected commitments from private sector partners.

(b) Number incorrectly recorded as “16” in final ITT; however, the source document indicates the value reported here.

(c) Despite the indicator definition, casual and part-time jobs have not been included; and the indicator reflects a cumulative addition during implementation, rather than a level or annual increase as noted in the Namibia M&E Plan. Furthermore, the indicator does not reflect the net number of jobs created as approximately 50 jobs were lost during implementation across a number of conservancies, and this is not reflected in the reported figure. As a result, the Percent Complete would be misleading and has been omitted.

(d) The baseline figure included in the M&E Plan, and on which the target was based, conflicts with that reported in the implementer’s final report.  Compared with the implementer’s baseline of 35%, the Year 5 figure actually suggests a slight decrease during the compact period.  Using the baseline and target figures from the M&E Plan to report a % complete yields a 367% increase, which is misleading.  Therefore, this figure has been omitted from the table above and the average completion rate reported below.  Assuming the 35% baseline is correct, given the substantial increase in conservancy revenue during the compact period, a larger overall amount of benefits went to members in Year 5 despite the slightly smaller share of distribution.

 

The average completion rate of output targets is 121 percent and targets were met or exceeded in 2 of the 2 output indicators.  The average completion rate of outcome targets is 261 percent and targets were met or exceeded in 6 of the 6 outcome indicators with targets above. 1

 

Evaluation Questions

The evaluation was designed to answer questions such as:

  • Do technical support and grants to conservancies increase business partnerships between conservancies and private businesses, and, in turn, increase conservancy revenue?
  • Does technical support to conservancies improve conservancy governance? Does improved governance impact the equitable distribution of conservancy benefits?
  • Is there an increase in conservancy-related employment as a result of the CS activities (including grants)? If so, how many new jobs are created and at what levels of employment?
  • Do the Conservancy Support activities (including grants) lead to an increase in household wellbeing over the life of its program?
  • What have been the effects of game acquisitions on the conservancies and on their members?
  • How sustainable are the results of business partnerships in terms of increased employment and improved mechanisms for distribution of revenue?
  • What impact does MCA-N support have on conservancy members in terms of the distribution of benefits by gender?
  • What is the perceived impact on recipient-household gender relationships from the intervention?

 

MCC’s evaluation standards require greater alignment between evaluation metrics and economic analysis than was required when the Conservancy Support evaluation was designed. As a result, this evaluation did not directly measure the following benefit streams that were modeled in the economic analysis of the program:

  • Value of increase in game
  • Revenue to conservancies
  • Total benefits from joint ventures

However, it did look at the relationship of these benefits to other outcomes, such as the distribution of conservancy benefits to members, employment from joint ventures, and the relationship between partnerships and conservancy revenues.

 

Evaluation Results

This evaluation relied on a model-based approach complemented by qualitative data. The credibility of these findings hinge on acceptable assumptions underlying the model and consideration of all reasonable confounders. The results are summarized below.

 
Evaluator             NORC at the University of Chicago
Impact or Performance? Performance
Methodology    Continuous Treatment, Pre-Post, Other
Evaluation Period Baseline household survey was conducted in 2011; endline household survey was conducted in 2014
Outcomes ·         Qualitative results indicate joint venture grants for lodges and game translocations were important for business partnerships, and that partnerships have increased revenue to conservancies. Some of the other grants and technical assistance were considered less critical for increasing business partnerships, while transaction costs were high.

·         Conservancy managers and implementers report that governance and equality of benefit distribution has improved; members’ perceptions of benefit distribution are less favorable; quantitative analysis shows that technical assistance is generally associated with more equitable benefits distribution.

·         Qualitative methods indicate progress in gender relationships but many problems of gender inequity persist.

Objective-level Outcomes ·         Conservancy joint ventures are positively correlated with full-time employment in some model specifications but not others

·         Grants to the small and medium enterprises created under this program are not correlated with increases in full-time or part-time employment

·         Capacity to sustain new business partnerships is noted as a concern in the absence of ongoing support to conservancies

Effect on household income attributable to MCC ·         N/A


Lessons Learned

  • Plan for sustainable business partnerships. This activity did not focus on the long-term business and management skills conservancies would need to assume control of newly-established tourism operations from their private sector partners.  A further challenge is the regular turnover of conservancy managers.  It is possible a transfer of skills from private sector partners to conservancy managers will happen naturally; however, the intervention could have also tried separating the roles of conservancy managers from enterprise managers to help with continuity or increasing the emphasis on business skill development.
  • Carefully consider data availability and implications for analysis and contracting mechanisms. The evaluator notes that research questions and methods had to be adapted to fit limitations in data availability.  In the future, evaluations ought to plan more carefully and transparently for what they will measure, how, and when, and either be contracted by MCC so compact expiration is not a constraint, or continued into the post-compact period when necessary to get a complete picture of outcomes. The latter option should also carefully consider the costs and benefits of splitting evaluation and contract management between an MCA during compact implementation and MCC in the post-Compact period.

 

Next Steps

This evaluation is complete and there are no planned next steps.

Footnotes
  • 1. These figures are calculated using all non-evaluation indicators with targets in the Conservancy Support Activity. Targets for a number of indicators were set very conservatively.