Measuring Results of Namibia Livestock Support Activity

The MCC compact with Namibia was a five-year investment (2009-2014) of $304.5 million. The $20.6 million Livestock Support activity is the subject of an independent performance evaluation summarized here.

  • The Livestock Support Activity provided grants, constructed infrastructure, tagged cattle for tracking, and established a livestock traceability system. Collectively, these outputs were expected to improve livestock health and productivity, and increase international market access and formal livestock sales, while ultimately increasing household income and reducing rural poverty.
  • Five state veterinary offices were constructed; two quarantine camps were rehabilitated; and nearly 1.8 million cattle were tagged with RFID tags.
  • The evaluation summarized results against the following five criteria for each component of the Activity: relevance, effectiveness, efficiency, impact, and sustainability; most components were found to be relevant, efficient, and effective, while the impact of most was expected to be limited, and expected sustainability varied.
  • This evaluation is complete and there are no planned next steps.

In Context

The MCC compact with Namibia was a five-year investment (2009-2014) of $304.5 million in three projects:  the Education Project, the Tourism Project, and the Agriculture Project. The Agriculture Project included three major activities, the Land Access and Management activity, the Livestock Support activity, and the Indigenous Natural Products Development activity. The Livestock Support activity consisted of three components:  the construction of five veterinary centers and two quarantine camps (Q-camps), the development of a livestock traceability system in the Northern Communal Areas, and the establishment of a Livestock Market Efficiency Fund (LMEF). The LMEF financed seven grants that were designed to improve various aspects of the livestock sector. The $20.6 million Livestock Support activity is the subject of an independent performance evaluation released by MCC in 6/13/2017, the results of which are summarized here. This activity represents 6.8 percent of the total compact. Other components of the compact are the subject of independent evaluations still underway when this summary was published.

Pie Chart: Namibia Compact Spending

These figures are based on MCC obligations as of August 2015.

Program Logic

The Livestock Support activity produced the following outputs: state veterinary offices (SVOs) constructed, Q-camps rehabilitated, cattle tagged and tracked, LMEF grant funds disbursed, and the establishment of a livestock traceability system. These outputs are expected to lead to improved livestock health and productivity, and increased international market access and formal livestock sales. In the long term, the expected overall impact is an increase in household income and reduction in rural poverty.

Flow Chart: Program Logic

There were several key assumptions underlying the Livestock Support program logic during the design of the investment:

  • Farmers in the NCAs will respond to efforts to increase marketing and sales of livestock.
  • Construction of SVO in underserved areas will – assuming the Government of the Republic of Namibia maintains, staffs, and equips them appropriately – ultimately improve livestock, health, and indirectly lead to increased household wealth and incomes.
  • Tagging cattle, as part of a broader traceability system, will facilitate management of disease outbreaks and improve marketing opportunities for farmers in the NCAs.
  • Traceability, as implemented by the Government of the Republic of Namibia, will be recognized by the World Organization for Animal Health as an important step towards achieving disease-free status in the NCAs, which is expected to open international markets for Namibian cattle.
  • A constraint to increased livestock sales is a lack of marketing opportunities, and this constraint can be addressed through specific interventions that increase the efficiency of the marketing process.

 

For a more detailed version of the program logic, please refer to page 42-44 of the Livestock Final Evaluation Report, which can be found here: https://data.mcc.gov/evaluations/index.php/catalog/190 .

 

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources.  Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program.  However, monitoring data is limited in that it cannot reflect the full range of targeted outcomes and cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention.  The limitations of monitoring data is a key reason why MCC invests in independent evaluations to assess the achievement of a broader set of program outcomes.  When feasible, MCC supports impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone.  When estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance in terms of output and outcome indicators specific to the evaluated program.

Indicators Level Baseline (2009) Actual Achieved (09/2014) Target Percent Complete
New state veterinary offices (SVOs) operational Outcome 0 5 5 100%
New state veterinary offices (SVOs) constructed Output 0 5 5 100%
Q-Camps rehabilitated Output 0 2 2 100%
Cattle tagged with RFID tags Output 0 1,770,313 1,000,000 177%
Value of grant agreements signed under the Livestock Efficiency Fund* Process 0 5,583,514 6,102,000 92%

* There are no Output or Outcome indicators associated with the Livestock Market Efficiency Fund sub-activity; instead, a process indicator was included. This indicator is not included in the average completion rate or number of targets met calculations below.

Source: (December 2014 ITT, based on reporting from MCA-Namibia and Namibia’s Ministry of Agriculture, Water and Forestry)

 

The average completion rate of output targets is 126 percent and targets were met or exceeded in all three of the output indicators.  The average completion rate of outcome targets is 100 percent and targets were met or exceeded in the only outcome indicator.  1

Evaluation Questions

The original evaluation questions included the following:

  • To what extent does the LMEF contribute to increased incomes as well as other measures of livelihood among beneficiaries?
  • Does the LMEF contribute to disease free status for the NCAs? If so, how and to what extent?
  • Does the LMEF contribute towards the identification and elimination of existing marketing barriers and other challenges to successful commercial marketing of livestock in the NCAs? If so, what is the impact (e.g., increased volume of livestock and livestock products sold into existing markets in the NCAs and in existing and new market destinations)?
  • Has the application of the LMEF led to any multiplier effects in terms of replication of grantee projects, extension of project outcomes, and dissemination of information?
  • To what extent has the Fund as a whole achieved its stated objectives?
  • To what extent, if any, has the Namibia Compact achieved stated objectives related to the quarantine camp, SVO, and livestock traceability investments?
  • What is the impact of the Livestock Support activity overall on: cattle health; attainment of FMD-free status; cattle marketability; international market access; livestock sales?

 

The evaluation did not cover the following benefit streams that were modeled in the economic analysis of the program:

  • Average price
  • Total production
  • Income from cattle

 

As described further below, the evaluation did not conduct primary data collection or analysis that would have allowed for measuring the above benefit streams.  However, they will be measured in other evaluations of the Agriculture Project.

Evaluation Results

The Livestock Support activity evaluated each of the sub-activities and seven grant projects funded by the LMEF Grant using a methodology called Real Time Evaluation, which is an iterative process of regular evaluation missions, following, assessing, and documenting implementation progress based on document review and triangulation with field observations and interviews. Rather than focusing directly on the evaluation questions outlined originally, the evaluation used the Development Assistance Criteria (DAC) defined by the Organization for Economic Cooperation and Development (OECD), which assesses the success of a program using five criteria: relevance, effectiveness, efficiency, impact, and sustainability.  2

 

Relevance is assessed by evaluating the extent to which an action is suited to the priorities and policies of the donor agency, the appropriateness of the action to the problems, needs and priorities of an action’s target groups or beneficiaries, and the quality of the design through which the objectives are to be reached. Most of the actions were found to be relevant, although the rehabilitation of the two Q-camps was found to be less relevant since this sub-activity was considered a regular maintenance task of the Namibian government, which would have been completed without MCA-N intervention.

 

Efficiency measures the qualitative and quantitative outputs of each action, in relation to the inputs. The majority of the actions were also found to be efficient. Several of the marketing-related LMEF Grants were considered the least efficient. The evaluator considered the construction of five veterinary centers and two quarantine camps, and the development of a livestock traceability system in the Northern Communal Areas to be among the most efficient actions.

 

Effectiveness is defined as the contribution made by an action’s results and outcomes to the achievement of its purpose. The majority of actions were found to be sufficiently effective, with the construction of five veterinary centers and two Q-camps and the development of a livestock traceability system in the Northern Communal Areas found to be among the most effective.

 

Impact 3  is defined as the change or effect an action has on its wider environment. Effects can be intended or unintended, as well as positive and negative. The evaluator looked at the impact of actions on a variety of factors including, social, economic, environmental, and other development indicators. The impact, as identified by the evaluator, varies greatly according to the action. Since many of the actions targeted only a small number of participants, the evaluator reports that any potential impact will be limited. Additionally, the evaluator notes that the impact of studies is contingent on the recommendations of the studies being adopted.

 

Sustainability assesses the likelihood of a continuation of benefits produced by an action after the period of external support has ended. Sustainability also varied greatly between the different actions and depended on the type of action, how well it was designed, and the extent of uptake. The construction of five veterinary centers and the development of a livestock traceability system in the NCAs are expected to be among the most sustainable interventions, whereas the rehabilitation of the two Q-camps are expected to be among the least sustainable interventions.

 

 
Evaluator             Paul Sijssens (Individual Consultant to MCA-Namibia)
Impact or Performance? Performance Evaluation
Methodology    Real-time Evaluation
Evaluation Period March 2011 – June 2014
Outcomes Monitoring data indicate that five new State Veterinary Offices were established and are operational. The evaluation did not measure intermediate outcomes such as improved livestock health and increased international market access and formal market sales due to the timeframe and methodology used.
Objective-level Outcomes Assessing the achievement of the objective-level outcomes of the Livestock Support activity, such as increased value-added from livestock, was not possible due to the timeframe and methodology of the evaluation.
Effect on household income attributable to MCC Assessing the achievement of the ultimate impact of the Livestock Support activity of increased household income in the NCAs and reduction in rural poverty, was not possible due to the timeframe and methodology of the evaluation.


Lessons Learned

There were several key lessons learned from the Livestock Support activity and its evaluation for MCC and future partner countries to consider when designing and implementing agriculture projects and evaluations:

 

  1. Broad consultation and sustainability mechanisms need to be incorporated into project design from the beginning. Because insufficient consultation and sustainability planning occurred early on, the Livestock Support activity experienced varying degrees of public/private sector and local community buy-in, which has implications for sustainability. MCC and MCAs need to ensure that project design incorporates extensive stakeholder consultations and post-compact sustainability plans, including written commitments from government partners, before projects begin. For interventions that are expected to benefit the private sector, MCAs and MCC might also seek private sector co-financing to increase their vested interest in sustainability.
  2. MCC-funded evaluations need to follow MCC’s evaluation standards. This lesson applies equally to evaluations managed by MCAs like this Livestock Support evaluation, and those managed by MCC. MCC’s evaluation standards were developed after this evaluation started and could have helped to improve its overall quality and relevance. For example, MCC now requires that the logic of an intervention be clearly documented before an evaluator is hired. The program logic should then guide the evaluation, including the questions to be answered. Similarly, each evaluation should be designed to answer a comprehensive set of questions. Unfortunately, the methodology used here shifted the focus from the initial evaluation questions, to an assessment of whether each component of the Livestock Support activity met the DAC evaluation criteria. As a result, it is difficult to understand what was accomplished in terms of targeted outcomes. MCC’s evaluation standards also require that a multidisciplinary Evaluation Management Committee review and clear on key evaluation products and decisions. This process is aimed at ensuring MCC-funded evaluations are worth their costs by preventing the kinds of issues encountered here, or resolving them early if and when they do arise.

Next Steps

This evaluation is complete and there are no planned next steps.

Footnotes
  • 1. These figures are calculated using all non-process indicators (except the one flagged above) with targets in the Livestock Support Activity.
  • 2. Please see the DAC Criteria for Evaluating Development Assistance page on the OECD website for more information on the five criteria. http://www.oecd.org/development/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm
  • 3. Please note that the terminology “impact” is applied more broadly in the Livestock Support activity evaluation report than in the program logic section above.  “Ultimate impact” in the program logic refers specifically to only the highest level of outcome.