Improving Farmers’ Skills and Productivity in Honduras

Program Overview

MCC’s $204 million Honduras Compact (2005-2010) funded the $68.3 million Rural Development Project which included the $26.6 million Farmer Training and Development Activity. The activity aimed to improve the skills, productivity, and risk management practices of farmers to cultivate horticultural crops – such as fruits and vegetables – on small and medium farms. By providing over 7,000 farmers with training and technical assistance, which included financial support, the activity focused on producing and marketing these crops to increase household incomes.

Key Findings

The original randomized evaluation design had to be abandoned due to implementation challenges. The MCC Management Response for this evaluation summarizes MCC’s concerns that despite efforts by the evaluation team to adapt the evaluation, the econometric model does not overcome the issue of selection bias. Given these concerns, MCC limits the conclusions that can be drawn from the evaluation report regarding the results of the farmer training activity.

Evaluation Questions

This final performance evaluation was designed to answer if training and technical assistance to farmers:

1. 1
   Increased cultivation of horticultural crops?
2. 2
   Increased employment on farms?
3. 3
   Increased household income?

Detailed Findings

The original randomized evaluation design had to be abandoned due to implementation challenges. The MCC Management Response for this evaluation summarizes MCC’s concerns that despite efforts by the evaluation team to adapt the evaluation, the econometric model does not overcome the issue of selection bias. Given these concerns, MCC limits the conclusions that can be drawn from the evaluation report regarding the results of the farmer training activity.

Training and Crop Mix

To participate in the training program, farmers needed to meet eight defined criteria, including legal access to land and demonstrated ability to produce agricultural products. Farmers originally needed legal access to at least one hectare of land available to transition from basic grains such as corn and beans to high-value agriculture like vegetables to achieve a minimum net income of $2,000 per year per hectare. After the first year of implementation, the land plot size criterion was revised to a minimum of .20 hectares. However, as discussed below, the evaluator was unable to replicate the implementers’ process for location and farmer selection based on the criteria.

Productivity and Income

For participating farmers, there was an increase in input expenditures for horticultural crops compared to basic crops, implying a higher level of activity in high-value crop cultivation. There was a corresponding decline among participating farmers in income from basic crops, as might be expected with changing crop mix; however, this decline was not statistically significant. The incremental increase of horticulture crop incomes for participating farmers was likely driven by increased production among farmers already growing horticultural crops and not from farmers who switched over for the first time due to the training. There was no increase in net household income, household expenditures, or consumption.

MCC Learning

• 
  Integrate implementers and evaluators early. Involving the evaluator in earlier stages of program design could have mitigated some of the challenges that emerged regarding farmer selection and feasibility of randomization.
• 
  Clearly define program participants. MCC and country counterparts must work toward clearly defined program participants and eligibility criteria to inform feasibility of rigorous impact evaluation design.
• 
  Align incentives. Implementers and evaluators should work in lock-step to produce a successful evaluation of an effective program. This requires integrating the evaluation with the program early, and aligning incentives to ensure understanding and cooperation between evaluator and program implementer. However, in Honduras, the implementer was contracted two years before the evaluator under a contract with no specific responsibilities to collaborate with the evaluator. That contract included strict commitments to train 6,000 farmers and increase their average annual incomes by $2,000. These requirements established strong incentives for the implementer to exhaustively search for potentially successful program participants. But because this exhaustive selection process could not be replicated by the evaluator, it conflicted with the original evaluation design.
• 
  To better integrate evaluation and implementation activities, and align incentives for future evaluations, MCC instituted the Evaluation Management and Review Process in 2013. Turning these lessons into action was documented in “Learning from Evaluations at the Millennium Challenge Corporation” by Sturdy, Aquino and Molyneaux (2014).

Evaluation Methods

The evaluation design for this activity changed over the course of the evaluation due to problems faced during implementation. In its original conception, the evaluator and MCA-Honduras planned to use randomized assignment of communities to treatment and control groups. But because this did not align with program implementation, the final approach used was an econometric model that relied on a model-based approach. As raised in the evaluator report, one key assumption is that the causal models are correct.

In its Management Response, MCC recognizes the efforts undertaken by the evaluation team to identify a valid counterfactual as a basis for measuring the causal impacts of this activity. The report clearly documents the substantially different characteristics of the treatment and control groups of farmers, making it impossible to use the control group as a counterfactual in the evaluation. The evaluation report makes a compelling case for the use of an alternative model-based approach intended to distinguish program impacts from the effects of strategically selected program participants. While the analysis suggests there was a positive impact on farmers participating in the training, due to the limitations of this approach, the MCC technical team does not believe the model overcomes the issue of selection bias. The evaluation is therefore classified by MCC as a performance evaluation given the lack of a counterfactual.

After two failed baseline data collection efforts, the evaluation analysis relies on the quantitative data collected on 4,526 farmer households in April–July 2010 and 2,736 farmer households in April 2011, giving an exposure period of 7–12 months.

2020-002-2482