Targeting Stunting in Indonesia via Nutrition & Hygiene

Program Overview

MCC’s $474 million Indonesia Compact (2013-2018) funded the $120 million Community-Based Health and Nutrition to Reduce Stunting Project, which aimed to avert stunting and malnutrition of children in 11 provinces. It trained health workers on feeding, growth monitoring, and sanitation; conducted community sanitation events; provided iron-folic acid to pregnant women; conducted a communications campaign on stunting; and provided grants to incentivize the use of health services through the Generasi program.

Evaluator Description

MCC commissioned an independent interim impact evaluation of the project by Mathematica Policy Research. The interim study focuses on intermediate outcomes only. Full report results and learning: https://data.mcc.gov/evaluations/index.php/catalog/109

Key Findings

Implementation Quality

Training levels were significantly higher in project areas, but similar trainings were occurring in comparison areas.
Child nutrition training was implemented as intended and with high quality. Sanitation training was implemented, but not as comprehensively as intended.

Short-Term Outcomes

Modest improvements in health provider knowledge, but not uniformly. Impacts varied by question and provider.
No impacts on frequency of nutritional group counseling, which was of mediocre quality, nor on share of women receiving one-on-one health services.
Village meetings to initiate sanitation behavior change were more frequent in project areas, but omitted key steps.
Villages received Generasi grants as intended and most funding went toward health-related activities.

Medium-Term Outcomes

No improvement in village-level open defecation free status.
Interim findings overall indicate that long-term outcomes may not be achieved.

Evaluation Questions

This interim impact evaluation was designed to answer a small set of questions focused on results expected by the close of project implementation. It aimed to highlight implementation accomplishments and shortcomings and assess progress toward achieving targeted longer-term outcomes, by answering:

1 What was the content and quality of Project implementation? What was the quantity and quality of nutrition and sanitation training, including whether the training occurred, topics discussed, and how they were discussed? How did villages spend Generasi grants?
2 What were the impacts of training on health service provider knowledge?
3 How did trainees apply lessons from the training in a professional setting?
4 Did villages improve sanitation practices as a result of sanitarian training?

Detailed Findings

Nutrition—Training, Group Counseling, & One-on-One Services

Percentage of Service Providers Who Know That Stunting Affects Both Mental and Physical Development

Health service providers in project areas were significantly more likely to receive child nutrition training than those in comparison areas, confirming that the training was implemented. The quality of training content and facilitation was high: trainers provided detailed information on relevant practices, followed the training manual closely, and used interactive methods to keep participants engaged. Knowledge of child nutrition topics improved modestly among some service providers. For example, 94 percent of sub-village-level child health volunteers in project areas knew when to initiate solids compared to 84 percent of volunteers in comparison areas. Further, nutritionists in sub-district health clinics in project areas were more likely than comparators to know about the effects of stunting, but as strong an impact was not found across all trained providers.

Percentage of Service Providers Who Reported Holding a Counseling Session for Mothers of Young Children in the Last Month

The project had only modest impacts on the frequency of nutritional counseling. Village midwives in project areas were slightly more likely to report having held a group counseling session for pregnant women in the last month compared to village midwives outside of project areas. However, there was no significant difference in classes being held by sub-village child health post volunteers. Moreover, counseling session quality demonstrated much room for improvement. They did not comprehensively cover key topics like breastfeeding techniques, and the classroom environment was often chaotic and distracting for participants due to the presence of young children.

One-on-one health service provision was uniform across project and comparison areas, indicating no project impacts on services, such as number of women seen for prenatal care or availability of child height measurement devices at sub-village-level health posts. The limited impacts on nutrition-related counseling and service provision suggest that the project is unlikely to change community behavior surrounding nutrition in the long-term.

Sanitation—Training

Significantly more sanitarians were trained in project areas than comparison areas, confirming that the sanitation training was implemented. Sanitarian knowledge improved regarding some topics. For example, sanitarians in project areas were 14 percentage points more likely than those in comparison areas to know the effects of stunting.

However, the delivery of training content fell short of expectations. Training covered key sanitation-related topics like eliminating open defecation, handwashing with soap, and solid/liquid waste management, but failed to cover many key steps in the process of helping communities become open defecation free (ODF). Community ODF status is a critical link for improving child nutritional status.

Number of Sub-Villages/Villages Reported as ODF by their Assigned Sanitarian

Sanitation—Community Behavior Change

The project was successful at generating interest and participation in village-level sanitation behavior change meetings: 87 percent of villages in project areas held these meetings compared to 74 percent in comparison areas. However, these meetings omitted key steps hypothesized necessary for villages to become ODF. These shortcomings likely account for the lack of project impact on the number of villages that achieved ODF status. There was no statistically significant difference in the number of villages reported to be ODF by their assigned sanitarians between project and comparison areas. In addition to deficiencies in village meetings, another main impediment to villages becoming ODF was a lack of resources available for households who needed additional support constructing a latrine.

Generasi Community Grants

Use of Generasi Community Block Grants

Generasi was the main demand-side intervention funded by the project. On average, villages received about US$5,400 in Generasi community grants per year between 2012 and 2017. About two-thirds of Generasi funds went toward funding health-related activities, which reflects the fact that 10 of the 12 Generasi monitoring indicators tied to village-level funding were related to the health of pregnant women, infants, or children. Among health spending, nearly half was devoted to in-kind food support or nutritional group counseling classes.

Conclusion

Overall, the interim study shows that after four years of implementation of multiple demand- and supply-side interventions at various geographic levels, few of the short- and medium-term outcomes examined at the community-level had improved beyond conditions in comparison communities. There were improvements in awareness about stunting and related factors at the government level, and village sanitation behavior change meetings occurred with greater frequency in project areas, but none of the other short- or medium-term outcomes were yet achieved. Therefore, it might be difficult for the project to achieve its envisaged long-term outcomes. It will be important to take this into account when finalizing plans for the final round of data collection, in particular to understand why targeted outcomes also improved in comparison areas.

MCC Learning

A detailed theory of change linking each project intervention to expected outcomes, with timelines and targets, is critical for assessing the results of a project and designing useful evaluations.
Think carefully about designing projects whose success relies on synchronized implementation of multiple components that have to work through separate government ministries across a wide geographic area.
It is critical to document and assess the quality of the program being implemented, at a minimum to assess fidelity to the original project design.
The evaluation attempted to use SMS surveys, but had to switch to phone surveys. Despite having been successful in other countries, SMS surveys do not seem appropriate for village-level health surveys in Indonesia due to connectivity issues in remote areas.
A more robust system for monitoring similar interventions occurring in comparison areas was needed throughout the implementation period so that the interim evaluation could have explored this issue, rather than solely detecting it.

Evaluation Methods

Methods

Mathematica is evaluating the impact of the Nutrition Project on stunting and key maternal and child health outcomes using a randomized control trial in 3 of the 11 provinces in which the project was implemented. Random assignment took place in 2013 and resulted in 95 treatment and 95 control sub-districts. The exposure period is up to 1 year and varies by community and intervention.

Data Sources

The interim study relies on three main data sources: (1) quantitative phone surveys with nutrition and sanitation service providers (1,061); (2) qualitative data gathered from focus group discussions and in-depth interviews with providers and administrators, and direct observations of nutrition and sanitation activities; and (3) a database of village-level activities funded by Generasi. Interim data collection occured from September 2017 to January 2018, while project implementation occurred from 2014 to January 2018.

Next Steps

A final study covering the full set of project evaluation questions, including the project’s impact on stunting, is underway and results will be available in 2019.

2018-002-2169-01