Community-Based Health and Nutrition to Reduce Stunting Project
- $131.5 millionOriginal Compact Project Amount at Signing
- $120.4 millionTotal Disbursed
Estimated benefits correspond to $112 million of project funds, where cost-benefit analysis was conducted:[[As estimated in the cost-benefit analysis at time of signing.]]
| Estimated Economic Rate of Return | Estimated beneficiaries | Estimated net benefits | |
|---|---|---|---|
| At the time of signing | 13% | 2.9 million children | $46.6 million |
| Updated | 16.5% | 1.7 million children | $113 million |
| At compact closure[[
MCC will not re-estimate the ERR for this project at closeout, but will contract with the Independent Evaluator to produce an evaluation-based ERR.]] |
N/A | N/A | N/A |
Economic Analysis: MCC undertook an initial economic analysis of the Community-Based Health and Nutrition to Reduce Stunting Project (the Nutrition Project) at the time of compact approval in 2011. The economic rate of return (ERR) for the project was estimated at 12-13 percent and it was projected that the project would benefit 2.9 million children in 7,000 villages. This analysis was based on the project design as of mid-2011. Several project components were modified since then, including service provider training, sanitation and hygiene activities, provider incentives, provision of micronutrients, the national stunting awareness campaign, and the private sector response. The expected potential benefits of these activities were not fully captured in the initial analysis.
As a result, following detailed design, the economic analysis was updated in November 2013 to better reflect the evolving project implementation plans. The revised analysis incorporates the updated program design as well as additional data from the impact evaluation of the PNPM Generasi which facilitated quantification of the anticipated education benefits from Generasi block grants, adding substantially to the estimated total benefits of the project.[[Olken, Benjamin A.; Onishi, Junko; Wong, Susan. 2011. Indonesia's PNPM Generasi Program : final impact evaluation report (English). Washington, DC: World Bank.]] With these changes, the revised model increased the estimated ERR from 12-13 percent to 16.5 percent. The selection of the final project locations (499 sub-districts in eleven provinces) and a revised implementation timeline resulted in a decrease in the estimated number of beneficiaries to approximately 1.7 million children in 5,300 villages. The impact evaluation for the Nutrition Project in Indonesia will include an economic analysis of the program that reflects further project design changes, final costs, and evaluation findings.
Project Summary
In Indonesia at the time of compact development, over one third of children under five were stunted, which can have a major impact on their lives, putting them at higher risk of chronic disease, delayed cognitive development, delayed enrollment in school, and reductions in future earnings. The Nutrition Project was originally conceived as a way to build community knowledge of and demand for health services that could combat stunting, and strengthen the health system
infrastructure at the local level to deliver these services. The project was designed to respond to a problem not directly linked to the binding constraints, but noted as an issue of inequality of opportunity in the country. In particular, the constraints analysis contains a statement regarding unequal access to economic opportunities, namely that “some groups of people have weaker human capabilities than others, partly due to unequal access to education; health; and/or other social services such as clean water and sanitation.”[[Asian Development Bank (2010), Country Diagnostic Studies, Indonesia: Critical Development Constraints. This quotation is from the Executive Summary on page 4 but the larger discussion can be found in section 4.2.1 Human Capabilities beginning on page 57.]] The analysis pointed out that this lack of access was due to both the “supply” side of services available to communities and the “demand” side of communities seeking options to improve their healthcare.
In particular, the objective of the Nutrition Project was to reduce and prevent low birth weight, childhood stunting, and malnourishment of children in project areas and increase household income through cost savings, productivity growth, and higher lifetime earnings. The project also aimed to determine the effectiveness of the use of MCC funding in a multi-donor trust fund, managed by a multilateral institution, and its impact on poverty reduction. Design was based on the findings of a rigorous impact evaluation for a previous intervention, which found that an existing GOI program was delivering positive health and school enrollment impacts at the community level, but qualitative data suggested that communities that wanted services could not get them from their local health posts.[[Olken, Benjamin A.; Onishi, Junko; Wong, Susan. 2011. Indonesia's PNPM Generasi Program : final impact evaluation report (English). Washington, DC: World Bank.]] The evaluation suggested that greater impact might be possible if the “demand‐generating” community empowerment and education activities were coupled with a “supply‐side” set of interventions to meet this demand.
The Nutrition Project consisted of the following three Activities:
| Activity | Revised Compact Allocation | Final Disbursements |
||
|---|---|---|---|---|
| 1 | Community Projects Activity | Community block grants and participatory technical assistance to communities | $81.6 million | $78.6 million |
| 2 | Supply Side Activity | Training to service providers, sanitation and hygiene activities, provision of multiple micronutrient packets, materials to measure children’s height, and private sector interventions | $33.9 million | $23.5 million |
| 3 | Communications, Project Management and Evaluation Activity | Communications outreach, project management, and monitoring and evaluation | $18.6 million | $18.2 million |
| Total | $134.2 million | $120.4 million | ||
The Community Projects Activity, also known as the “demand side, provided community block grants and participatory technical assistance to communities. This included the National Community Empowerment Program, and Healthy and Smart Generation. Alongside the World Bank, AusAID, the European Union, and other development partners participating in a World-Bank-managed, multi-donor trust fund, this activity aimed to scale up the GOI’s National Community Empowerment Program – Healthy and Smart Generation, known as Generasi, which sought to improve access to health and early childhood education in more than 5,300 villages across 64 districts in 11 provinces.
The Supply Side Activity funded training to service providers, sanitation and hygiene activities, provision of multiple micronutrient packets, materials to measure children’s height, and private sector interventions. More specifically, it included several subactivities:
- Increasing Knowledge and Skills of Health Workers: Through this sub-activity, Indonesian Ministry of Health (MOH) staff and volunteers from the provincial to village level received training and support for community activities aimed to reduce stunting, covering topics such as infant and young child feeding, community-led sanitation, sanitation entrepreneurship, growth monitoring, and micronutrient quality assurance. This sub-activity also included the delivery of Sanitation Triggering Events, a method of inciting behavior change at the village level around sanitation practices using interactive demonstrations to show how bacteria can be transmitted through poor sanitation. Health centers were also provided with anthropometric kits to facilitate more frequent and reliable child length measurement in communities;
- Micronutrients for Pregnant Women: Iron, folic acid and a micronutrient powder were delivered to district health offices for distribution to pregnant women by Ministry of Health and community-based staff, with the aim of reducing maternal anemia. The project planned to distribute micronutrient sachets for children, but was not able to do so due to conflicting specifications for the supplement; and
- Private Sector Response: Grants were awarded to leverage private sector resources and develop market-driven solutions to address community needs for safe water and sanitation.
The Communications, Project Management and Evaluation Activity included communications outreach, project management and monitoring and evaluation. Under the activity, the National Nutrition Communications Campaign provided mass media and training for interpersonal communication at the community level aimed to increase awareness of stunting, evoke commitments from stakeholders to tackle stunting, and foster behavior change related to health, nutrition, and sanitation.
Most of the components of the Nutrition Project were implemented during the compact term, but almost none of them were initiated on time, nor were they implemented in tandem. While the Community Projects Activity took place from 2014-2017 as planned, the remaining project activities had a very slow start as the MOH struggled to understand MCC requirements. In some cases, this led to lower realization of targets (e.g. number of sanitation triggering events), in other cases to rushed but ultimately successful activities (e.g. training for health workers) and in at least one case to the component not being implemented at all (micronutrients for children). While each component was different, the common thread was that MCC’s requirements for competitive procurement and tight fiscal accountability standards were not compatible with the implementing entity’s (the MOH) experience of implementing projects funded by other donors. As an example, it took over one year to develop a sufficiently accountable payment system to enable individual MOH health workers to attend trainings.
The project’s design had been predicated on the “demand” (Activity 1) and “supply” (Activities 2 and 3) sides taking place simultaneously and in the same locations so that the coordinated interventions could affect three cohorts of children during the implementation period. Since this did not materialize, it is likely that the impact of the project will differ from what was originally envisioned. MCC did not estimate the magnitude of impact on any of the quantitative outcomes targeted by the project; however, the impact evaluation was designed to detect a five percentage point reduction in stunting prevalence.[[During the design stage, the independent evaluator proposed a 5 percent effect size as a reasonable effect size to expect based on the project cost. The power calculations were driven in large part by the number of sub-districts in the three treatment provinces.]]
Nonetheless, the Nutrition Project has formed a cornerstone for and been supported by a larger movement to increase awareness about stunting in Indonesia and channel resources to address the problem. During the compact term, Indonesia played a pivotal role in the global Scaling Up Nutrition movement, which it joined shortly after compact signature.[[http://scalingupnutrition.org/]] MCC was the first donor to fund an explicitly anti-stunting project in Indonesia, instead of one focused on malnutrition or sanitation alone. Since then, other donors have made or are planning similar investments in a multi-sectoral approach to fight stunting, including a $400 million follow-on investment to the Nutrition Project made by the World Bank in 2018. Similarly, the MOH has recognized the links between malnutrition and sanitation, combining its directorates of sanitation and nutrition under a single director general. The MOH also introduced a new word into Bahasa Indonesia: “stanting.” Prior to the introduction of this project, there was no word for stunting in the language, with even health professionals using “orang pendek” (“short person”) or “kerdil” (“dwarf”) to refer to the condition. Today, stunting is a national priority for Indonesia, with national and local governments making very public efforts to coordinate between sectoral agencies and bring additional resources to bear in tackling the issue.
Project Sustainability
For most of the components of the Nutrition Project, sustainability was a critical element of project design. As the project was implemented, the level of GOI commitment and buy-in was very high, and the project benefited from co-financing by the MOH at the national level during some years. Local governments also provided significant funding towards project activities, with MCC funding causing greater investment by local authorities into existing maternal and child nutrition budgets in order to replicate MCC-funded trainings for larger groups of healthcare workers. In 2016 alone, over 25 districts had already replicated the infant and young child feeding training and several districts had replicated the sanitation triggering activity, all using their own funds. This is mainly because the project activities were substantially aligned with national, provincial, and district government policies and represented a renewed emphasis on an existing policy trajectory. Similarly, the Community Projects Activity funded a scale-up and enhancement of an existing government program, further intensifying the GOI’s focus towards the issue of stunting and bringing co-financing and cooperative efforts to bear. At the end of the compact, the GOI integrated the Nutrition Project, particularly the training and sanitation activities, into the MOH’s existing health and nutrition programs.
During project implementation, MCA-Indonesia revised, in partnership with the MOH, the national guidelines on Mother Infant Young Child Feeding, Growth Monitoring, Community-Led Total Sanitation, Sanitation Entrepreneurs, Sanitation Entrepreneur Mentors, Supportive Supervision, Iron Folic Acid and Taburia Quality Assurance, and Distribution and Behavior Change Campaign Procedures. These guidelines are being used by the MOH after the end of the compact. The MOH has built upon the national nutrition communications campaign model developed by MCA-Indonesia to inform future MOH and NGO health and nutrition behavior change work. While the MOH will be the primary entity responsible for project activities post compact, MCA-Indonesia also signed an Implementing Entity Agreement with the Ministry of Villages in May 2017 to promote the sustainability of the Nutrition Project’s investments by supporting the creation of minimum services standards and guidelines for a stunting response that will utilize funds provided directly to the village level.
The MOH was quite supportive of supply-side activities when they were congruous with its own policy priorities. However, MCA-Indonesia’s ability to move the MOH in new directions was not very encouraging. Substantial delays in supply-side activities were the result of innumerable pressures on MCA-Indonesia to conform to existing policies and resistance by the MOH to change business models to tackle the issue of stunting in different ways. Over the compact period however, the MOH made a greater effort to include other ministries and parts of government in its dialogues on health issues, recognizing that the problems need a multi-sectoral solution. These new dialogues also led to some realignment within the MOH structure to take a more comprehensive approach to nutrition, including establishing a clear link between sanitation and nutrition by combining the respective directorates.
Evaluation Findings
A rigorous impact evaluation is underway for the Nutrition Project which will use both quantitative and qualitative methods. The quantitative approach is expected to be a randomized controlled trial. The impact evaluation will measure the project’s impact on child and maternal health outcomes (including stunting), behavioral practices related to nutrition and sanitation, and receipt of health services.
Status of the evaluation: Community Based Health and Nutrition to Reduce Stunting Project
| Component | Status |
|---|---|
Baseline |
Data collection completed between November 2014 and February 2015. Report is publicly available. |
| Interim | Data collection completed between November 2017 and February 2018. Report is publicly available. |
| Endline | Data collection planned for early 2019. Report expected by early 2020. |
The key findings of the interim evaluation of the Nutrition Project include:
Implementation Quality
- Training levels were significantly higher in project areas, but similar trainings were occurring in comparison areas.
- Child nutrition training was implemented as intended with high quality. Sanitation training was implemented, though not as comprehensively as intended.
Short-Term Outcomes
- Modest improvements in health provider knowledge, but not uniformly. Impacts varied by question and provider.
- No impacts on frequency of nutritional group counseling, which was of mediocre quality, nor on the share of women receiving one-on-one health services.
- Village-level meetings to initiate behavior change were more frequent in project areas, but omitted key steps.
- Villages received Generasi grants as intended and most funding went toward health-related activities.
Medium-Term Outcomes
- No improvement in village-level open defecation free status.
Interim findings overall indicate that long-term outcomes may not be achieved.
Key Output and Outcome Indicators
Community Projects Activity
Outcome: Improved health and education outcomes, including nutrition
| Key Performance Indicators | Baseline | End of Compact Target | End of Compact Achievement | Percent Target Complete |
|---|---|---|---|---|
| Value of Generasi block grants funded to sub-districts | 0 | $68,816,000 | $88,256,848 | 128%[[This indicator reports total Generasi block grant spending against the target for MCC’s contribution to Generasi’s block grant budget. The percent complete can be interpreted to mean that Generasi distributed block grants in excess of MCC’s contributions, by 28%. MCC’s targeted distribution toward Generasi block grants was met.]] |
| Number of Generasi Activity proposals approved | 0 | No Target | 181,912 | No Target |
Supply-Side Activity
Outcome: Improved ability of health service providers to prevent, diagnose, and treat stunting; improved nutrition of pregnant women and infants; improved sanitation behavior; and reduced incidence of diarrhea
| Key Performance Indicators | Baseline | End of Compact Target | End of Compact Achievement | Percent Target Complete |
|---|---|---|---|---|
| Number of open-defecation-free villages in MCA-Indonesia working areas | 0 | 800 | 218 | 27% |
| Number of Sanitary Toilets constructed by Private Sector Response Activity grant partners | 0 | 1,080 | 1,182 | 109% |
| Iron folic acid tablets delivered to district | 0 | 35,491,680 | 35,626,390 | 100% |
| Taburia (micronutrient) packets delivered to district | 0 | 18,943,200 | 0 | 0% |
| Number of sanitation triggering events held at sub-village level | 0 | 6,400 | 4,225 | 66% |
| Number of anthropometric kits distributed | 0 | 1,408 | 1,186 | 84% |
| Number of service providers trained on growth monitoring | 0 | 1,558 | 1,564 | 100% |
| Number of service providers trained on Infant and Young Child Feeding | 0 | 18,578 | 17,531 | 94% |
| Number of service providers trained on community-led total sanitation triggering | 0 | 7,433 | 6,724 | 90% |
| Number of service providers trained on supportive supervision | 0 | 1,558 | 1,207 | 77% |
| Guidelines on integrating health, nutrition, and sanitation into village planning and budgeting process developed | N/A | 31-Dec-17 | March 26, 2018 | Not applicable |
Communications Activity
Outcome: Increased awareness about stunting
| Key Performance Indicators | Baseline | End of Compact Target | End of Compact Achievement | Percent Target Complete |
|---|---|---|---|---|
| Number of people trained on interpersonal skills and communication | 0 | 930 | 1,588 | 171% |
| Stakeholders and policymakers engaged on stunting prevention | 0 | No Target | 8,455 | No Target |
| Number of television spots aired | 0 | 2,450 | 4,155 | 170% |
Explanation of Results
The targets for the Nutrition Project indicators were set during the compact implementation period, after implementation plans were developed and finalized. In some cases, particularly for sanitation training and triggering and the communications campaign, they do not reflect the original vision for the project to have uniform implementation of a package of interventions across all target sub-districts. The project was not adequately funded to implement all envisioned components equally across all targeted geographic areas. While many targets were met by the end of the compact, they were intended to have been met much earlier during the implementation period to allow for the demand- and supply-side interventions to have a coordinated impact on the target population of children under the age of three.
The implementers of the Generasi program were not able to provide timely and reliable monitoring data, so little meaningful monitoring data exists of this activity despite it representing the bulk of project spending. The only targets set related to the Generasi grants were for disbursements to the Support Facility, not for activities related to the use of those funds. MCC funds were expected to cover only a share of Generasi grants across the 11 provinces, but ultimately the compact funded the majority of Generasi grants from 2014-2017. No target was set for the number of Generasi activity proposals, as this was dependent on community needs.
The Supply-side Activity training and equipment targets were mostly met; when they fell short, it was generally attributed to delays in processing of paperwork and payment for trainings. The Taburia distribution activity was cancelled in the final year of compact implementation after the first batch of production because the specification of Taburia produced by the vendor did not match with the specification stated in the bidding document. The sanitation triggering work, and the associated outcome of open-defecation-free village status, fell short of expectations due to delays in trainings that had to precede the triggering events. One reason why the target number of open-defecation-free villages was not met could also be that the coverage of triggering events across sub-villages was not sufficient to result in open-defecation-free status in the short time period allowed. The communications campaign work exceeded its targets, though the sub-national activities were only conducted in 11 of the 64 project districts.
Lessons Learned
Beginning at the concept paper stage of compact development, the Nutrition Project enjoyed a privileged position as the investment with the greatest level of country buy-in within the Indonesia compact. This was positive in that high-level declarations and policy positions that were supportive of the intervention were relatively easy for the national GOI to enact. Similarly, bureaucratic reforms like combining directorates (as discussed above) were quick to come to fruition. There also appear to have been positive effects at the provincial and district levels, with motivated bureaucrats in both MCC and non-MCC funded areas beginning to direct resources towards a multi-sectoral fight against stunting.
However, as discussed elsewhere in this report, the less positive result of this close coordination with the GOI was that significant project delays were experienced any time that the project needed Ministry of Health permission or regulatory action in order to move forward. This dynamic played out repeatedly, including leading to a delay of over a year to put in place a mechanism for paying training participant expenses in a way that ensured no funds were misplaced and to a re-issued regulation for the composition of Taburia that came too late for the project to implement successfully.
Ultimately, the project was not high enough priority for the Ministry of Health to focus on, yet it still retained approval rights for each operational detail. For MCC, this is an important lesson to consider. The more MCC retains control over the “how” of project implementation—in other words being able to independently manage inputs and have more control over outputs—the more a project’s theory of change will be adhered to, but this will have an impact on sustainability if MCC’s country partners do not engage in implementation. In the Nutrition Project in Indonesia, there was significant emphasis on ensuring GOI buy-in for sustainability. As a result MCC did see high-level commitment to continue the project objectives, but this came at a cost of operational speed and effectiveness, as each implementation action was elaborated to meet MCC standards of fiscal accountability as well as GOI’s standards.
The findings of the interim evaluation correspond to these lessons, pointing to potential breakdowns in the project theory of change, which could be due to the lack of synchronization among the various components of the project. This raises a number of additional lessons for MCC:
- Given MCC’s operational model, the agency should think carefully about the implementability of projects during the design phase of a compact. In particular, issues seen in the Nutrition Project demonstrate the difficulty of implementing a project with multiple components that have to work through separate government ministries across a wide geographic area, using different implementers. As the evaluation report discusses, the project that was ultimately implemented differed significantly from what was designed, and this presented challenges for achievement of the objective of reducing stunting in a way that was attributable to the project. To improve implementation outcomes for complex projects, MCC may also consider imposing more discipline in adhering to work plans and timelines that are approved as part of each quarterly disbursement request.
- The lack of a detailed, evidence-based theory of change with timelines and targets for key outcomes for the project presented a continuous challenge for both the project and the evaluation. While the project logic could have served as a detailed tool to guide project decision-making in the context of results, it was not well understood or meaningful to key government decision makers who instead relied on project budgets and work plans that had no firm deadlines other than the end of the compact. As a result, decisions were made that potentially hindered the ability of the project to reduce stunting. The cost-benefit analysis was not able to serve this purpose because there was no evidence on which to model the benefits of certain project components. For the evaluation, this was challenging because there was no quantitative accountability framework around which to design the evaluation, e.g. there was no targeted impact on stunting prevalence, despite stunting reduction being the primary objective of the project. The lack of a more detailed theory of change, linking each of the project components to the key outcomes, presented challenges both for designing the interim evaluation and interpreting its results. Going forward, MCC should ensure that project objectives are both feasible to achieve within the given timeframe, measurable in a cost-effective manner, and possess targets that are explicitly drawn from the cost-benefit analysis.
- The implementation study included in the interim evaluation provided important insights into the quality of project implementation and the likelihood of achieving targeted results. It is critical to document and assess the quality of the program being implemented, at a minimum to assess fidelity to the original project design. MCC should assess the quality of implementation as a standard monitoring practice for other projects moving forward.