Targeting non-communicable diseases and injuries in Mongolia

Program Overview

MCC’s $268.9 million Mongolia Compact (2008-2013) funded the $41.8 million Health Project based on the theory that training more than 13,000 healthcare providers, purchasing equipment, implementing new clinical protocols, and funding more than 219 research and other behavior change campaign grants would reduce premature deaths and disabilities from non-communicable diseases – such as cancer and cardiovascular disease – and preventable injuries among 35-59 year olds. This would lead to reductions in health expenditure, more productivity, and an increase in life expectancy.

Key Findings

Evaluation Questions

Due to an evolving program design with a lack of clear and defined results and targets, MCC did not procure this evaluation until the final year of the compact. This decision limited the independent evaluator to an ex-post performance evaluation design focused on the following questions:

1. 1
   Were activities implemented according to the original plans or were there course corrections?
2. 2
   Were there demonstrated changes in outcomes, including knowledge, attitudes, and practices?
3. 3
   Were project interventions sustainable?

Detailed Findings

All outcome data reported by the independent evaluator are based on surveys funded by MCA-Mongolia during program implementation, including: Facility Based Impact Study (FBIS) 2010 and 2013; Knowledge and Practice (KAP) Surveys of General Population 2010 and 2013; and World Health Organization’s STEPwise approach to Surveillance (STEPS) (a simple, standardized method for collecting, analysing and disseminating data on NCDs) STEPwise 2009 and 2013 (STEP).

Generating Knowledge and Building a Culture of Health

According to a Facility Based Impact Study (FBIS), the percentage of primary health center doctors who reported they had necessary NCD-related protocols (39.3 percent to 41.5 percent), guidelines (39.8 percent to 85.8 percent), and standards (51.2 percent to 78.7 percent) increased from 2010 to 2013. In addition to training, protocol, and guideline provision, the project successfully lobbied food producer Talkh Chiher to reduce salt in bread, and Mon Suu Co to reduce fat in milk by 2-2.5 percent as well as sugar in yogurt by 3-5 percent. Regarding road safety, 13 locations where road traffic accidents were historically concentrated were treated in 2012-2013 to reduce traffic dangers.

According to KAP, while knowledge regarding healthy behaviors remained unchanged on some indicators, others did improve. For example, the percentage of 30-64 year old women who knew vaccination can prevent cervical cancer increased from 15.3 percent in 2010 to 43.5 percent in 2013.

Prevention, Screening, and Treatment Services

The percentage of primary health center doctors reporting their health center had the equipment and supplies to implement NCD prevention, screening, and treatment services increased from 4 percent in 2010 to 59.2 percent in 2013. The pilot HPV immunization campaign provided insights to inform scale-up. More than 9,000 (of the targeted 14,000) 11-15 year old girls received three doses of the HPV vaccine. The evaluator observed the four screening programs were fully integrated in the health system. During a field visit to Bayanzurkh district, the evaluator estimated 10,653 (17 percent compared to 66 percent target) screenings for Arterial Hypertension (AH) and 16,070 (25 percent compared to 66 percent target) screenings for Diabetes Mellitus (DM) took place. The evaluator suggested several more years would be needed to achieve the 80 percent coverage targets set. For treatment of AH patients, compliance remained a challenge due to factors including: drug costs remained unaffordable for some patients; the supply of subsidized drugs in primary health care institutions was insufficient; and patients who did access treatment did not take medication according to protocol.

Behavior, Health, Productivity, and Sustainability

According to STEP, while there were reductions in excessive alcohol consumption (7.4 percent in 2009 to 0.3 percent in 2013), tobacco use failed to change (27 percent), and the percentage of 15-64 year olds with at least three risk factors increased from 26.4 percent in 2009 to 36.9 percent in 2013. The evaluator stated that other outcomes, such as productivity and life expectancy, could not be expected to change in the relatively short project implementation and evaluation time frame. Political and technical leadership in Mongolia was motivated to continue providing guidance and resources, although short-term setbacks were expected given the lack of a sustainability plan, human resources, and funding.

Economic Rate of Return

MCC considers a 10 percent economic rate of return (ERR) the threshold to proceed with an investment. While the evaluator did not recalculate the ERR, they provided feedback on the validity of the ERR produced by MCC (13 percent) in light of the evaluation findings. Issues the evaluator identified include (1) unlikely predictions on the timing of reductions in disease incidence; (2) unlikely assumptions that reductions in NCD incidence will continue every year for 15 years post intervention; (3) an underestimate of the percentage of the Mongolia population that will benefit from the project; (4) lack of sensitivity analysis around a plausible range to highlight the variable nature of the ERR; and (5) questionable valuation of additional productive life years at the avg GDP per person, given that evidence from other countries shows that NCDs affect poorer populations more than the wealthier ones.

MCC Learning

• 
  Identify how to coordinate activities and with who to leverage appropriate technical expertise and build individual activities into the existing and complex system.
• 
  Examine how specific social and gender issues may affect expected results to inform relevant project design elements early in the design stage. Certain issues were not identified until midway through implementation, including: higher traffic accident risks for males, which informed the need for education campaigns targeting males; and mobility issues that could prevent disabled people from benefitting which informed a targeted campaign for disabled populations.
• 
  Include a sustainability plan in project design as a way to ensure the country develops an appropriate operations and maintenance budget for equipment, appropriate staffing, and training plans post-compact.
• 
  To facilitate timely hiring and design of an independent evaluation, aim to clearly define program objectives and targets early in project design.

Evaluation Methods

The performance evaluation relies on an ex-post thematic analysis methodology. Although the Health Project was originally planned for $12 million as a pilot in six provinces and two districts in Ulaanbaatar, it was scaled up to a $41.8 million national project in all 21 provinces in Mongolia following compact restructuring and reallocated funds.

The project was implemented from January 2011-September 2013, giving an evaluation exposure period of 24-33 months for most activities. However some activities, such as the Stroke and AMI Units, were not fully operational at the time of the evaluation.

The performance evaluation relies on information gathered from August-November 2013 with the following methods:

• Desk review of existing data and documentation, identifying, sorting, and reviewing the large amount of available English-language documentation and survey data – FBIS 2010 and 2013; KAP 2010 and 2013; STEP 2009 and 2013 - funded by MCA-Mongolia during program implementation.
• Interviews with a wide range of stakeholders who have been involved in the project over the years.
• Three focus group discussions with a total of 31 health and 20 non-health staff responsible for implementing program components.
• Site visits to two provinces - Arkhangai and Uvs - and one district – Bayanzurkh.
• Workshops on November 5-12, 2013 in Ulaanbaatar for selected stakeholders to provide opinions on preliminary conclusions of the evaluator.