Better air quality with energy efficiency in Mongolia

Program Overview

MCC’s $268.9 million Mongolia Compact (2008-2013) funded the Energy and Environment Project (EEP) which included the $33.8 million Energy Efficient Innovation Facility Activity under which the energy efficient stove subsidy was implemented. This program was based on the theory that subsidies and awareness campaigns would motivate more than 100,000 households to purchase and use energy efficient stoves. This switch was expected to improve health by reducing air pollution and easing energy-related economic burdens through reduced fuel and health care expenditures.

Evaluator Description

MCC commissioned Social Impact to conduct an independent final impact evaluation of the Energy Efficient Stove Subsidy, a component of the Energy Efficient Innovation Facility Activity. Full report results and learning: https://data.mcc.gov/evaluations/index.php/catalog/133.

Key Findings

Stove and Fuel Usage

Households with energy efficient (EE) stoves fueled less often, but with more coal per fueling, resulting in no impacts on overall fuel usage (estimated 13 kg coal used per day).
Households with EE stoves reduced daily coal use by 17 percent when instructions were followed, though compliance was low.
Low compliance with EE stove instructions and home insulation likely contributed to lack of fuel savings.

Comfort and Air Quality

Households with EE stoves enjoyed higher average nightly indoor temperatures compared to households using traditional stoves.
EE stoves lowered emissions of fine particulate matter (PM_2.5) by 65 percent and lowered carbon monoxide emissions by 16 percent compared to households with traditional stoves.
The subsidy program reduced PM_2.5 in Ulaanbaatar's air by an estimated 30 percent, with greatest reductions in heavily polluted areas targeted by the program.

Health Outcomes

Health impacts were not directly measured by the data collection, but the evaluation modeled the PM_2.5 dose-response functions for five air pollution-related diseases and estimated 47 avoided deaths and 1,643 disability-adjusted life years (DALYs) averted.

Expenditures and Income

The evaluation found no impact on fuel expenditures or household income.

Evaluation Questions

This final impact evaluation was designed to answer whether or not a subsidy for energy efficient heating stoves:

1 Reduces fuel usage?
2 Improves air quality and health outcomes?
3 Reduces fuel and health care expenditures?
4 Increases household income?

Detailed Findings

Stove and Fuel Usage

Energy effcient stove owners whose homes had 3 or more layers of felt insulation used 2.23 kg less coal each day.

Unlike traditional stove fueling - where an owner first lights wood then adds coal to the top - EE stoves required new approaches to fueling - fully loaded with coal, kindling lit atop the pile of coal, and previous fires completely extinguished prior to refueling. Only 4 percent of EE stove owners were consistent with these fueling requirements, with many households still refueling when previous fires were still warm. EE stove owners whose home had three or more layers of felt insulation used 2.23 kg less coal each day than traditional stove owners. Those with two or fewer layers used the same quantity of coal as traditional stove owners, suggesting insulation may be a key factor that facilitates or inhibits fuel saving benefits. Additionally, EE stoves were not a direct substitute for traditional stoves, as 35 percent of EE stove owners continued to use traditional sources for cooking.

Comfort and Air Quality

Energy efficient stoves demonstrated 65 percent lower emissions than traditional stoves.

Households using EE stoves enjoyed higher average nightly indoor temperatures – almost 3°F – suggesting some users may sacrifice fuel savings for comfort. Based on household emissions, EE stoves demonstrated 65 percent lower emissions of PM_2.5 and 16 percent lower carbon monoxide emissions from project households compared to households with traditional stoves. In addition to finding reduced stove emissions from households, the ambient air quality modeling suggests the EE stove subsidy program lowered ambient PM_2.5 concentrations over Ulaanbaatar, the capital of Mongolia, by 30 percent compared to the counterfactual of all households using traditional stoves.

Health Outcomes

The evaluation estimated improvements in air quality would lead to reductions in air pollution-related illnesses, including: lung cancer (-9 percent), chronic obstructive pulmonary disease (-8 percent), acute lower respiratory infection in ages 0-4 years (-8 percent), ischemic heart disease (-5 percent), and stroke (-2 percent). Considering the range of health outcomes that could be impacted by the project, the evaluation also modeled the impact on disability-adjusted life years (DALYs) which combine the burden of mortality (death) and morbidity (illness) into a single number. The evaluation estimated these improvements in health outcomes would have resulted in 47 avoided deaths and 1,643 DALYs averted for one year of impacts (2012-2013).

Expenditures and Income

Despite households with EE stoves fueling less often, they did not spend less on fuel than households with traditional stoves, estimated at $25 USD per month. The lack of fuel savings is potentially driven by a combination of lack of proper stove operation, preference for warmer homes, and preferences to maintain use of traditional stoves for cooking. The evaluation also detected higher health expenditures in households with EE stoves, however this data was considered potentially inaccurate and counterintuitive, making it difficult to draw definitive conclusions. Given that no impacts on fuel use or expenditures were found, the evaluation unsurprisingly found no impact on income.

MCC Learning

To reduce fuel expenditures, behaviors related to stove use and preferences need to be better understood. In particular, understanding how households prefer to cook versus heat homes would inform understanding of energy and fuel needs and preferences.
Given the variation across households for compliance with EE stove instructions, more understanding of consumer preferences, behaviors, and variation in actual stove operation, particularly for understanding differences between men and women, could inform complementary training and product demonstration activities.
To maximize fuel efficiency and potential savings, the program could have considered “bundling” strategies to incentivize the purchase of energy efficiency packages and use of the stoves for both heating and cooking. Meaningful bundles might have combined the purchase of stoves with insulation, cooking devices, fuel, cooking pots, etc.
Project designers need to consider how changes in the fuel supply market may affect project design. Original stove performance was tested with the main fuel type available in markets, however the evaluation showed that various fuels were used in households. Future projects should develop tests and strategies for alternate stove/fuel combinations.

Evaluation Methods

The final impact evaluation uses a propensity score matching (PSM) methodology to estimate impacts of the Energy Efficient Stove Subsidy. All households living in Ulaanbaatar were eligible to purchase energy efficient stoves through the subsidy program. This market-driven selection process raises challenges for a rigorous evaluation given households could select into the program based on observable and unobservable characteristics. However, the evaluator acknowledged that the PSM methodology can only account for observed characteristics that may influence a household’s adoption of the intervention. The omitting of any potentially unobserved characteristics that may influence a household’s adoption of the intervention, and outcomes of interest, may contribute to remaining bias in estimates.

The evaluation sample was drawn from all five of the most heavily polluted districts of Ulaanbaatar targeted by the project. Given timing of program implementation and the evaluation survey, the exposure period was two winter seasons (2011-2012 and 2012-2013).

The impact evaluation relies on data collected during the 2012-2013 winter months:

Households surveys sampled 1,057 households (856 treatment and 201 comparison) at three points in time during the 2012-2013 winter months.
Stove use monitors recorded fueling event and temperature data for a random subsample of 421 households over more than 100 days during the winter months.
Household stove emissions and indoor air quality measurements of pollutants were obtained from a random subsample of 143 households.
An ambient PM_2.5 sampling and chemical speciation study was conducted to collect data for the air quality modeling and provide additional insights into PM_2.5 emission sources and spatiotemporal patterns. Ambient air quality measurements relied on 36 sampling events per site over the time period January 22 – April 22, 2013.