Improving rural access to water sources in Lesotho

Program Overview

MCC’s $362.5 million Lesotho Compact (2008-2013) constructed latrines and water systems in rural areas through the $40.1 million Rural Water Activity. The Activity, which had significant effects on key short-term and intermediate outcomes, was built on the theory that access to improved water sources and sanitation saves time on water collection, decreases water-related illness, and reduces medical expenditures and sick care, which ultimately leads to increased productivity and household income.

Evaluator Description

MCC commissioned NORC at the University of Chicago to conduct an independent final impact evaluation of the Lesotho Rural Water Activity. Full report results and learning: https://data.mcc.gov/evaluations/index.php/catalog/137.

Key Findings

Water Sources and Sanitation

Over 29,000 latrines and 175 water systems were constructed by the end of the compact, and an additional 2,768 latrines and 75 water systems were completed post-compact with funding from the Government of Lesotho.
Relative to the comparison group, the likelihood of treated households using improved water sources increased by 50 percentage points; the share of treated household members using an improved toilet increased by 59 percentage points; and the likelihood of all treated household members using an improved toilet increased by 35 percentage points.
The program reduced the average time spent collecting water by 44 minutes per day relative to the comparison areas.

Health and Productive Activity

Despite testing numerous outcomes, no consistent, statistically significant impact on productivity was found. Similarly, no impacts on water-related illness or medical expenditures were found.

Household Income

Despite improvements in access to water sources and sanitation, no impact on household income was detected.

Evaluation Questions

This final impact evaluation was designed to answer the following questions:

1 Does access to improved water systems and installing ventilated improved pit (VIP) latrines increase household use of safe drinking water, improve sanitation, and reduce time spent collecting water?
2 Does access to improved water and sanitation decrease water-related illness, reduce medical care expenditures, increase productive activity, and/or increase household income?

Detailed Findings

Water Sources and Sanitation

Households impacted by the program are located in villages that were identified as lacking access to safe drinking water and adequate sanitation. As a result of the Activity, households in treatment areas were 50 percentage points more likely than comparison households to use improved water sources and the likelihood that all members in a household use a toilet is 35 percentage points higher in treatment areas than in comparison areas. In addition, the share of household members using a toilet was 59 percentage points higher in treatment households vs. comparisons. These results confirm that the program successfully increased the availability of improved water and sanitation infrastructure in targeted communities. The program also reduced the average amount of time spent collecting water in project areas by 44 minutes per day relative to the comparison areas, meaning that treatment households spent 42% less time collecting water than before the program.

Latrine constructed in Ha Salae, District Mafeteng

Health and Productive Activity

Multiple outcomes related to diarrhea were tested, but no statistically significant impacts were found. In addition to the weak findings on diarrhea, there were no statistically significant impacts on medical care expenditures, or missed school or work days due to diarrhea. These results are not surprising, considering the low preexisting incidence of these problems (only 10% of households had a member with diarrhea in the two weeks prior to initial data collection, less than 1% spent money seeking medical care, and only 3% missed school or work due to diarrhea).

Out of 11 different labor outcomes tested, the only statistically significant impact of the program detected was on the likelihood that women worked one hour or more in the past two weeks.

Household Income

According to the program logic, improving access to water would decrease the amount of time spent collecting water and thereby free up time for income-generating work. However, the evaluation did not detect an impact on household income. These results suggest that time availability does not translate into better labor outcomes, such as increased income and number of hours worked. This could be because the latter are not only restricted by time availability, but by other conditions, such as the labor market itself.

MCC Learning

Better adherence to MCC’s investment criteria will help ensure the efficiency of MCC investments going forward. The ex-ante cost-benefit analysis of this Activity estimated that costs exceeded benefits, and the evaluation supported this expectation, finding relatively small impacts on time saving and no impacts on water-borne illness. These analyses suggest that there were likely more efficient uses for MCC’s funding.
Coordinating infrastructure and behavior change interventions in terms of design and timing of implementation will improve effectiveness. Hygiene and sanitation training sometimes preceded the water system construction by years, which may have hindered impacts.
Improving coordination between intervention and evaluation may help to ensure that the evaluation can produce evidence on the effectiveness of all key components of the program logic.
Test critical assumptions underlying the program logic, such as assumptions about water quality, to be able to better understand why targeted results are or are not achieved.
Rigorously plan for and monitor data quality and ensure the evaluator has sufficient sample sizes and continued oversight of survey operations throughout the data collection period.
Be cautious when pursuing randomized roll-out designs, since they require commitment to a strict project design and implementation timeline.

Evaluation Methods

The Rural Water Activity was based on a randomized roll-out design with a 6-9 month gap planned between the end of construction of 50 (early) treatment water systems, and the start of the 50 control (i.e., delayed treatment) water systems. However, construction delays resulted in 11, or approximately 20% of the treatment systems undergoing construction concurrently with the control systems, which raised questions about the internal validity of the original evaluation design. The evaluator addressed these concerns by using different model specifications. The evaluator’s preferred specification is the instrumental variable (IV) model because it exploits the randomized treatment assignment, while accounting for the delays that occurred. The evaluation targeted an exposure period of 9 months, however the analysis reflects an exposure period of 8-17 months across the sample.

Impact Evaluation Multipurpose Surveys (IEMS): Baseline (December 2010) and follow-up IEMS (November-December 2012 and April 2013) were used for this evaluation. The IEMS was a longitudinal analytic survey designed to collect data for the impact evaluations of the Lesotho Compact. One hundred villages targeted for the rural water interventions were sampled for IEMS. Fifty of these 100 villages were randomly assigned to treatment, while the remaining 50 were assigned to the control group. Within each treatment and control village a systematic random sample of 13 households was selected, for a total of 952 completed interviews at baseline and 673 panel interviews at follow-up. Interviews were conducted with the head of the household or the person in the household most knowledgeable about household water and sanitation issues. Information gathered by IEMS questions included water and toilet use, time spent collecting water, diarrhea incidences and treatment, school and work attendance, and income.