Evaluations designs need to be adaptable as the situation on the ground shifts. This evaluation started as a continuous treatment approach using Generalized Propensity Score Matching (GPSM). At the midline, MCC realized that this was not a viable way to account for attribution of results. The design then pivoted to a binary treatment with Lower Ruvu households as the treatment group and Upper Ruvu households as the control. However, the Upper Ruvu households were contaminated by a new water treatment plant in that area. The design then pivoted again to a rigorous pre-post performance evaluation for always connected households and an impact evaluation for newly connected households, using Coarsened Exact Matching (CEM) to compare newly-connected with still-unconnected households. All of the changes to the design reflect the need for flexibility as the situation on the ground changes. In order to allow evaluations to be adaptable, MCC has internal processes such as annual evaluation risk reviews and the Evaluation Management Committee (EMC) to have cross-departmental input to inform evaluation decisions.
Lesson Learned