Measuring Results of the Improve the Education of Girls in Niger Project

The IMAGINE 1  project was designed to improve educational outcomes in Niger, with a specific focus on girls, through the construction of primary schools and the implementation of a set of complementary interventions designed to increase girls’ enrollment and completion rates. An independent evaluation measured village-level impacts using a randomized control trial. The impact of the project on targeted population was generally positive, observed mainly by an 8.3 percentage point positive impact on primary school enrollment during the 2012–2013 school year, a 7.9 percentage point decrease in children being absent more than two consecutive weeks during the same school year, and a 0.13 standard deviation impact on math test scores. The evaluation did not find a statistically significant impact on French test scores. In keeping with the project’s emphasis on girls’ educational outcomes, impacts were generally larger for girls than for boys.

In Context

The Millennium Challenge Corporation (MCC) threshold program with Niger was planned as a three-year investment (March 2008 – September 2011) of $24.7 million (of program and administrative funding) in three projects: Girls’ Primary Education; Improve the Control of Corruption; and Streamline Business Creation and Land Access.

 

The IMAGINE project, as originally designed, consisted of the construction of 68 primary schools with high quality infrastructure, along with implementation of a set of complementary interventions designed to increase girls’ enrollment and completion rates. The complementary interventions included the design and dissemination of training modules for teachers, promotion of extracurricular activities, provision  of teacher incentive awards, and implementation of a mobilization campaign in support of girls’ education.

 

In December 2009, MCC suspended the Niger Threshold Program (NTP) due to a pattern of action by the Government of Niger (GoN) that was inconsistent with the democratic governance criteria MCC uses to select its partner countries. 2  At the time of the suspension, $12 million of the planned $18.1 million budget for the IMAGINE project had been expended. The IMAGINE project was ultimately implemented in 10 departments in Niger with low girls’ enrollment and primary school completion rates, with 62 functional schools constructed. A few complementary activities, such as the provision of textbooks and materials for the schools, were fully implemented, whereas teacher training, mothers’ literacy training, and societal awareness campaigns were only partially implemented, and merit-based awards for female teachers, student tutoring, and hygiene and sanitation education were not implemented at all.

 

This report evaluates the impact of the IMAGINE project three years after completion of school construction and partial implementation of complementary activities. Previously, in January and February of 2011, an impact evaluation of the IMAGINE project was completed and found small positive impacts on school enrollment but no impact on attendance or test scores.

 

Program Logic

The primary intervention (listed in the first row of the table below) was the construction of girl-friendly schools. These schools can directly affect enrollment and attendance of girls, which in turn could improve their academic skills and, in the long run, their employment and incomes. The additional activities—such as designing and disseminating teacher training modules, supplying schools with materials and guidebooks for teachers, developing and implementing a communication strategy to advocate for girls’ education, and adult literacy and income-generating projects—are likely to contribute to improving girls’ enrollment and academic skills, but may also improve other outcomes.

Table: Program Logic

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during threshold program implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data are a key reason why MCC invests in independent impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. Where estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output indicators specific to the IMAGINE project.

Indicators

Level

Target

Actual Acheived

Completion rate

Construct new girl-friendly schools

Output

68

62

91%

Train pedagogical inspectors and counselors in gender, spelling, active learning, and evaluation of student performance

 

Output

 

100

 

52

 

52%

Train teachers on gender, spelling, active learning, evaluation of student performance, and tutoring by pedagogical inspectors and counselor

 

Output

 

1,800

 

96

 

5%

Train teachers in spelling and writing

Output

110

96

88%

Equip project schools with teacher guidebooks and school manuals

Output

68

68

100%

Source: Mathematica Policy Research, Impact Evaluation of Niger’s IMAGINE Program, September 2011

 

Evaluation Questions

The principal questions targeted by the evaluation were:

  • What is the current availability of and functionality of the infrastructure constructed under the IMAGINE project?
  • Did the IMAGINE project have any lasting impacts on key educational outcomes?
    1. What is the impact on primary education enrollment?
    2. What is the impact on attendance rates?
    3. What is the impact on learning as measured by test scores?
    4. What is the impact on other measures of education quality?
  • Are the impacts different for girls than for boys?
  • Are the impacts different for children from households of different socioeconomic status?

 

Evaluation Results

The IMAGINE project had an 8.3 percentage point positive impact on primary school enrollment during the 2012-2013 school year, a 7.9 percentage point decrease in children being absent more than two consecutive weeks during the same school year, a 0.13 standard deviation impact on math test scores, and no impact on French test scores. The project impacts were larger for girls than for boys. For girls, the project had an 11.8 percentage point positive impact on enrollment and a 10.6 percentage point impact on attendance, whereas for boys the project had a 5.0 percentage point impact on enrollment and a 5.3 percentage point impact on attendance. The difference in impacts between the genders was statistically significant for both enrollment and attendance. For learning, the impacts on math and French test scores

 

for girls were consistently large and statistically significant, whereas the impacts for boys were smaller and not significant. Impacts on math test scores for girls were 0.11 standard deviations higher than for boys, whereas differences in impacts for the French test between the genders were not statistically significant. The intervention did not appear to affect children from families with different socioeconomic status differently. The only outcome on which socioeconomic status appeared to have a small but statistically significant effect (at the 10 percent level) was on child attendance. In addition, impacts were larger and more statistically significant than those found after the one-year impact evaluation. 3

 

The IMAGINE project had no effect on the availability or number of schools in a village, as schools were widely available in villages prior to project implementation. It did, however, have a sustained positive effect on the presence, quality, and functionality of school infrastructure. IMAGINE schools had greater numbers of classrooms and greater numbers of finished classrooms than non-IMAGINE schools. In fact, on every measure of school infrastructure quality that was gathered, including water source, toilet facilities, preschools, presence of a playground, and teacher lodging, IMAGINE schools’ infrastructure was observed to be of higher quality than that of non-IMAGINE schools.

 

Evaluation results of the activity are summarized in the table below:

Evaluator

Mathematica Policy Research

Methodology

Random assignment

 

The GoN chose 204 villages to take part in the project based on certain eligibility criteria, such as the number of school-aged girls in the village, access to water within the village, and distance to a major road. Sixty-five villages were randomly selected to receive the IMAGINE project in their schools; the remaining 136 were randomly selected control villages.

 

The final sample for this evaluation included a treatment group of 57 villages and 4,092 children, and a control group of 121 villages and 8,977 children.

 

The main sources of data were a household survey of randomly selected families with school-aged children, math and French tests administered to children living in households interviewed in the household survey, a village and school infrastructure questionnaire administered to a village leader and with direct observation of school infrastructure of the  primary school in the village, and a village-level census.

Evaluation Period

October – November 2013

 

Outcome data were collected approximately five years after random selection occurred and approximately three years after school construction ended.

Immediate Outcomes

School Infrastructure Quality

  • IMAGINE schools had on average 6.5 classrooms, of which 5.0 were made of finished materials. This is significantly larger than non-IMAGINE schools, which averaged 5.2 classrooms, of which 2.7 were made of finished materials.
  • IMAGINE schools were 40.8 percentage points more likely to have a functioning potable water source, 69.4 percentage points more likely to have functioning toilet facilities, and 68.8 percentage points more likely to have separate latrines for boys and for girls

   School Enrollment

  • Children were 7.8 percentage points more likely to report having ever been enrolled in school, 7.3 percentage points more likely to report having been enrolled during the 2011-2012 school year, and 8.3 percentage points more likely to report having been enrolled in school during the 2012-2013 school year
  • Girls realized a 6.8 percentage point greater impact than boys on enrollment

  Absenteeism

  • Children were 7.9 percentage points less likely to report being absent more than two consecutive weeks during the last school year
  • Girls realized a 5.3 percentage point greater impact than boys on absenteeism

Intermediate Outcome

Learning Outcomes – Math

  • Children in treatment villages scored 0.13 standard deviations higher on the math assessment
  • Impacts were 0.11 standard deviations higher for girls than for boys

Learning Outsomes – French

  • Test scores in French, measured as both the percentage correct and as an age-normalized score, were higher for children in treatment villages but were not statistically significant
  • Girls in treatment villages scored 0.10 standard deviations higher on the French assessment, significant at the 5% level
  • Impacts were larger for girls than for boys, but the difference between genders was not statistically significant

Ultimate Impact

The evaluation was not designed to assess the impact of the project on employment and incomes; moreover, it is likely too early to do so.

 

Lessons Learned

The main lessons from this activity are:

“Girl-friendly” schools: The IMAGINE project had a large and significant impact on girls' enrollment, attendance, and test scores. Viewed through the lens of the larger impact of the project for girls, it appears that there is indeed a “girl friendliness” about these schools that may be working. Historically in Niger, boys have a much higher rate of school enrollment than girls, potentially because parents may be reluctant to enroll their girl children in shcool due to cultural values or because of the large role girls often play in househole chores. The IMAGINE project has successfully diminshed the difference between boys' and girls' enrollment, attendance, and math schores. We do not know which specific components of the project or the environment were the key drivers of such distinct impacts for girls, but we hypothesize that the elements specifically designed to attract girl students (such as the gender-segregated latrines and female teacher housing), as well as the complementary activities in support of girls education, were responsible for the differentiated impacts.

 

Effects of school characteristics: Since all but one of the surveyed villages 4  had a school prior to project implementation, the effects observed in the current analysis are primarily driven by differences in the characteristics of the schools. IMAGINE schools continue to have significantly better educational infrastructure and resources than non-IMAGINE schools. Because the IMAGINE project did not affect the presence or number of schools available in villages, the impacts on enrollment and attendance are most likely based solely on the school infrastructure and the complementary educational interventions that were implemented during IMAGINE. Indeed, the higher quality infrastructure of schools in IMAGINE villages may be driving parents to enroll their children in school at a higher rate as well as encouraging more consistent attendance.

 

Multiple years of schooling required to acheive improvements in learning: The evaluation conducted in early 2011, approximately one year after completion of school construction, found a 4.3 percentage point positive impact on primary school enrollment, but no impact on attendance or on math and French test scores. The present evaluation, conducted in late 2013, found statistically significant impacts on enrollment, attendance, and math test scores. These results suggest that it may take more than one year of schooling in Niger for an improvement in learning to manifest. Because children stay in school longer in IMAGINE villages than in non-IMAGINE villages, they have more of a chance to learn, which would be consistent with the improvements in test scores after three years, when there were none after one year.

 

Next Steps

The Niger Education and Community Strengthening (NECS) project will be the subject of an impact evaluation. Data collection will occur in spring 2016, with the final evaluation report slated for 2017.

The other projects of the NTP were not sufficiently implemented at the time of the suspension to allow for a rigorous evaluation of their intended impacts.

Footnotes
  • 1. IMAGINE’s official name is “IMprove the educAtion of Girls In NigEr”.
  • 2. In June 2011, MCC approved the reinstatement of threshold program assistance to Niger following the country’s return to democratic rule. In July 2012, the NTP was amended to reduce program funding from $23,066,914 to $16,898,160, of which $2,000,000 was available to complete and expand the girls’ education component of the program as described in the amendment. At the same time, USAID, using some of those funds as well as its own funds, agreed with the GoN to undertake the Niger Education and Community Strengthening (NECS) project to continue and complement girls’ education activities begun under the NTP. NECS will be the subject of a separate evaluation.
  • 3. The one-year impact evaluation found a 4.3 percentage point positive impact on primary school enrollment, no impact on attendance, and no impact on math and French test scores.
  • 4. i.e. both treatment and control villages