Strengthening Secondary Education in Guatemala

Program Overview

MCC’s $27.3 million Guatemala Threshold Program (2016-2021) aimed to provide Guatemala’s youth with quality educational opportunities that have relevance to the labor market and mobilize additional government resources that are needed to address binding constraints to economic growth. The Education Project consisted of three activities: (1) Quality of Education in Support of Student Success; (2) Improving Technical and Vocational Education and Training; and (3) Strengthening of Institutional and Planning Capacity. This Evaluation Brief focuses on Activities 1 and 3.

Key Findings

Evaluation Questions

This final impact and performance evaluation was designed to answer the following questions:

1. 1
   To what extent were Éxito Escolar’s planned activities for teacher professional development, pedagogical support, and development of parent organizations and school networks implemented as designed? What impact did the interventions have on student learning?
2. 2
   What were the main facilitators of and barriers to implementing Éxito Escolar activities in terms of reaching hypothesized medium-term outcomes including:
   a. Improvements in student learning
   b. Higher graduation rates
   c. Increased retention and promotion of secondary education students
3. 3
   Were the Institutional Strengthening activities implemented as planned (if not, why)? To what extent did the project lead to changes in budget for secondary education? To what extent did the project lead to changes in teacher hiring/retention?

Detailed Findings

Teacher Training, Recruitment and Selection

Éxito Escolar was a comprehensive set of interventions to strengthen Ciclo Básico education in five regions of Guatemala. The activity included training of teachers and school directors, and engaging parents to strengthen school leadership and raise awareness of the importance of Ciclo Básico. In interviews and focus groups, teachers, school directors, and pedagogical advisors shared that they perceived that teachers were using the innovative methods and that those methods were motivating students to remain in school. Teachers expected lower dropout rates, higher promotion rates, and lower grade repetition. Evidence pointed to improved teacher competency in the treatment schools. However, this was also the case for teachers in the control schools. Stakeholders felt that the parent organizations were supporting schools to implement aspects of the action plans. Management advisors and school directors reported that they were able to implement at least a portion of the action plans. However, the interventions did not have an impact on student learning outcomes. It is likely that the COVID pandemic caused further learning loss in students who were already behind grade level making it difficult for Éxito Escolar to overcome learning loss in treatment schools. Differences in estimated impacts were not significantly different between male and female students for language or science, but were significantly lower for female students than male students in math (p = 0.018).

The Guatemalan government officially adopted a Ministerial Agreement to implement the new teacher recruitment and professional development process developed in January 2021. However, hiring was suspended during COVID and has not restarted as of August 2023, which stalled implementation of the new system. The Ministry is using one of the project’s data systems which registers and calculates the demand for teachers, but other capacity building systems are not in use, including those to support teacher recruitment, selection, and monitoring.

Service Standards for Planning and Budgeting

The project identified seven learning standards, but had mixed results. For example, stakeholders do not expect meaningful changes in budgeting. While administrative data shows that the budget for Ciclo Básico increased relative to per-pupil primary school spending, the increase is unlikely related to the institutional strengthening activity. Funding for two high-priority learning standards increased, but it is unclear to what extent the activity contributed to these changes. Despite these small gains, low investment in Ciclo Básico continues to be a structural barrier in Guatemala.

Decision Support Systems for Planning and Budgeting

The project created 14 new information systems to support the use of data to drive decision-making, but the Ministry is currently only using one. Data is rarely used to support decision-making. Most study respondents agree that the distribution of education resources does not respond to the human and material requirements for implementing the national curriculum. Respondents also indicated that information to recruit and select teachers, ensure that students have access to adequate school facilities, and manage learning materials is accessed “with a lot of effort.”

Barriers to institutionalizing reforms include: lack of staff capacity to run, maintain and use the information systems; lack of prioritization of Ciclo Básico; and the absence of a “data champion” within the Ministry to institutionalize the use of data and evidence in policy.

MCC Learning

• 
  Offering varied modalities of programming, including virtual or hybrid instruction, can improve take-up and participation rates in teacher training programs.
• 
  Teacher training programs should consider the competing demands on teachers' time and provide adequate incentives and support relevant to their work.
• 
  Consider early on how to incentivize and build momentum for partner country institutional reforms.
• 
  For student assessments to accurately discern changes in student performance, the difficulty of the questions needs to be aligned with average student performance in the context they are used.

Evaluation Methods

Activity 1 is a is a mixed-methods evaluation organized into two main components. The first component is an implementation study based on document review and three rounds of key informant interviews and focus group discussions with stakeholders. The first round was conducted in Spring 2019, the second round was conducted in 2020 and 2021, and the third round of qualitative data collection was done in 2022. The second component is a randomized controlled trial (RCT) to estimate Éxito Escolar’s impacts on teachers’ time on task, student learning, and enrollment outcomes. Student learning was measured with a student assessment and enrollment outcomes are captured through the Ministry of Education’s administrative data.

The Activity 3 evaluation uses a mixed-methods approach to conduct a performance evaluation composed of two main components: fidelity of implementation and political economy analysis. The Fidelity of Implementation Study draws on multiple rounds of key informant interviews with national and departmental staff, staff from the implementing organization, and staff of the Accountable Entity, PRONACOM. The study also draws on education management information system data, and a document review to examine factors affecting the design, implementation, and results of Activity 3.

The Political Economy Analysis (PEA) uses a mixed-methods approach. The qualitative approach to this analysis draws on multiple rounds of key informant interviews with national and departmental staff and staff of FHI360, the implementing entity, to gather information that allows us to employ DFID’s Drivers of Change framework during analysis (Warrener 2004). The quantitative approach draws on Delphi surveys to systematically gather and consolidate opinions about the system, its features and status, prior to the COVID-19 pandemic.

2024-002-2942