Promoting policy reform in Honduras

Program Overview

MCC’s $15.6 million Honduras Threshold program (2013-2019) focused on improving the transparency and efficiency of public financial management (PFM) and public-private partnerships (PPPs) by providing technical assistance to key government institutions. It aimed to save the government money, improve service delivery, and reduce corruption through operationalizing best practices for budget and treasury management; streamlining procurement; strengthening auditing capacity; providing grants for civil society oversight; and augmenting capability of PPP professionals.

Evaluator Description

MCC commissioned Social Impact (SI) to conduct an independent interim performance evaluation of the program. Full report results and learning: https://data.mcc.gov/evaluations/index.php/catalog/208.

Key Findings

Program Implementation

At inception, this policy and institutional reform (PIR) program only identified high-level objectives, and lacked a detailed theory of change with clear pathways to achieve stated objectives.
This flexible approach to program implementation has allowed the program to respond to opportunities and adapt to challenges while driving towards those objectives. Conversely, this evaluation cannot measure fidelity to original design.

Public Financial Management

Due to the implementation of a PFM best practice (“baseline budgeting”), government institutions’ budgets increasingly reflect the actual costs for service delivery.
Procurement assessments and trainings have led to improvements in government procurement. Innovative civil society oversight of government institutions has resulted in public, agency improvement plans.
Nonetheless, several ambitious budgeting, invoice payments, and procurement goals will require longer-term reform efforts beyond the program.

Public-Private Partnerships

Two key government PPP institutions have not taken advantage of program support, limiting improvements in the PPP process.
Instead, the program has supported and built the capacity of a unit in the Secretariat of Finance to ensure that PPPs do not pose a financial risk to the government.

Evaluation Questions

The interim evaluation was designed to contribute to an understanding of intermediate outcomes, along the path to the final outcomes framed by the research questions below. Additionally, it documented how the program had adapted, informed by improved specificity within the theories of change. The interim results were critical to identifying the focus of the final evaluation.

1 What were the results of the interventions – intended and unintended, positive and negative?
2 What are the lessons learned and are they applicable to other similar programs?
3 What is the likelihood that the results of the program will be sustained over time

Detailed Findings

Visual showing Activities within the Threshold Program's PFm and PPP Projects

Program Implementation

The program has adapted to take advantage of new opportunities. For example, when a civil society organization signed an agreement with the government to conduct civil society led assessments of government institutions, the program stepped in to fund the initiative. In addition, with a willing partner institution and two capable consultants, the program expanded the scope of its procurement reform efforts to include a robust training program and a certification process for procurement officials. Nonetheless, prioritizing flexibility has undermined articulation of a clear theory of change, basic monitoring, and timeliness.

Public Financial Management

Efforts to improve PFM offer considerable promise, but the most ambitious elements will require more time. For example, baseline costing is helping Honduran government institutions ensure that budgets reflect the cost of service delivery; however, transitioning to multi-year, government budgeting (a “medium-term expenditure framework”) is inherently a long-term endeavor, as budget innovations can only be piloted or rolled out annually. Procurement assessments, Honduras’ electronic catalogue, and trainings are leading to improvements in government procurement; however, creating a formal certification for procurement officials and closing the loopholes that allow for corruption will require ongoing, long-term support. The Threshold program identified the complex and vicious cycle that perpetuates cash shortfalls and delays in paying invoices; however, their proposed solutions are dependent on major changes in how government institutions plan, budget, and procure goods and services.

Three of the PFM interventions entail audits or assessments of government agencies conducted by the Supreme Audit Tribunal, a civil society organization, and the government’s procurement agency (ONCAE). The impact of these assessments on promoting internal reform has varied across the three initiatives and across the targeted institutions. This approach is more likely to have an influence on assessed institutions if there are:

Baseline report from the Secretary of Education

Complementary interventions beyond the assessment itself
Robust follow-up mechanisms
Public release and dissemination of the results; and
Political and ministry leadership support.

Public-Private Partnerships

The lack of buy-in at the main PPP institution, COALIANZA, and insufficient ownership at the Secretariat of Public Works and Infrastructure have prevented the program from achieving many of its original PPP objectives. Instead, the program has worked closely with the Secretariat of Finance to support the establishment of a Fiscal Contingencies Unit with the authority to prevent PPPs from moving forward if they present a financial risk to the government. In addition, the Threshold program has rolled out an extensive training program to build PPP capacity throughout government institutions involved in the PPP process. The program’s experience promoting PPP improvements demonstrates the importance of partner ownership to achieve donor goals and objectives.

MCC Learning

Problem definition in PIR programs is not a linear process and will continue to evolve throughout program implementation. The intervention should adapt accordingly to achieve stated objectives.
PIR programs require a higher level of effort from MCC dollar-for-dollar than an infrastructure program due to the complexity of institutional change. MCC should be mindful to resource adaptive PIR programs accordingly for implementation.
An adaptive monitoring and evaluation framework and strategy is critical to support data-driven evolution of the program. This may include multiple detailed revisions to program logics and reassessment of measurement indicators.
The sustainability of PIR programs is critically dependent on champions and the specific political environment. MCC must have a willingness to back away from programs that are not working, and to tilt on the program design partway.

Evaluation Methods

This interim evaluation is part of a larger longitudinal performance evaluation that involves diverse data collection activities. To achieve stated objectives, the program sought to improve performance of 4 key government institutions through a multi-pronged approach (Secretariat of Finance, ONCAE, Supreme Audit Tribunal, and COALIANZA). Early in the program, it was unclear what pathways would lead to critical outcomes, hence the evaluation was designed to collect a wide swath of data to be able to correspondingly adapt to program implementation. Each round of data collection also narrows the focus and prioritizes outcomes for assessment.

Visual showing selected findings from baseline evaluation surveys.

This interim report relies on data collected through (1) a review of relevant documentation; and (2) key informant interviews, group interviews, and case studies. The exposure period was assumed to begin at the initiation of a long-term ongoing technical assistance activity. The goal was to have a minimum of 6 months of exposure to the intervention.

Administrative Data and Document Review

The document review included reports, deliverables, audits, and assessments, which were provided by program implementing partners. This evaluation method assessed the program design, documented the evolving focus of the program, and validated the improved specificity in the theories of change developed during the penultimate year of the program.

Key Informant Interviews to Assess Intermediate Outcomes

Qualitative data is critical in this evaluation to contextualize reports, indicator data, and survey results. The evaluation team conducted key informant and group interviews with Threshold program stakeholders, including consultants, implementing partners, partner government institutions, MCC and MCA-Honduras personnel, personnel from case study government institutions, and vendors to the government in March 2016 for baseline, and in October 2017 for the interim round. In total, 120 individuals were interviewed during interim data collection. The evaluation team also explored the process of structuring, procuring, and managing PPPs across six case studies of programs.

Next Steps

Enumerators from partner ESA Consultores attend survey training

The final rounds of qualitative key informant and group interviews with Threshold program stakeholders are underway (October 2018, and August 2019).

In addition to the qualitative research presented in the interim report, the final evaluation will also include pre-post analysis from two surveys. In 2016, the evaluation team collected baseline data from (a) 1719 public employees at three key institutions, and (b) a panel of 853 vendors who sell goods and services to the Honduran Government. Final rounds of both surveys will be conducted after program closure in late 2019 to inform a pre-post analysis of outcomes.

A report containing final analysis of program outcomes will be available by 2020.

2018-002-2194-01