Modernizing Government Procurement in Indonesia

Program Overview

MCC’s $474 million Indonesia Compact (2013-2018) funded the $73.2 million Procurement Modernization Project focused on achieving procurement efficiency and ensuring that the quality of procurements serves the public interest. More broadly, the program aimed to achieve significant government expenditure savings with no loss of procured goods and services. The project included activities to build a career path for procurement civil servants, create the roles and structures that provide procurement professionals with the authority to implement good practice, and strengthen controls like procurement and financial audits to improve institutional performance. The project was expected to improve procurement efficiency and lead to time and cost savings.

Key Findings

Evaluation Questions

This interim performance and impact evaluation was designed to answer the following questions. This list is a subset:

1. 1
   Did the program result in a change in culture or shared values?
2. 2
   What types of organizational or operational changes are taking place at the PSU (Procurement Service Unit) level?
3. 3
   What types of procedural changes are taking place in the execution of procurements?
4. 4
   Have the procurement knowledge and skills of trainees improved?
5. 5
   Is there evidence that the project activities have resulted in the outcomes outlined in the project logic?
6. 6
   Did the program contribute to a change in perceptions of corruption or transparency?

Detailed Findings

Improved Organization in Procurement Systems

Interviewees reported that the PM Project improved a range of public procurement aspects in Indonesia along the five measures of organizational change (5-S): shared values, structure, systems, skills, and staff. Most interviewees reported that shared values improved among procurement professionals throughout the project regarding commitment to and perceptions of procurement reform. The monitoring data indicates that the majority of PSUs that participated in the project had increased permanence and independence by the end of the project. Further, the project helped PSUs implement several new procurement tools and systems that helped to strengthen the procurement process, including e-tendering and e-procurement. According to interviewees, PSU staff gained both procurement and organizational skills, and knowledge through training and mentoring. Finally, the project helped create more functional positions. Though project targets were not met in this regard, most interviewees thought procurement careers became more attractive as a result of project efforts.

Some Significant Changes in Procurement Quality Outcomes

Procurement reform did not change the time taken to evaluate tenders. However, it increased the amount of time allocated to ensure that the tenderers are qualified. The project also led to a decrease in the number of bidders per tender. These changes may indicate an improved adherence to procurement guidelines, leading to more thorough reviews of bids, which increased the time allocated to verifying the tenders, but likely also weeded out fraudulent bidders. The evaluation found no measurable reduction in the amount offered for each tender, although qualitative evidence suggests that there is some movement toward this goal.

Decreased Opportunities for Corruption

All interviewees agreed that though corruption is still an issue, some reforms have had a positive effect. Framework agreements now include fraud filters and have lowered the number of procurement deals, which lessens the opportunities for backroom deals. Interviewees noted that the e-procurement system also reduced bias in the procurement process, including in the selection of the winners. In addition, there is a perception of increased commitment from LKPP to alleviate corruption in the procurement process. However, several interviewees noted that due to these reforms, some corruption has shifted to other parts of the process, such as budgeting.

MCC Learning

• 
  MCC’s investment model assumes that project teams, including sector specialists, economists, and all partner country team members, would ensure that all components of project designs essential for sustained success are adequately planned, and that the project's results would be adequately measured to understand their likely economic effects. But this model was not well implemented in the Procurement Modernization Project.
• 
  For management information system activities that must both inform project implementation and leave an improved and lasting information system, the project should explicitly plan for the different requirements of these dual objectives.
• 
  During project development, MCC should seek strong sector-specific and theoretically sound guidance to identify conditions under which critical intermediate outcomes would have a predictable relationship with the desired program effects. Evaluation results are most useful when intermediate outcomes have clearly defined relationships with the desired ultimate program effects.

Evaluation Methods

The interim evaluation was a mixed-methods performance and impact evaluation designed to be a light-touch effort with a limited number of interviews focused on capturing key results before the PM Project closed, particularly from staff who implemented the PM Project.

The performance evaluation involved 14 qualitative interviews with key PM Project leaders and PSU staff in fall 2018. The interview data was synthesized to answer questions to assess change along five components of organizational change – shared values, structure, systems, staffing and skills. There may be a positive bias in the results because the majority of the interviewees were also staff who implemented the project.

The impact evaluation used administrative data from the Procurement Management Information System (PMIS) from the end of 2015 through September of 2018 to compare procurement quality outcomes between the treatment and comparison groups. The treatment group was comprised of PSUs selected for Phase 2 of implementation; the comparison group was comprised of PSUs shortlisted but not selected for Phase 2. Between the two groups in this time period, there were roughly 18,000 tender level observations in the data set. To account for any initial differences that might have existed between the treatment and comparison groups, data on baseline characteristics was used to assign analysis weights to PSUs in the comparison group. The analysis weights used predicted probabilities of being selected for the project from a logistic regression with a treatment dummy as the dependent variable regressed on baseline characteristics of the PSUs that were closest to the factors that influenced their selection by the PM Project. Results are likely an underestimate given the comparison group also received some treatment.

Next Steps

The endline evaluation covering the full set of evaluation questions and involving a full quantitative survey and more comprehensive qualitative interviews is underway and results will be available at the end of 2019/early 2020.