In Context

The MCC compact with Namibia was a five-year investment (2009-2014) of $304.5 million in three projects: Education, Agriculture, and Tourism. The Education project sought to improve the quality of the workforce in Namibia by enhancing the equity and effectiveness of basic, vocational, and tertiary education and of technical skills. It included six major activities: Improving the Quality of General Education, Vocational Education and Skills Training, Improving Access to and Management of Textbooks, Investment in Regional Study and Resource Centers (RSRCs), Expanding and Improving Access to Tertiary Finance, and Cross-Project Support. The Vocational and Skills Training activity consisted of three sub- activities: (i) establishment of a National Training Fund (NTF); (ii) competitive grants for high priority vocational skills training through the Vocational Training Grant Fund (VTGF); and (iii) expansion and improvement of nine Community Skills Development Centers (COSDECs) and training of the Community Skills Development Foundation’s management staff. The $28 million Vocational Training activity, including its three sub-activities, is the subject of an independent qualitative evaluation released by MCC on September 21, 2016, the interim results of which are summarized here. Given slightly different objectives and targeted populations, each sub-activity is being evaluated separately with different evaluation questions and methodologies; the results from the qualitative interim evaluations are presented in turn below. The Vocational and Skills Training activity represents 10 percent of the total compact. Other components of the compact are the subject of forthcoming independent evaluations.

National Training Fund Sub-activity

Program Logic

The NTF sub-activity within the Education project was designed to address the quality of the country’s labor force by supporting the Namibia Training Authority (NTA) in the creation and operationalization of the National Training Fund, a levy collection and distribution system that aims to provide a sustainable source of funding for high priority vocational education and skills training. Key pieces of the NTF sub- activity included developing the regulations required for fully operationalizing the NTF, establishing the NTF council, and piloting all aspects of the levy. Eligible employers are required to contribute to the NTF through a payroll-based vocational education and training levy. Levy funds are then disbursed to levy paying employers for training carried out and to training providers in order to conduct vocational training in the high-priority skill areas the NTA identifies.

                        

Acronyms: LCDRS = Levy collection, distribution, and reporting system; ISC = Industrial Skills Committee; RPL = Recognition of Prior Learning

* Details for the RPL pilot and operationalization are described in the VTGF program logic below

There were several key assumptions underlying the NTF program logic during the design of the investment:

• NTA receives employer database from Inland Revenue Services, integrates it with the NTA database and updates it regularly.
• Sufficient compliance in levy payment occurs after the LCDRS is in place.
• ISCs identify key priority areas using information from the national plans, market surveys, and stakeholder input.
• RPL certificate is valued in the same way as a traditional vocational training certificate.
• Individuals eligible for RPL, that is, have worked but lack certifications, are aware of the RPL system.
• Training is of sufficient quality and may be improved to the extent that lower-quality providers are screened out of the system.
• Registration and accreditation processes are fair, transparent, and effective.

For a more detailed version of the program logic, please refer to page 6 of the Vocational Training Evaluation Design Report, which can be found here: https://data.mcc.gov/evaluations/index.php/catalog/90/download/388.

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. Where estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

Source: Closeout ITT from December 2014, which includes data through the end of the compact, based on reporting from MCA-Namibia and the Namibia Training Authority.

By the end of the Namibia compact, no progress had been made on the NTF output and outcome indicators. This is largely due to the delay in establishing an operational NTF levy collection system. While the NTF levy was targeted to be completed in spring 2013, it was not completed until April 1, 2014, only six months before the compact closed.

Evaluation Questions

The qualitative interim evaluation sought to address three important junctures in the evolution of the NTF’s levy collection, distribution, and reporting system. These are the system’s establishment, initial operations, and likely sustainability, as outlined in the following questions:

• Was the establishment of the NTF levy collection, distribution, and reporting system implemented as planned?
• How is the NTF levy collection and distribution system operating in practice compared to the specifications outlined in the regulatory framework?
• What are the stakeholder perceptions of sustainability of the NTF levy collection and distribution system?

Evaluation Results

The first round of qualitative data collection, covered in this summary of findings and the associated evaluation report, was conducted shortly after the establishment of the NTF levy collection system and six months before levy disbursements were scheduled to begin. As such, this evaluation primarily focused on outputs, immediate outcomes and initial perceptions about longer-term effects. Specifically, the evaluation focused on the operationalization of the NTF, establishment of the levy collection, disbursement, and reporting IT system, staffing needed for operating the levy, the initial operations of the levy, the NTF’s ability to measure and respond to the demand for skilled labor, and perceptions about the NTF’s sustainability.

Lessons Learned

• Experience “learning by doing” should help the NTA manage the NTF post-compact. The NTA received valuable preparation for its management of the NTF by managing a large portion of the related VTGF sub-activity, which is described further below. Although this opportunity to “learn by doing” does not guarantee that the NTF will be successful, it does make it more likely that the process of awarding and managing grants will operate smoothly and effectively under the NTF. Specifically, the NTA will be able to use improved VTGF procedures and templates, and apply the lessons learned in evaluating the proposed costs of training to expedite the process of awarding grants. MCC compacts should be encouraged to use the compact period to provide implementing entities with similar hands-on experience, where feasible.
• Key policy decisions that affect sustainability should be taken before MCC compacts end. To the extent possible, policy and regulatory hurdles should be identified before Entry into Force, with consultant support as needed, to ensure that partner governments can reap the benefits of intensive technical assistance to operationalize new government functions and increase the likelihood of sustainability. Important policy decisions were still to be determined at the end of the Namibia compact. For example, the process ISCs would use to identify priority skill areas and what constitutes vocational training were both unclear.  This lack of clarity could easily impede the NTA’s ability to identify the skills that are truly in demand and prevent employers from being reimbursed for sponsored trainings. It will be important to resolve these issues soon so that stakeholders maintain support for the system.

Next Steps

A second round of qualitative data collection was conducted in late 2015, and focused on the longer- term evolution of the NTF after the compact ended, including assessing whether the potential barriers in using levy funds effectively had been addressed. A final evaluation report is expected to be released to the public in 2017.

Vocational Training Grant Fund Sub-activity

Program Logic

The VTGF sub-activity within the Education project was designed to address the quality of the country’s labor force by providing funding for vocational skills programs in high-priority areas while the NTF was being established. VTGF included several components, the primary being awarding grants to training providers through a competitive bidding process. The grants were then used to award scholarships to eligible disadvantaged applicants. Additional components of the VTGF sub-activity included piloting a Recognition of Prior Learning (RPL) program, through which experienced employees who do not have formal qualifications receive vocational skills certifications, and funding the marketing of the VET levy to increase public awareness about the levy with the goal of increasing the likelihood of employer compliance. VTGF also intended to establish a framework by which Industrial Skills Committees (ISCs) would identify high priority skill areas; however, not all ISCs were functional during the compact period.

                                   

Acronyms: LCDRS = Levy collection, distribution, and reporting system; ISC = Industrial Skills Committee; RPL = Recognition of Prior Learning

Detailed levy development and processes are reflected in the NTF program logic above

There were several key assumptions underlying the VTGF program logic during the design of the investment:

• Training providers are on the road to the NTA registration and Namibia Qualifications Authority (NQA) accreditation.
• RPL certificate is valued in the same way as a traditional vocational training certificate.
• Training is of sufficient quality.

For a more detailed version of the program logic, please refer to page 4 of the Vocational Training Evaluation Design Report, which can be found here: https://data.mcc.gov/evaluations/index.php/catalog/90/download/388.

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. Where estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

Source: Closeout ITT from December 2014, which includes data through the end of the compact, based on reporting from MCA-Namibia, the Namibia Training Authority.

The average completion rate of output and outcome targets is 81 percent; and in 1 of the 3 indicators, targets were met or exceeded.[[These figures are calculated using all non-evaluation indicators with targets in the VTGF sub-activity.]]

Evaluation Questions

The qualitative portion of the VTGF evaluation was designed to answer questions such as:

• Was the VTGF sub-activity implemented as planned?
• How were the VTGF grants managed?
• What were beneficiaries’ perceptions of the VTGF grants?
• How did employers hire VTGF graduates, and what were their perceptions of the graduates?
• Were the RPL and employer-provided training pilots implemented as planned? How did employers’ perceptions of and attitudes towards their RPL-certified employees change after they became certified? How did the perceptions of employees about their job security and mobility change?

These questions were designed to provide valuable evidence to inform the implementation of similar training mechanisms envisaged under the NTF. Further analysis that will be completed through an impact evaluation will seek to assess the impact of the VTGF sub-activity on its long-term outcomes.

Evaluation Results

The qualitative analysis found the VTGF sub-activity was primarily implemented as planned with some enhancements added during implementation. The first addition was capacity-building grants for training providers to improve infrastructure and equipment and to support instructor training. The grants were intended to make longer-term changes to the vocational training sector by expanding providers’ physical infrastructure in order to meet registration and accreditation requirements. They also served as an incentive to compete for VTGF grants. The second addition was a board and lodging allowance, which aimed to increase attendance and reduce the number of dropouts among VTGF-funded trainees.

Trainees and training providers alike agreed that it was an important addition to the basic training grants, although the sustainability of the allowance is of concern to NTA.

Lessons Learned

• It is important to identify, understand and mitigate risks, e.g., barriers to entering the vocational training market. There are limitations in the number of training providers in Namibia and the courses and levels at which they offer training. This posed a challenge in meeting VTGF training targets and will be an important ongoing challenge in addressing high-priority skill areas through the NTF. This challenge could be especially important if the high-priority areas include new skills and higher levels of skill not currently available from Namibian training providers (which might be the reason for their being high priority in the first place). As an example of a specific barrier to entering the market or expanding offerings, the registration and accreditation process for training providers is widely viewed as lengthy, cumbersome, and unnecessarily duplicative since many of the requirements for registration and accreditation are similar. Future Compacts should identify such challenges when analyzing risks and seek ways to mitigate them, such as streamlining redundant procedures, mapping service provision, examining the possibility of partnering with providers in the region to fill gaps in quality provision (e.g., South Africa has a strong training market), and identifying and addressing weaknesses in the government’s quality assurance mechanisms.
• Compacts must understand the participants they are targeting, what might interfere with their participation or adoption of new skills, and attempt to overcome such challenges, if appropriate. Training providers agreed that the board and lodging allowance was very helpful for decreasing the number of dropouts and increasing attendance, and trainees universally appreciated the allowance. A board and lodging allowance will likely be necessary for future programs like the NTF, even though it may impose a burden on funding of training in high-priority skill areas. If an allowance is offered, it should be included in the early planning phase rather than implemented later, which made administration of the allowance more challenging in the case of VTGF. Ideally, this allowance should be targeted at disadvantaged trainees, but it may be challenging to develop a mechanism to effectively target the allowance (or funding more generally) at this group.

Next Steps

The VTGF sub-activity is also the subject of an impact evaluation using random assignment, which seeks to assess impacts on key outcomes of the sub-activity, including increased completion of training in key priority skill areas, increased paid employment or self-employment, increased earnings, and increased job security and mobility. The evaluation will also try to measure the variation in impact by gender, income level, and language group. The VTGF final evaluation report is expected to be released to the public in 2017.

Community Skills Development Centers Sub-activity

Program Logic

The Community Skills Development Centers (COSDEC) sub-activity within the Education project was designed to address the quality of the country’s labor force through rehabilitating and constructing community-based institutions that provide basic levels of vocational training to clients from disadvantaged backgrounds – particularly out-of-school youth who lack access to the formal vocational training system – to improve their employment prospects. These COSDECs offer two main types of programs: (i) national programs, which typically last between seven and ten months and target out-of- school youth; and (ii) local programs, which typically are shorter and can be taught as center-based programs at the COSDEC itself or as outreach programs on an ad-hoc basis. MCC funded the construction or rehabilitation of seven COSDECs, provided new tools and equipment, and provided technical support for the management of the new or renovated COSDECs. Additionally, MCC helped establish small- and medium-enterprise (SME) units in four of the COSDECs.

There were several key assumptions underlying the COSDEC program logic during the design of the investment:

• Limited availability of qualified trainers will not constrain improvements in training quality and relevance.
• Employers recognize and value the improved quality of the COSDECs.

For a more detailed version of the program logic, please refer to page 9 of the Vocational Training Evaluation Design Report, which can be found here: https://data.mcc.gov/evaluations/index.php/catalog/90/download/388.

Measuring Results

MCC uses multiple sources to measure results, which are generally grouped into monitoring and evaluation sources. Monitoring data is collected during and after compact implementation and is typically generated by the program implementers; it focuses specifically on measuring program outputs and intermediate outcomes directly affected by the program. However, monitoring data is limited in that it cannot tell us whether changes in key outcomes are attributable solely to the MCC-funded intervention. The limitations of monitoring data is a key reason why MCC invests in independent impact evaluations, which use a counterfactual to assess what would have happened in the absence of the investment and thereby estimate the impact of the intervention alone. Where estimating a counterfactual is not possible, MCC invests in performance evaluations, which compile the best available evidence and assess the likely impact of MCC investments on key outcomes.

Monitoring Results

The following table summarizes performance on output and outcome indicators specific to the evaluated program.

Source: Closeout ITT from December 2014, which includes data through the end of the compact, based on reporting from MCA-Namibia, the Namibia Training Authority.

*This total includes the seven COSDECs that were constructed or renovated and a bulk store at the Namibian College of Open Learning that was simply intended to allow the Namibian College of Open Learning free up space for vocational training.

The average completion rate of output and outcome targets is 100 percent; and in all of the indicators, targets were met or exceeded.[[These figures are calculated using all non-evaluation indicators with targets in the COSDEC sub-activity.]]

Evaluation Questions

The qualitative portion of the COSDEC evaluation was designed to answer questions such as:

• Was the COSDEC sub-activity implemented as planned?
• How did COSDEC training affect the employment outcomes of trainees?
• How were the new and renovated COSDECs managed?

Evaluation Results

The first round qualitative analysis of the COSDEC sub-activity focused on implementation and early evidence of changes in operations or perceptions of the COSDECs as a result of the intervention. The qualitative analysis found mixed results on the implementation of the COSDEC sub-activity. Although the construction and renovation of the COSDECs was largely implemented as designed, there was some discontent with the design of the new facilities, the equipping of the COSDECs with new tools encountered challenges, and additional support related to training of instructors and marketing of COSDECs is needed.

Lessons Learned

• Key stakeholders should be consulted during project design in order to ensure investments address existing needs. The technical assistance component of the COSDEC sub-activity could have been better tailored to the Namibian context and the initial capacity of the COSDECs.