Supplying STEM trainees to meet market needs in Georgia

Program Overview

MCC’s $138.6 million Georgia II Compact (2014-2019) funded the $15.7 million Industry-Led Skills and Workforce Development Project, which aimed to improve the alignment between the skills of technical and vocational education and training (TVET) graduates and the skills demanded by the labor market. It provided larger grants to 10 TVET providers to establish 51 new or improved courses with industry support; awarded 27 smaller grants to document and disseminate best practices in the TVET sector; provided technical assistance to the government related to sector policy; and held three annual TVET conferences.

Evaluator Description

MCC commissioned Mathematica to conduct an independent final performance evaluation of the ISWD Project. Full report results and learning: https://mcc.icpsr.umich.edu/evaluations/index.php/catalog/195.

Download the Georgian translated evaluation brief.

Key Findings

Labor Market Outcomes of Trainees in New TVET Courses

More than three-quarters of technical and vocational education and training (TVET) trainees found employment within a year of the course ending; however, only one-third found a full-time job relevant to their course.
Benchmarking against all public TVET courses in Georgia suggests that the new courses likely improved the employment rate and earnings of trainees during the pandemic.

Sustainability of New Courses

The 10 TVET providers with larger grants continued to offer all of the new courses two years after the end of the compact.

Sustainability of Activities to Strengthen TVET Policy and Practice

Smaller grants supported some best practices that grantees sustained, but evidence of widespread adoption is limited.
The project’s technical assistance efforts may have contributed to important policy reforms after the compact.
The annual conference activities were sustained in the form of an annual TVET awards ceremony.

Evaluation Questions

This final performance evaluation was designed to answer the following questions:

1 What were the labor market outcomes (employment and wages) for graduates from Program Improvement Competitive Grants (PICG) supported courses?
2 What were employer perceptions of the graduates of PICG-supported courses?
3 Did TVET providers continue to offer PICG-supported courses post-compact?
4 Were activities which strengthened TVET policy and practice sustained after the compact?

Detailed Findings

The findings of the final evaluation report build on the interim evaluation report results published in 2019.

Labor Market Outcomes of Trainees in New TVET Courses

Since the end of the PICG course, 79 percent of trainees were ever employed in a paid job or self-employed and 35 percent were ever employed in a full-time job relevant to training. One year after the end of the course, 64 percent of trainees were employed in a paid job or self-employed and 29 percent were employed in a full-time job relevant to training.

The 51 courses established through the PICG component enrolled 1,935 trainees by the end of the compact, exceeding the compact’s target by about 30 percent. The evaluation focuses on trainees in 41 of these courses, including all 38 of the degree courses. Seventy-nine percent of trainees in these courses were employed at some point in the year after the end of the course. However, only 35 percent obtained a full-time job that they reported was related to their PICG-supported training course. Among the trainees who did not obtain a course relevant job, about one-half never searched for one due to a lack of interest or availability for work, or perceptions that wages would be too low. A lack of available job opportunities might have also been a constraint for some trainees, especially those with limited relevant work experience. Comparing male and female trainees in the same courses, job placement rates were similar by gender but wages for employed male trainees were about 32 percent higher.

For pandemic-affected cohorts in PICG courses, 65 percent of graduates were employed in a paid job or self-employed and 41 percent were employed in a full-time job relevant to training. For pandemic-affected cohorts in all courses, 50 percent of graduates were employed in a paid job or self-employed and 29 percent were employed in a full-time job relevant to training. — Note: The endline data collection period overlapped with the COVID-19 pandemic.

The evaluation’s benchmarking analyses suggest that PICG courses likely improved the job placement and earnings outcomes of trainees. Compared to a national benchmark of all TVET courses in Georgia, graduates of PICG-supported courses (1) maintained higher rates of employment during the pandemic, by 15 percentage points, and (2) earned wages that were 12 to 13 percent higher for male and female graduates, respectively. For nine existing courses enhanced by PICG grants, trainees during the compact appear to have earned wages that were 16 percent higher than earlier cohorts. Finally, trainees who were employed before enrolling in a PICG-supported course increased their earnings by 10 percent after the end of the course, and trainees who returned to the same job experienced a 21 percent wage boost.

Sustainability of New Courses

Picture of an aircraft hangar with open doors and the rear of a plane in Kakheti region owned by the Georgian Aviation University, a PICG grant recipient — An aircraft hangar in Kakheti region owned by the Georgian Aviation University, a PICG grant recipient

During the compact, PICG-supported TVET courses attracted almost $6 million of private industry co-investment in TVET provision, more than tripling the compact’s target, signaling high industry satisfaction. TVET providers continued to offer all of the PICG-supported courses two years after the end of the compact, and nearly half of TVET providers also launched new course offerings related to the PICG-supported courses. State voucher funding fully covers tuition at most PICG-supported courses, strongly supporting continued operations.

Sustainability of Activities to Strengthen TVET Policy and Practice

In several cases, the project’s small best-practice grants supported development of a new course or a new set of relationships between a grantee and other organizations that continued after the compact—an encouraging result given the limited size of these investments. However, several grantees cited the absence of additional external funding as a major factor limiting broader adoption of these practices.

Two areas of technical assistance progressed particularly strongly after the compact. First, the government’s new Skills Agency is fostering deeper public-private partnerships in the TVET sector in ways that build on the reform efforts supported by the project. Second, the quality assurance framework for TVET providers and courses that the compact helped develop has been directly codified into legislation, and the government’s authorization procedures for TVET providers and courses durably reflect these reforms.

Annual TVET conference activities have continued in the form of an annual TVET awards ceremony. Two rounds of the ceremony were held in 2020 and 2021 with close government involvement and public and private financial support, and a third round will be held in 2022. Stakeholders expect the event to continue in future years, and report that these continued events have the potential to contribute to improving perceptions of TVET in Georgia in a sustained fashion.

Economic Rate of Return

MCC considers a 10% economic rate of return (ERR) as the threshold to proceed with investment.

While the evaluator did not recalculate the ERR, they provided feedback on the validity of the ERR produced by MCC (20.9%) in light of the evaluation findings.

While MCC exceeded its end-of-compact enrollment target for trainees in PICG-supported courses by nearly 30%, COVID-19 disruptions meant total enrollment fell short of MCC’s projections through 2021. On the other hand, the evaluation found that trainees’ employment rates are similar to or higher than those assumed by MCC, and their wages exceeded what MCC assumed as well. As a result, updating the economic analysis to reflect data from the evaluation is unlikely to change the ERR estimate substantially. Evidence suggests that the project’s ERR remains strongly positive.

MCC Learning

PICG-supported TVET courses succeeded in strengthening linkages between TVET providers and large employers’ skills needs, thereby driving this project’s improved performance relative to past TVET programs.
Targeted support is needed to recruit and retain women in non-traditional TVET programs. Closing gender gaps in labor market outcomes will require interventions specifically designed to improve the country’s work environment for women.
Trainees who had prior experience and who used TVET programs to gain further skills enjoyed the largest improvement in labor market outcomes in terms of employment and wages. This demonstrates that in Georgia, employers regard TVET programs as a valuable way to upskill existing workers.
Flexible contracting structures like the one used for the policy technical assistance component can lead to support being nimble and responsive to government needs.

Evaluation Methods

This was a mixed-methods final performance evaluation. Quantitative data sources included trainee survey data, administrative data. Qualitative data sources were semi-structured interviews and high-level stakeholder interviews.

This mixed-methods final performance evaluation was conducted about two and a half years after the end of the compact, in order to capture longer-term labor market outcomes. It drew on a quantitative follow-up survey of trainees, qualitative interview data, and course-level administrative data.

Trainee Survey Data Between mid-2019 and mid-2021, a local data collection firm conducted a follow-up survey with 992 trainees enrolled in 41 PICG courses. The follow-up survey was conducted about one year after trainees were due to finish their course, and captured information about training experience, employment, and earnings.

Qualitative Data In late 2021, a local data collection firm conducted semi-structured interviews with all 10 TVET providers that received PICG funding, 8 entities that received smaller best-practice grants, and 24 employers of TVET graduates. The evaluation team also conducted high-level stakeholder interviews with Ministry of Education and Science officials, former project implementers, and international donors.

Administrative Data To help assess the sustainability of PICG-supported courses, the evaluator used course-level administrative data measuring the number of enrollees and graduates through the end of 2021. These data were collected regularly by MCA-Georgia and the Millennium Foundation (the post-compact successor entity to MCA-Georgia) from TVET providers that received grants.