STEM Higher Education Project

Project Summary

Feedback from more than 70 Georgian employers during compact development indicated high interest from the private sector in improved quality of higher education in Georgia. Highly trained professionals in the fields of computer science and civil, electrical, and mechanical engineering were in high demand and short supply in Georgia. This shortage deprived businesses of the ability to employ updated technology and business practices needed to innovate and increase their productivity, forcing them to hire foreigners to fill these jobs in some cases. MCC and the GoG conducted a wage survey of over 50 Georgian businesses, and found that employers would pay 44 percent more to a U.S.-educated engineer than to the “best” Georgian-educated engineer.

While access to higher education was widespread in Georgia, institutions with STEM programs were not of sufficient quality to provide the skilled graduates needed by industry. In particular, there were two factors preventing the establishment of quality STEM programs in Georgia: 1) the outdated knowledge and approach of faculty educated largely under the Soviet system; and 2) poor quality laboratory facilities and equipment, due to the substantial cost necessary to establish a modern STEM program. Women’s self-selection into non-STEM fields, with women comprising only 27 percent of engineering students in 2009, posed a major gender equity challenge in higher education.

To address this human capital binding constraint to growth, the STEM Higher Education Project was designed to support the delivery of high-quality STEM bachelor’s degree programs in Georgia by: 1) bringing a U.S. university to Georgia to partner with Georgian public universities to offer U.S. bachelor’s degree programs in the STEM academic disciplines; and 2) providing capacity enhancement for Georgian public universities with the goal of reaching international standards and acquiring international program accreditation.

San Diego State University (SDSU) was selected through a competitive process to offer U.S. accredited degrees in Tbilisi, Georgia. In the fall semester of 2015, SDSU began offering bachelor’s degrees in STEM fields, in partnership with three public Georgian universities: Tbilisi State University; Georgian Technical University; and Ilia State University.

In the long run, the STEM Higher Education Project was expected to improve the productivity of firms, reduce reliance on foreign graduates, and reduce the need for Georgian students to study abroad to obtain an adequate education. Students benefiting from the project were expected to have better employment opportunities and higher incomes. The graduates of SDSU in Georgia and accredited partner university programs would gain the skills demanded by industry and go on to earn higher wages, contributing to Georgia’s economic growth by building the pipeline of skilled higher education graduates.

In 2014, the GoG and SDSU signed a twenty-year memorandum of understanding. During the compact term, SDSU offered six U.S. Bachelor of Science programs in chemistry/biochemistry, computer science, electrical engineering, computer engineering, civil engineering, and construction engineering in Georgia. More than 600 students enrolled in these programs over the life of the compact, and the first 55 graduates received their degrees in June 2019. Students receive both an SDSU and a partner university degree upon graduation. With their U.S. degrees in hand, graduates of SDSU in Georgia are expected to have better employment opportunities and higher incomes.

The project invested in facility upgrades at the public partner universities, including:

Rehabilitation of nearly 5,000 square meters of science and engineering laboratories and classrooms. These laboratories are very similar to those at SDSU’s home campus in San Diego, are the only labs of their kind in the country and are providing the students enrolled at SDSU in Georgia with opportunities to advance their skills in a high-demand technical field.
Construction of a new four-story building on the campus of Ilia State University to house additional engineering laboratories and serve as a central hub for the three partner universities.
Installation of world-class equipment at all three partner universities.

The classrooms, laboratories, and equipment provide the infrastructure required for partner universities to apply for and obtain international accreditation for several degree programs.

The project also invested significantly in Georgian partner university faculty development, contributing to the objective of acquiring international accreditation or certification for partner university programs. Eighty-six Georgian faculty received training in the U.S., emphasizing how to use and deliver SDSU’s curricula with compact-provided state-of-the-art laboratory equipment. Faculty training is expected to continue through July 2023.

The STEM Higher Education Project was successful at recruiting female and socially vulnerable students into the SDSU-G STEM majors. Targeted recruitment for this program resulted in a student body that was 36 percent female, an impressive achievement even by American STEM higher education standards. In 2018, only 26 percent of students enrolled in STEM degree programs at SDSU’s home campus in California were female.

SDSU in Georgia worked to empower female students in various ways. They created a Women’s Empowerment Club, which provides peer mentoring and support to female STEM students through mentorship by female faculty. Through the compact, SDSU in Georgia also created a career development center and held internship and career fairs to introduce students to employers in STEM fields. These events prepared female students in particular for their eventual job search process while challenging stereotypes by introducing potential employers to qualified female STEM students. The career development center also provided access to internship and other professional development opportunities, which was particularly valuable for students from the regions outside of Tbilisi who may not have had access to this information otherwise.

The STEM Higher Education Project was also successful in recruiting those students who would not otherwise have been able to afford a U.S. university degree. Generally, students interested in obtaining an American education and with the ability to pay still went to U.S.-based universities rather than enroll at SDSU in Georgia. As a result, the highly qualified students who were interested in enrolling at SDSU in Georgia largely came from families that lacked the means to afford the $7,500 annual tuition. Eighty-five percent of all Georgian students enrolled at SDSU-Georgia during the compact term came from families whose mean income was less than 40,000 Georgian lari, or under $17,000, and 30 percent of students came from outside Tbilisi.[[San Diego State University Strategic Growth Plan, September 2017.]] The project was only able to successfully enroll these students because of significant financial aid packages made available through the GoG and private sector contributions, provided through SDSU in Georgia’s public-private partnership fund. Given this student body makeup, the upward mobility impacts of the project on graduates and their families are potentially quite large.

During compact development, MCC, the GoG, and SDSU envisioned that this partnership would continue well beyond the life of the compact, with SDSU continuing to offer U.S. degrees to Georgian students for at least 20 years from the start of program delivery. That vision, however, was premised on certain financial assumptions that did not prove accurate. In particular, the number of students enrolling and the ability of these students and their families to pay full tuition was significantly lower than estimated, and budget gaps ensued due to lower revenues. SDSU, MCC, and the GoG worked collaboratively over the life of the compact to successfully cover these budget gaps, including through: 1) GoG co-financing for student scholarships, student support services, and research grants; 2) compact reallocations (described in the Compact Changes section of this report); and 3) the mobilization of more than $3.5 million in private sector funds for student scholarships through a public private partnership fund.

The project envisioned that SDSU would initially deliver U.S. degrees, but that they would later transition to a support role as Georgian universities acquired and maintained international accreditation and certification, allowing them to eventually deliver internationally recognized degrees on their own. While this transition was originally planned to occur after twenty years, priorities of both SDSU and the GoG shifted once the shortcomings of the financial assumptions came to light. Given the higher-than-expected financial risk of the partnership, SDSU and the GoG agreed to accelerate the transition process, with SDSU delivering degrees directly to five cohorts of students (through 2023), after which partner university programs are expected to have achieved accreditation and to assume responsibility for direct degree delivery.

Georgian universities are delivering the following U.S. degree program:[[U.S. degree program enrollment in Computer Science and Construction Engineering concluded in September 2018. U.S. degree program enrollment in the remaining four disciplines concluded in September 2019.]]

Tbilisi State University: Chemistry (Biochemistry Focus), Computer Engineering, Electrical Engineering, and Computer Science
Ilia State University: Computer Engineering and Electrical Engineering
Georgian Technical University: Computer Engineering, Electrical Engineering, Civil Engineering, and Construction Engineering

Project Sustainability

International accreditation of partner university degree programs is critical to project sustainability, and SDSU considered accreditation equivalent to having a U.S. degree in the global labor market. As part of the project, SDSU assisted the three partner universities in meeting the requirements for the Accreditation Board for Engineering and Technology (ABET) and American Chemical Society (ACS), and MCA-Georgia contracted the ABET Foundation to deliver additional technical assistance to partner university accreditation efforts. Two programs at TSU received ABET accreditation retroactive to 2017 and 2018. Accreditation offers benefits similar to those guaranteed by a U.S. degree by providing external validation of program quality and ensuring programs incorporate industry needs, thereby enhancing employability of graduates and driving quality enhancement across the university. The project has placed significant emphasis on the partner universities developing their own ABET/ACS accredited programs. Nonetheless, it will take sustained government support and coordination with the partner universities to continue the program.

With internationally accredited programs, the project aimed to allow the partner universities to continue to produce a pipeline of world-class engineers and scientists and grow the Georgian economy long after the compact ended. Each of the project’s investments—from building state-of-the-art infrastructure, training faculty, and introducing continuous improvement in program delivery—supports this sustainability objective. These internationally accredited programs, which are expected to require students to pay somewhat higher tuition fees than the average Georgian university tuition, may also be an important revenue stream for sustaining program quality.

During the compact term, SDSU’s three partner universities made progress towards receiving international accreditation, with the first two-degree programs from Tbilisi State University (Computer Science and Electrical Engineering) receiving ABET accreditation in October 2020. These programs are the first STEM programs in Georgia and the larger region to receive this global stamp of quality. Ilia State University and Georgian Technical University are on track to obtain accreditation for several degree programs in 2022 and 2023. Tbilisi State University’s chemistry/biochemistry program is on track to receive ACS certification by 2021.

To ensure the sustainability of the project’s infrastructure and equipment investments, SDSU signed partnership agreements with each of the three partner universities, which include plans for facilities-sharing and operations and maintenance through 2023.

Since the compact ended, SDSU has maintained a presence in Georgia and continues to assist all three partner universities in obtaining internationally recognized accreditation and certification. The successor entity to MCA-Georgia will also continue to assist SDSU and partner universities with the international accreditation process. SDSU’s final cohort of U.S. degree students will graduate in spring 2023. MCC, SDSU, and the GoG are exploring opportunities for a longer-term SDSU engagement in Georgia, including potentially through summer programs, executive education, continued accreditation support, and/or other collaborative opportunities.

Economic Analysis

Original Compact Project Amount: $30 million
Total Disbursed: $36.1 million
Estimated benefits at compact closure corresponds to $36.1 million of MCC project funds, $8.4 million of country contribution project funds, and administrative and M&E costs, as included in the related cost-benefit analysis.

	Estimated Economic Rate of Return over 20 years	Estimated beneficiaries over 20 years[[For economic analysis, direct beneficiaries are students who graduate from with project schools and become employed, resulting in increased incomes. Total beneficiaries, which are reported in the table, are calculated by multiplying the direct beneficiaries by the average household size in Georgia (3.4). ]]	Estimated net benefits over 20 years
At the time of investment decision	10%	31,000	$3.5 million
Updated at entry into force	11%	47,000	$19.8 million
Updated at compact closure	9.0%	16,235	$-9 million

The original ERRs were estimated for an investment decision by MCC on April 3, 2013, and revised after the compact entered into force. The updated ERR for the STEM Higher Education Project increased from 10 to 11 percent to reflect several assumptions including a combined annual student intake of 495 in the first year of the program, rising to 610 by year five, and an average annual operating cost intended to be covered by tuition of $7,500 per student. The economic model also assumed a 44 percent wage differential attributed to a Georgian student who graduates with a U.S. degree in a STEM field, compared to a Georgian student who graduates with a non-U.S. degree from a program in the same STEM field that is not internationally accredited (for students who earn an ABET-accredited degree from a Georgian partner university, the assumed wage differential is 22 percent). This wage differential is the key benefit of the CBA model and was based on a wage survey of more than 50 Georgian companies in fall 2012.

The economic logic underlying the STEM Higher Education Project CBA model remained consistent in calculating the project-level closeout ERR. The major change in the closeout CBA model—a reduction of the ERR from 11 percent at EIF to 9 percent—is due to lower-than-expected enrollment, particularly for U.S. degree programs, that drastically increased per-student operating costs and reduced the number of project beneficiaries. Whereas the EIF CBA model assumed an initial cohort size of 495 students, which would increase to 610 by year five, total first-year enrollment in U.S. degree programs (for five cohorts) was only 780 students (from a minimum of 86 students enrolled across all programs in the first cohort to a maximum of 230 students in the fourth cohort in the final year of the compact). However, the decline in the closeout ERR due to low U.S. degree program enrollment is mitigated by an accelerated transition of SDSU to a support role for Georgian partner universities. The lower ERR at compact closure also incorporates the probability of employment in the CBA model and beneficiary count, which reflects a change from previous analysis.

A secondary benefit stream of the STEM Higher Education Project is savings to the Georgian economy from reduced imports of highly educated and more expensive expatriate STEM professionals. Due to skills shortages in Georgia, some high-skill positions are currently occupied by expatriate workers. Due to low enrollment in U.S. degree programs, however, these benefits contributed only 4 percent to the closeout ERR.

While small due to low enrollment in U.S. degree programs, additional benefit streams include (i) increased “productivity spillovers,” where having more well-educated STEM professionals in the workplace increases earnings and wages of other workers; and (ii) cost savings for students who, without access to the U.S. degree programs offered in Georgia, would have pursued undergraduate STEM degrees at U.S. or European universities. Because of the large increase in per-student operating costs for U.S. degree programs, the “cost savings” benefit stream negatively affects the compact closure ERR.

Evaluation Findings

The independent evaluation utilizes a mixed-methods approach that involves a quantitative analysis of surveys along with document and literature reviews and qualitative analyses of interviews, focus groups and case studies. Such mixed-methods evaluation will allow us to examine a variety of aspects of the STEM Higher Education Project including partnerships, design and implementation, outcomes, barriers to and facilitators of high-quality implementation, cost, and sustainability.

The study utilizes student and alumni surveys as a quantitative approach to examine such project outcomes as student perceptions of the SDSU-Georgia degree programs, employment during university studies and after graduation, and wages after their degree is completed. The evaluation will examine how the outcomes of interest compare between SDSU-Georgia students and other students who attended and completed degrees in similar disciplines at different university programs.

Evaluation Questions:

How was the partnership established and carried out? How did it change over time?
Were the activities implemented through the project aligned with the program design, as documented in the logic model?
How do SDSU and comparison group students view their programs? To what extent are the project activities sustainable?

Key Findings from the interim report’s Evaluation Brief include:

University Partnerships

San Diego State University (SDSU) led the development of partnerships with three Georgian public universities: Georgian Technical University, Ilia State, and Tbilisi State.

Project Implementation

Georgians found tuition too expensive but MCA-Georgia, MCC, and GoG collaborated to generate scholarship funds to increase demand for the program.

Student Perceptions

Surveys show that students in the program were very satisfied with new facilities and equipment but less so with their experience with faculty and courses offered.

Sustainability

Partners have taken steps to sustain STEM programs and relationships after the compact ended in July 2019 through program accreditation and other activities.
Readiness for international program accreditation and certification varies across the partner universities. Once accredited or certified, the tracks at Georgian Technical University, Ilia State University, and Tbilisi State University will be the first STEM programs in Georgia and the larger region to receive this global stamp of quality.

Component	Status
Interim	Data collection completed in 2019. Interim Report published in January 2021.
End-line	Data collection to be completed mid-2023. Report expected late 2023.

Lessons from the evaluation:

MCC should work to obtain a better understanding of students’ ability and willingness to pay for higher education. MCC should have a clear picture of what student expectations are for higher education, and, if a partner government is proposing something new/innovative that neither they nor MCC have undertaken in the past, make sure there is a market for it.
Sharing potential student enrollment numbers from MCC’s initial Cost-Benefit Analysis as part of the initial Request for Proposals gave bidders the incentive to overestimate their ability to recruit and retain students.
New education programs require ramp up periods for outreach and to build their image or “brand.” MCC should plan for an initial subsidy period, with mechanisms in place (for example, marketing of success stories and job placements) that reduce the risk that the subsidy phase-out period is insufficient to generate demand at cost. In addition, MCC should have in place proper mechanisms that allows the program to improve based on feedback from students.
Creating a partnership between a U.S.-based university and public Georgian universities was more complicated than any of the parties anticipated. New partnerships should be structured to be flexible to allow for adaptation by all stakeholders, with certain “non-negotiables” clearly identified from the outset, e.g., there was a push in the third year of the compact to recruit international students that took away valuable time and human resources away from other program activities, as the GoG sought to make the SDSU program in Georgia part of a Georgian “hub” to attract international students.
MCC should agree on site identification with partner governments prior to committing compact resources to infrastructure rehabilitation. The partner universities identified about 20 locations across Tbilisi for science labs and classrooms, but many were located far apart from each other and it was hard for students to travel between sites and arrive on time for class. The dispersed locations of renovated spaces across three partner universities also made MCC’s infrastructure and equipment investments less visible to the public.
Getting responses from comparison group students at Georgian partner universities was initially very difficult. Survey planning should consider the lack of incentives for respondents to answer surveys when they are not benefiting from the compact-funded program. Especially when surveying groups that were not targeted for the program. It is increasingly important to plan adequate time and effort for building relationships and troubleshooting problematic respondent groups.

Key Output and Outcome Indicators

Outcome: Increased quality of tertiary education in STEM fields in Georgia. Improved employment opportunities and higher income for graduates of the improved programs.

Key Performance Indicators	Baseline	End of Compact Target	Q1-Q20 Actuals (as of July 2019)	Percent Target Satisfied
Students participating in MCC-supported education activities	0	2,008	642	32%
ABET Committee Formed	N/A	No Target	April 27, 2016	N/A
Retention Rate of Students in MCC-funded Bachelor's programs	0	80%	77.2%	96.5%

Explanation of Results

The STEM Higher Education Project did not meet its original target for student participation. MCC, SDSU, and MCA-Georgia determined early on in implementation that the willingness or ability of Georgian students and their families to pay full tuition at SDSU in Georgia was significantly lower than originally estimated by all parties involved, resulting in a need for additional scholarship financing and lower than expected student enrollment. While enrollment in SDSU-Georgia increased from cohorts one through four, the amount of family tuition pay has been lower than expected, and in the first two cohorts more than 75 percent of tuition was paid through scholarships provided by MCC/MCA-Georgia or the GoG. This gap led to an annual budget deficit that the GoG was able to cover but had to be re-visited each year, including a contribution of $11.2 million to sustain the program through at least five cohorts (through June 2023). This deficit ultimately led to a cap on student enrollment for cohorts four and five.

SDSU has projected eight percent student attrition annually through graduation of the final class of U.S. degree students in spring 2023. Out of the seniors that graduated in May 2019 after four years of SDSU education in Georgia, 20 percent graduated summa cum laude. After the unexpectedly low enrollment of the first cohort, SDSU and MCA intensified recruitment efforts, and at the same time expanded the pool of candidates by offering a “bridge” or preparatory program, whereby students could enroll in the SDSU degree program, take required additional math and English courses in the fall semester, and then enroll in the full program in the freshman spring semester. Although this increased the number of enrolled students, it also increased the level of attrition, as many students who had been accepted into the “bridge” program were not able to keep up in subsequent years. Others did not, or will not, complete the program in four years, partially due to the Georgian regulations which allow students to take a year or two off without losing their student status, and then return to their same Georgian university to complete their degree. Besides attrition, this also affects the overall graduation rate.