Upgrading Vocational Education Equipment in Mongolia

Program Overview

MCC’s 284.9 million Mongolia Compact (2008—2013) funded the $49.3 million Vocational Education Project, which included the $21.8 million Improvement of Learning Environments Activity. This activity aimed to increase employment and income among unemployed and marginally employed Mongolians. The activity was built on the theory that providing modern equipment to vocational schools would improve the quality and relevance of the technical and vocational education system, thus leading to improvements in individuals’ employment and increased incomes.

Key Findings

Evaluation Questions

This final impact evaluation was designed to answer the following question:

1. 1
   What is the effect of learning and training with upgraded equipment on graduates’ employability and wages?

Detailed Findings

Implementation

Banks and their clients reported an improved customer experience because of the reduced waiting time and faster service. All rural banks indicated the average transaction time lowered from 15 or 20 minutes at baseline to 5 minutes or less. Some banks also reported a reduction in check clearance time from up to 5 days to 1 day. Waiting time only exceeded these times at the end of the month when many people collect salaries or pay school fees. Most rural banks agreed that the i-transfer system (the inter-rural-bank transfer of funds), was a major upgrade to the previous telephone system.

The Improvement of Learning Environments Activity provided equipment to 18 schools across 15 provinces in Mongolia. The types of equipment provided included heavy machinery, lathe-milling, electricity and electronics, plumbing, welding, heating and cooling technology, hydraulics and pneumatics, concrete and reinforcement, and auto mechanical training. With a few short delays, all equipment was delivered and installed as planned.

Impacts on Employment & Income

No significant impacts were found on measures of employment or income as a result of increased time spent learning with the improved equipment. The 2011 cohort (15.8 months of exposure) had increased employment and income compared to the 2010 cohort (6.8 months of exposure), but the results were not statistically significant. The 2012 cohort’s (24.4 months of exposure) employment and income measures were worse than both earlier cohorts. There were no differences by gender. Note that the top graph on the right is essentially a summary of the bottom two. It shows the differences between individuals admitted to an upgrade trade versus those that were not, across cohorts.

Similarly, no significant impacts were found on intermediate results, such as trade-related knowledge, work intensity or employment type. Exposure to improved equipment also did not have an impact on other higher-level outcomes, such as future expectations, assets and expenditures.

More accurate bank statements improved the customer experience, and bank clients who were interviewed confirmed this finding. In interviews with key management at select rural banks, the majority agreed that customers have overall gained greater confidence in dealing with them, and their reputation has greatly improved compared with the pre-project period.

Results show a positive effect on the value of accounts and average balances per customer (change in the value of deposit accounts for baseline and endline depicted in figure), which is consistent with improvements in the customer experience. The evaluation also found a positive effect on bank net income; however, bank expenses per customer have not changed.

The project had a strategic impact in convincing the rural banks to embrace information technology and its implications and generated a growing appetite for learning and adopting modern technology for their operating needs. After the intervention, rural banks expressed interest in more sophisticated and diversified IT products such as ATMs, internet banking, and integration with mobile money, which assure seamless and uninterrupted services.

Possible Explanations

The lack of impacts may result from differing labor market opportunities each year. Labor market conditions for those who studied trades that received the upgraded equipment could have deteriorated relative to those who studied other trades. There is some evidence of a general deterioration of labor market opportunities for individuals admitted to technical and vocational schools in the 2012 cohort.

Another hypothesis is that individuals’ exposure to upgraded equipment did not improve skills that employers valued. There were no significant increases in scores on tests measuring trade-related knowledge (although it is possible the tests did not capture all skills learned from upgraded, compared to older, equipment). There is strong evidence that schools received equipment upgrades, but there is no data on whether teachers used equipment upgrades effectively or whether individuals learned skills that were useful to employers.

MCC Learning

• 
  Understand the link between equipment and expected outcomes as improving employment outcomes and household incomes may require interventions beyond providing technical equipment alone.
• 
  Fully define the theory of change to ensure intermediate outcomes are articulated and measured, which increases the learning that it is possible to obtain from the evaluation.
• 
  Include a detailed problem diagnosis as a precursor to effective program design. MCC now uses better problem diagnosis processes, including the constraints analysis, which identifies the binding constraints to economic growth, and detailed root cause analysis. In the case of the Technical and Vocational Education and Training (TVET) sector, a root cause analysis should include a comprehensive institutional assessment, a credible approach to identifying skills gaps in the labor market and an understanding of the current/potential role of the private sector in TVET provision.

Evaluation Methods

The impact evaluation used a difference-in-differences quasi-experimental methodology to estimate the impact of exposure to upgraded equipment on students’ employment and earnings. The evaluation took advantage of the gradual rollout of equipment upgrades to TVET schools over time, exploiting the variation in exposure to equipment upgrades across the 2010, 2011 and 2012 cohorts. The exposure period varied from 6.8 to 24.4 months.

Three types of surveys were administered to individuals: an admissions survey (12,011 individuals) collecting baseline demographics data and an aptitude test, a phone-based tracking survey (18,087 total student surveys) each following year to update contact information and collect basic information on education and job-related achievements, and an in-person graduate follow-up survey (10,950 individuals). Individuals completed the graduate follow-up survey in the year after their expected graduation. The follow-up survey included all tracking survey information, further questions about asset ownership, consumption, expenditure and other household activities, and a written trade-based skills test. The skills tests contained technical questions specific to the trade individuals studied and general questions on numeracy and computer literacy.

In addition to the student surveys, administrative surveys were conducted with teachers and administrators at up to 50 TVET institutions over three years to capture characteristics, such as school size, funding, and the availability and use of equipment.

2021-002-2619