Data Collection in the Time of COVID-19

The $268 million Connectivity Project in the MCC – El Salvador Compact built, improved, or rehabilitated 23 bridges and 223 kilometers of road in El Salvador.

MCC’s model is based on a set of core principles essential for effective development assistance — good governance, country ownership, focus on results, and transparency. MCC’s focus on results means that the agency produces evidence on the effectiveness of every MCC project through an independent evaluation. Given the significant amount of time and resources spent on evaluations, especially data collection, the agency is constantly looking for innovative ways to measure results in a more cost-effective manner.

In 2020, MCC’s drive to innovate took on increased urgency. The COVID-19 pandemic ground the world to a halt, including many of MCC’s data collection efforts for its independent evaluations. We faced a challenge we had never faced before — deciding how to move forward with data collection in a manner that was safe yet still timely.

In El Salvador, the pandemic pushed us and our evaluators to assess the trade-offs in shifting from traditional data collection methods to a new approach to evaluate road conditions. As a result, we tested and successfully applied more cost-effective methods that will be adopted for future road evaluations.

From 2007 to 2012, MCC’s $449.6 million El Salvador Compact funded the $268 million Connectivity Project that built, improved, or rehabilitated 23 bridges and 223 kilometers of road. The project was based on the theory that improving road quality would lead to reduced travel costs and decreased travel time, leading to increased use of the road. Increased usage of the road would then enable communities to participate more fully in the national and regional economy. MCC’s independent evaluation of the Connectivity Project was published in 2022 and found that 8 years after construction, the road was still high-quality and had significantly reduced travel time. However, road usage had not met projections and the full potential of MCC’s investment had not yet been realized.

So how was MCC able to complete the independent evaluation’s final round of data collection in 2021 during a global pandemic?

Changing lanes

In alignment with MCC’s best practices for road evaluation, MCC’s independent evaluator set out to measure any road deterioration and road maintenance since the MCC-funded improvements of the El Salvador Northern Transnational Highway. The evaluator originally planned to measure road roughness with the conventional method that highway agencies use. This method uses lasers to provide precise data on surface roughness, but requires expensive equipment, a specialist to operate it, and would need to be imported into El Salvador. In addition to road roughness, the evaluator was planning to send international and local staff to conduct traffic counts along the improved road sections.

As the evaluator was preparing to conduct the road condition and traffic surveys, the COVID-19 pandemic triggered travel restrictions, which unexpectedly halted all conventional data collection. The original evaluation data collection plan, which involved international travel to provide training and oversight, was no longer possible. MCC and the evaluator discussed either adjusting the data collection plan or postponing data collection indefinitely. The evaluator proposed alternative data collection methods using mobile phone apps that use the phone’s built-in sensors, dash cameras, and other less precise, but more practical instruments. Local staff can implement these options without specialized expertise. The tools provide a relatively small trade-off in data accuracy, but , at the time, they had only been piloted and not yet used at scale by MCC. Rather than postponing indefinitely, our team knew it was the right time to test these new data collection methods.

Test driving

One of our key considerations when contemplated how to pivot the data collection methods was to ensure that we still had sufficiently accurate data. While collecting extremely precise data may be appropriate for highway agencies conducting road maintenance, the evaluator reported that with adequate testing and remote oversight, the reduced precision of the lower-cost instruments was still suitable for the evaluation’s purposes.

Evaluators analyzed dash camera footage to collect traffic data.

For roughness measures, the evaluator did not simply replace the laser equipment (an extremely precise instrument) with a mobile phone app (a less precise instrument). To account for the downgrade and the fact that this was new and untested technology, the evaluator collected the same outcome measure — road roughness — through three different sources: a mobile phone app, a visual scale, and a prototype road assessment tool.

The use of multiple data sources enabled the evaluator to increase confidence in the estimates. For traffic counts, evaluator staff analyzed the video from the dash camera footage to estimate traffic volume and composition. Those video traffic counts were compared to traffic projections made by extrapolating traffic data from the Ministry of Public Works. The final traffic counts were not as precise as a traffic counting machine would be over a week-long period. However, they provided sufficient confidence in the traffic projections to use them in the economic model.

Kicking the tires

As we tailored the data collection methods to the evaluation needs, we also knew data collection must be set up to ensure these new technologies would collect data of sufficient quality despite remote oversight. MCC’s independent evaluator, Jacobs and Transport Research Laboratory, provided support from the United Kingdom, while their local partner, AS Solutions and 3ipe, implemented these methods on the ground. The evaluator put significant effort into testing and calibrating the equipment and remotely training the local team on the iterative data collection protocols.

Step by step illustration of the data collection process.

In the first phase, data was collected along highway sections and uploaded for the evaluator to conduct a remote quality check and provide feedback, which resulted in minor corrections. In the second phase, the team returned to specific road sections, where the data showed specific issues, to collect additional data to support analysis.

To successfully manage the adoption of the mobile phone apps, the evaluator divided the data collection process into these steps, with well-defined objectives, so we could discuss the progress at each milestone before agreeing to move on to the next step. If the evaluator had encountered major challenges or if the methods had proven to be unreliable, we could have course-corrected in time or decided not to move forward. Much of the success of this data collection effort was due to the frequent communication between the international evaluation team and the high-capacity local team.

Less than sticker price

As a result of the changes to the data collection plan, MCC was able to reduce the data collection budget for the evaluation by 65 percent. In addition, we realized that the low-cost technology and more portable equipment could provide greater flexibility to collect more rounds of data over time for other MCC projects. Collecting data could be as simple as mounting the mobile phone and dashcam on a vehicle each time a team member went on a site visit. The roughness measure phone apps and dashcam video footage could provide a record of what the road looked like before the improvements, track progress on which sections were upgraded, and ultimately show post-construction roughness and traffic levels.

In conclusion, the external restrictions that stemmed from COVID-19 compelled us to think outside the box, streamline data collection to align with the evaluation needs, and test new cost-efficient technologies that can now be used on other road evaluations. The willingness to consider alternatives and the learning-by-doing approach undertaken in El Salvador resulted in the successful implementation of an evaluation that could have otherwise been postponed indefinitely due to the restrictions that the pandemic imposed.