Stemming Learning Loss During the Pandemic: A Rapid Randomized Trial of a Low-Tech Intervention in Botswana

"To address the learning crisis, which has been exacerbated by the COVID-19 pandemic, approaches that can cost-effectively improve learning on a global scale are needed."

The COVID-19 outbreak caused education systems around the world to halt in-person learning. In Botswana, schools across the country were closed from March to June 2020; shortly after reopening, a new wave of COVID-19 cases prompted a subsequent school closure, with similar waves of reopening and closing expected, and a double-shift system in place as of July 2020. Despite the government's efforts to keep students engaged through learning programmes on national television and radio stations, families have expressed demands for additional remote educational activities for their children. However, in Botswana as elsewhere, the pandemic has laid bare existing educational inequalities, with high-income families having access to alternative sources of instruction (e.g., computers and smartphones) that are out of reach of many low-income families. In partnership with the non-governmental organisation (NGO) Young 1ove and the Ministry of Basic Education, researchers conducted a randomised evaluation to assess the effectiveness of a low-tech intervention on improving students' learning during school closure.

Prior to the pandemic, in an effort to improve learning levels in Botswana, a coalition of partners led by the Ministry of Basic Education had begun supporting the scale-up of an education programme called Teaching at the Right Level (TaRL) to all primary schools. This programme, which has been found to be effective in several contexts, involves regrouping students by their actual learning level rather than the grade-level curricula. At the time of COVID-19 school closures, TaRL had reached over 15% of primary schools in the country.

For the intervention, using phone numbers collected from primary schools for students in grades 3-5, the research team identified 4,550 households who were willing to receive remote learning support via mobile phone. Each household was randomly assigned to one of these groups:

1. Households received a weekly SMS (short messaging service, or text) message containing several simple math problems ("problem of the week" - e.g., "Sunshine has 23 sweets. She goes to the shops to buy 2 more. How many does she have altogether?").
2. In addition to the SMS messages, households in this group also received a 15- to 20-minute phone call in which "facilitators" (Young 1ove staff), answered any questions related to the "problem of the week" and provided additional practice questions. The calls required both parents and children to be present, as a way to encourage engagement and maintain accountability.
3. Comparison group: Households in this group did not receive any type of phone learning support.

In short: Both the SMS-only and SMS + phone calls interventions had positive impacts on students' learning, particularly for lower-performing students. The programmes also led to higher parental engagement in educational activities and improved the accuracy of parents' perceptions of their child's learning level.

More specifically, after the 4 weeks, students in households that received weekly text messages with math problems showed a 13% improvement on learning outcomes (a 0.16 standard deviation increase in scores from an average of 1.73 in the comparison group). Meanwhile, students who received additional support through phone calls experienced a 24% increase in their learning outcomes (a 0.29 standard deviation increase). This translates into a reduction in innumeracy of 34% among the SMS-only group, and 52% for the SMS + phone calls group (19% and 14% of students were innumerate in the 2 groups after the evaluation, respectively, compared to 29% in the comparison group).

The programmes were also found to be effective in closing previous learning gaps. When comparing students who had previously received learning support in math, measured by participation in the TaRL programme with those who had not, researchers found that the second group experienced higher gains in learning from the remote phone interventions after 4 weeks. A similar trend was observed when comparing girls and boys: Boys, who started with lower numeracy levels, benefitted more from the SMS + phone support than girls. These findings indicate that targeting the programme to students who need it the most might be important to enable its scale-up.

In households that received SMS-only and SMS + phone calls, parents were 7 and 12 percentage points more likely, respectively, to spend time engaging with their child on educational activities after 4 weeks. This represents a 10%-18% increase in parental engagement, from 67% in the comparison group.

Accurate perceptions of a child's learning level are thought to enable families to target learning activities to their child's level more effectively. Parents in the SMS-only group were 7 percentage points more likely to correctly estimate their child's learning level, while those in the SMS + phone calls group improved their perception by 11 percentage points: a 23% and 35% increase from a comparison group average of 31%, respectively. The results show, however, how hard it can be for parents to know the "right level" of their children's learning, which emphasises the need for targeted instruction.

Demand for the programme was very high: 99% of parents wanted to continue the programme after the first 4 weeks.

The researchers suggest that implementation and deployment of the programme is feasible to deploy relatively quickly. Within 6 weeks, they conceived and rolled out the programme to over half of the regions of Botswana. (A caveat is that they had a team of over 60 facilitators who could be readily deployed to make phone calls.)

Furthermore, the researchers deemed both interventions to be relatively cost-effective and easy to scale. For example, both rely on phones and do not require internet access. One way SMS text messages are feasible to implement at large scale in most countries using bulk texting platforms. The results suggest that attempting to reach low-performing students yields the highest returns; thus, direct phone calls at scale might consist of weekly teacher phone calls to the bottom 5% to 10% of their class.

Efforts to tailor the programme are underway. After week 4, researchers randomly selected a subset of the households from the SMS only and SMS + phone groups to receive an additional support programme: tailored instruction through customised text messages. In this subsequent intervention, which is ongoing, students receive text messages tailored to their individual learning levels over a period of 4 weeks. For example, students who know addition receive subtraction problems to push them to a higher level of learning, whereas students who know multiplication are sent division problems. Through weekly phone surveys, facilitators collect students' answers to the "problem of the week", which is used to target subsequent weekly SMS messages. (The results of the survey of this additional intervention are forthcoming.)

The results to date have been incorporated directly into day-to-day programming by Young 1ove, and the Ministry of Basic Education in Botswana is exploring a national scale-up. In addition:

• The research team is exploring replication trials in 3-5 countries, including with NGOs, governments, and multilateral institutions such as the World Bank.
• More advanced low-tech interventions could be tested, such as 2-way texts and interactive voice response (IVR).
• Follow-up research could help disentangle the mechanism behind learning gains found in the study.
• The researchers plan to explore the implications of these low-tech interventions as substitutes or complements of the traditional schooling system as schools open.
• The results and lessons learned from this investigation could have implications beyond the COVID-19 experience, as school closures occur during annual school holidays, other public health crises, natural disasters, and due to weather-related shocks.

As two of the paper's authors write in a related blog, "The Botswana experience, both before COVID-19 and during, has an important lesson: Meet students where they are and focus relentlessly on the foundations."