Diminishing Marginal Returns to Computer-Assisted Learning
GAICIL Insights
Every education technology pitch carries an unspoken assumption: if some is good, more is better.1 Diminishing Marginal Returns to Computer-Assisted Learning, by Eric Bettinger, Robert Fairlie, Anastasia Kapuza, Elena Kardanova, Prashant Loyalka and Andrey Zakharov, tests that assumption and finds it wanting. Educational technology had been expanding for years, and the pandemic forced a full-scale substitution of online for in-person learning, making one question urgent: how substitutable are in-person and online learning? The authors run a novel randomized controlled trial that varies the dosage of computer-assisted learning as a substitute for traditional learning through homework.
Dosage is the paper's signature. Previous evidence, even from randomized trials, was decidedly mixed and focused on a single dosage level. Here a field experiment involving approximately six thousand Russian schoolchildren and three treatment arms generates exogenous variation in the intensity of computer-assisted learning. It illuminates substitutability between computer-assisted and traditional learning, and more generally between inputs in the educational production function.
The findings trace a curve familiar to any economist and ignored by most marketers. Moving from zero to a low level of computer-assisted learning, the authors find positive substitutability for traditional learning. Moving from a lower to a higher level, substitutability is either neutral or even negative. Computer-assisted learning raises math test scores at both base and higher dosage, yet additional dosage adds little or may displace more valuable traditional learning. The estimates suggest a blended approach is optimal.
Read that against the AI tutor boom. AI tutors and adaptive learning systems are the next generation of computer-assisted learning, and adoption is rapid. The paper's evidence reframes the debate as a question of dosage. Some AI-assisted learning can substitute productively for traditional homework; beyond a point, additional technology-based learning may offer little benefit or crowd out more effective activities. The right question for a school board is how many minutes.
Schools should treat AI tutors the way clinicians treat medication: prescribe, measure, adjust. Enthusiasm should be matched by careful evaluation of how much to use them and how to combine them with traditional instruction. Blended approaches appear to capture the gains of technology while preserving the benefits of traditional learning. Programs that integrate AI tools in measured doses and evaluate their effects apply the lesson directly.
The money at stake makes rigor nonnegotiable. The authors note that billions of dollars are spent on computer-based learning in schools in developing countries each year, and substantially more was spent during the pandemic. Theoretical guidance on optimal levels is limited, and the newness of the technology means there is little track record. Trials that vary dosage supply the evidence needed to allocate those resources well. Single-dose trials answer the wrong question.
For AI developers, diminishing returns are a product brief. If value falls with dosage, the winning tool makes each unit of use as effective as possible and complements teachers. Tools that adapt to students' needs, focus on areas where technology has a comparative advantage and integrate with classroom instruction may extend the range over which returns stay positive. The best edtech metric may be learning per minute, with engagement time a poor substitute.
On policy, the findings caution against both extremes. Rejecting educational technology forgoes the positive effects found at low dosage. Replacing traditional learning wholesale risks the neutral or negative effects found at higher dosage. Evidence-based blending, guided by experiments, offers the most promising path.
The thread running through it all: technology and traditional methods are complements up to a point. AI can enhance learning, and its value depends on how it is combined with other inputs. As AI transforms education, experiments measuring returns to different levels of use will be invaluable. This study shows how to run them and provides an early, rigorous answer: blend, measure and adjust.
For ministries buying AI tutors at national scale, the sting is this: contract for the optimal dose, with a trial to find it, before contracting for maximum usage. Paying for screen time that displaces better learning is an expensive way to stand still.
Introduction
The previous expansion of EdTech as a substitute for traditional learning around the world, the recent full-scale substitution due to COVID-19, and potential future shifts to blended approaches suggest that it is imperative to understand input substitutability between in-person and online learning. We explore input substitutability in education by employing a novel randomized controlled trial that varies dosage of computer-assisted learning (CAL) as a substitute for traditional learning through homework. Moving from zero to a low level of CAL, we find positive substitutability of CAL for traditional learning. Moving from a lower to a higher level of CAL, substitutability changes and is either neutral or even negative. The estimates suggest that a blended approach of CAL and traditional learning is optimal. The findings have direct implications for the rapidly expanding use of educational technology worldwide prior to, during, and after the pandemic.
Takeaway
Billions of dollars are spent on computer-based learning in schools in developing countries each year and substantially more has been spent from the accelerated shifts to technology to facilitate remote learning resulting from the pandemic, but what are the effects of this massive shift towards EdTech? Unfortunately, there is limited theoretical guidance on what optimal levels of CAL should be, and the newness of EdTech in developing countries does not provide a long enough track record to determine what works, what does not work, and what are the impacts of the continued substitution of CAL for traditional learning. The empirical evidence, even from RCTs, is decidedly mixed and focuses exclusively on one dosage level in CAL. To remedy this deficiency in the literature, we study for the first time the effectiveness of CAL on the educational outcomes of school children at different levels of treatment intensity, which sheds light on the substitutability of CAL for traditional learning. Our field experiment involving approximately six thousand Russian schoolchildren and three treatment arms varying dosage levels in CAL generates exogenous variation in CAL use. CAL is substituted directly for traditional learning through homework in the experiment. The experiment provides novel evidence on the substitutability of inputs not only for CAL but for any input in the educational production function. Estimates from the field experiment indicate that CAL increases math test scores at both the base and higher dosage levels.1
As traditional learning is substituted for CAL from the base level to the higher level, however, we find similar effect sizes. Taken together, this suggests that the substitutability of CAL for traditional learning is positive when moving from zero to the base level of CAL, but is neutral when moving from the base level of CAL to the higher level of CAL. These estimates are consistent with CAL having a positive return in educational production. Turning to language achievement, which has been studied less in the previous literature, we find stronger evidence of diminishing substitutability of CAL for traditional learning. The experimental estimates for the returns to CAL for language depend on the level chosen. Importantly, if the experiment had only estimated the treatment effect at the higher dosage level in CAL, the positive effects at the lower level would have been missed. Better knowledge about substitutability is important especially as the widespread substitution of EdTech that happened around the world due to COVID is not likely going away entirely as we move towards more blended approaches in the future. A novel and important finding is that educational production does not appear to fit a situation in which teachers and students can simply substitute between CAL and traditional learning at any level with the same result. For both math and language achievement we find evidence of diminishing MRTS of CAL for traditional learning.
The marginal costs of shifting from a lower level to a higher level of CAL are very low because students already have computers and the software is online and can be replicated for essentially no cost. Although there are fixed costs of developing the software and keeping it up-to-date, the provider made it free of charge to all schools and teachers in the country. In any case, we do not expect that costs will shift the optimal levels much beyond what we find without detailed measures of costs. The primary constraint in this setting is total homework time mandated by the government. Why do we find evidence of diminishing substitutability between CAL and traditional learning? One possibility that is at least consistent with our experimental findings is based on changes in interest and engagement in the subject matter. We find that for both math and language, the base level of CAL resulted in the highest levels of interest. When the dosage level of CAL was doubled students reported lower levels of interest. The finding of diminishing substitutability might be due to these effects on interest and engagement in subject material. Another possibility is that at base level dosages of CAL students gain from being more engaged in learning the material through the technology, but at higher dosages they lose out on the positive effects of traditional learning.
In the end, a blended approach might be the optimal solution for schools and students. The blended approach might keep students engaged, but at the same time expose students to more beneficial methods of learning or just keep students switching around. The full-scale switch from in-person instruction to online instruction due to COVID is a good example of potential negative impacts on engagement. More research is needed on these important underlying questions regarding how students learn using technology and more broadly on the substitutability of other educational inputs.
Notes
- Eric Bettinger et al., Diminishing Marginal Returns to Computer-Assisted Learning, 42 J. Pol'y Analysis & Mgmt. 552 (2023), https://doi.org/10.1002/pam.22442. Takeaway source: 5 Conclusion, reproduced from a free author or repository copy (working-paper version). ↩
Cite as: Eric Bettinger et al., Diminishing Marginal Returns to Computer-Assisted Learning, 42 J. Pol'y Analysis & Mgmt. 552 (2023), https://doi.org/10.1002/pam.22442
Header image: Central Park Skyscrapers, December 2018.jpg by Fletcher, CC BY 4.0, via Wikimedia Commons.
