Virtual Classrooms: How Online College Courses Affect Student Success
GAICIL Insights
Hold everything constant except the screen, and see what happens.1 Virtual Classrooms: How Online College Courses Affect Student Success, by Eric P. Bettinger, Lindsay Fox, Susanna Loeb and Eric S. Taylor, runs that thought experiment with real data. Online college courses are a rapidly expanding feature of higher education, yet credible evidence on their effects relative to in-person classes had been scarce. Using an instrumental variables approach, the authors find that taking a course online reduces student success and progress in college. Grades are lower in the online course and in future courses, and students are less likely to remain enrolled.
Universities scaling AI courses might start here: report outcomes for the bottom third of students by prior GPA, alongside the average. If the technology helps the strongest and loses the weakest, the average will hide it.
The setting is unusually clean. The only difference between online and in-person courses is the medium of instructional delivery. Professor assignment, class size, syllabus and textbooks are identical. The study is among the first to estimate effects of at-scale online courses at nonselective 4-year institutions or for-profit institutions, which serve a substantial proportion of students. The findings are robust across many specifications. Same teacher, same book, worse outcomes.
The distribution is the sharpest part. The variance of student outcomes increases, driven at least in part by larger negative effects of online course taking for students with lower prior GPA. Online delivery widens gaps between stronger and weaker students. The results align with prior studies showing better mean outcomes for in-person courses, and with earlier estimates of lower grades and persistence. The medium magnifies prior advantage.
The authors handle welfare honestly. They cannot provide a full welfare analysis. An online option might have enabled some students to take college courses who otherwise would not have. The estimates apply to students who could choose either format, and expanded access cannot be measured in this setting. The estimates remain a critical input to a more complete welfare analysis. On mechanisms, they note prior arguments that procrastination in online settings worsens outcomes.
Anyone deploying artificial intelligence in education should read this as a baseline to beat. AI-enabled learning is often presented as the next stage of online education, and the paper shows that moving instruction online, all else equal, can harm students, especially those with weaker preparation. Changing only the medium does not automatically preserve learning. The interactions, structure and accountability of in-person classes appear to matter. Calling a product "AI-powered" does not exempt it from this evidence.
The diagnosis also points to where AI could earn its place. If online students struggle with engagement and self-regulation, tools providing timely feedback, prompts and personalized support could target the specific weaknesses of online formats. AI tutors that detect when a student is falling behind and intervene could reduce the larger losses of lower-GPA students. Procrastination is a problem software might plausibly address. Whether it does is an empirical question, and designs as clean as this one would be needed to answer it.
The distributional finding belongs in AI policy. Technologies that widen outcome gaps could reinforce inequality even while expanding access. Institutions adopting AI-enabled courses should monitor outcomes across student groups and design supports for those at risk. The goal: capture the access benefits of online education while limiting its costs for vulnerable students.
The welfare caveat keeps the argument balanced. Online options may expand access for students who would otherwise take no college courses. Restricting online education to protect students who could attend in person might harm those for whom online courses are the only option. The policy challenge is to improve online learning, and AI may be among the most promising tools for doing so.
Introduction
Online college courses are a rapidly expanding feature of higher education, yet little research identifies their effects relative to traditional in-person classes. Using an instrumental variables approach, we find that taking a course online, instead of in-person, reduces student success and progress in college. Grades are lower both for the course taken online and in future courses. Students are less likely to remain enrolled at the university. These estimates are local average treatment effects for students with access to both online and in-person options; for other students, online classes may be the only option for accessing college-level courses. (JEL I23, I26)
Takeaway
This study is the first, of which we are aware, to not only estimate the effects of at-scale online courses but to also understand the distributional consequences of online courses at nonselective 4-year institutions of higher education or at for-profit institutions, both of which serve a substantial proportion of students. In addition, this study uses an instrumental variables strategy for addressing selection and estimating causal effects that is arguably more convincing than prior approaches to estimating effects of online course taking at scale. Furthermore, in contrast to previous studies of online courses, our setting provides a clean counterfactual in which the only difference between online and in-person courses is the medium of instructional delivery. All other aspects of the course – professor assignment, class size, syllabus, textbooks, etc. – are identical across online and in-person courses. Our analyses provide evidence that students in online courses perform substantially worse than students in traditional in-person courses, and these findings are robust across a number of specifications. We also find that the variance of student outcomes increases, driven at least in part, by differentially larger negative effects of online course taking for students with lower prior GPA. The results are in line with prior studies of online education in showing that inperson courses yield better mean outcomes than online courses (e.g. Figlio, Rush, and Yin 2013, Streich 2014b).1
Our estimates of lower grades and lower persistence in online courses are similar to those in Xu and Jaggars (2013, 2014) and Hart, Friedman, and Hill (2014).2 While we find that online courses lead to poorer student outcomes, we cannot provide a full welfare analysis. Most notably, the existence of an online course option might have enabled some students to take college courses that otherwise would not have done so. First, our estimates are most appropriately generalized to students who could take a course online or in-person. Second, we cannot estimate the extent of this expansion in college course access in our setting.
Our estimates are, nevertheless, a critical input to a more complete welfare analysis. There are several potential explanations for the increase in student variance. Figlio et al. (2013) argues that procrastination in online settings was much more prevalent and led to the worsening of outcomes.3 While we have no measure of procrastination, we find the worsening of outcomes is especially large at the bottom of the distribution – students who either withdraw or fail the course. These students generally fail to complete the course requirements and may have procrastinated in their course responsibilities. We find fewer differences in outcomes at the top of the grade distribution suggesting that online courses may not be worse for high achieving students. If indeed procrastination is driving many of the results, courses could be redesigned to improve student participation. As for the change in the variance attributable to professors, the decrease in variance in online classes would be both good and bad. It might eliminate extreme outcomes from bad instructions, yet it might also stifle good teaching. A number of potential reasons for these differences exists. In the online courses, professors use the same syllabus, lectures, and materials, and in some cases, even the same scripts. The greater adherence to a script among professors surely contributes to a more similar and less varied experience across professors.
It may also be the case that online interactions are more limited than in-person so there are fewer opportunities for professors to influence students. Overall, the results – lower student performance, greater student variation, and lower professor variation – while not necessarily surprising, provide evidence that online courses are not yet as effective as in-person courses. This current state, however, is clearly not a necessary end state. The greater homogeneity of online courses points to the potential to scale good courses and reduce the variability in experiences that students have within the same course taught by different professors in traditional classrooms. Lastly, our results provide a rationale and jumping off point for exploring the potential mechanisms that could be driving the results such as student effort, procrastination, and professor behavior.
Notes
- Eric P. Bettinger et al., Virtual Classrooms: How Online College Courses Affect Student Success, 107 Am. Econ. Rev. 2855 (2017), https://doi.org/10.1257/aer.20151193. Takeaway source: III. Discussion and Conclusion, reproduced from a free author or repository copy (working-paper version). ↩
- Xu, Di, and Shanna Smith Jaggars. 2013. "The impact of online learning on students’ course outcomes: Evidence from a large community and technical college system." Economics of Education Review no. 37:46-57. doi: 10.1016/j.econedurev.2013.08.001.. ↩
- Figlio, David, Mark Rush, and Lu Yin. 2013. "Is It Live or Is It Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning." Journal of Labor Economics no. 31 (4):763-784.. ↩
Cite as: Eric P. Bettinger et al., Virtual Classrooms: How Online College Courses Affect Student Success, 107 Am. Econ. Rev. 2855 (2017), https://doi.org/10.1257/aer.20151193
Header image: Aerial of the Brevard College campus in Transylvania County, North Carolina 01.jpg by Harrison Keely, CC BY 4.0, via Wikimedia Commons.
