Do chatbots help students learn?
Many teachers are aware of the disruption that large language models (LLMs), the technology behind chatbots, have brought to education. There are obvious concerns about cheating and plagiarism, as well as a pedagogical worry about using LLMs to do schoolwork. If teachers allow classes to use LLMs for tests and assignments, will the chatbots help students learn? Or will the chatbots harm learning by doing the work for students?
For example, one popular argument in favor of letting students use LLMs for homework or exams has characterized chatbots as a calculator for words. Just as we now allow students to use calculators in math classes, so too should we let students use chatbots in writing classes. But does that argument really hold up to scrutiny?
Are LLMs really like calculators?
To be sure, it’s possible that chatbots could play a helpful role in education. However, the analogy between calculators and LLMs is specious, if not misleading. Here’s why this argument by analogy falls flat.
Calculation isn’t like composition
This point may seem obvious, but figuring out 1 + 1 = 2 is very different from figuring out how to write complete sentences that tell a good story or make a solid argument. The kind of thinking and problem solving required for calculation vs. composition is not the same.
For instance, determining whether a mathematical calculation is correct is relatively straightforward. One plus one will always equal two. Composing a piece of writing isn’t as straightforward. There may be many valid ways to express the same thought or message. The reasons for choosing one rhetorical or stylistic approach over another may depend on the context, intended audience, medium, and purpose of the writing, among other factors.
In short, writing a sentence and making a numerical calculation are different cognitive activities.
Students get calculators after mastering basic math
Even if writing weren’t that different from calculating, the analogy between LLMs and calculators still wouldn’t hold. For example, teachers don’t typically give students a calculator on day one of arithmetic class. Only after mastering how to do arithmetic on paper (or in one’s head) may students use calculators to learn more complex math, like algebra.
As research on math education indicates, the more students use calculators in earlier stages of learning (such as high school math), the lower their grades in later classes (like college calculus). Limiting their access to calculators, at least until they master doing math on paper, generally results in students earning higher grades later on.
Of course, the same holds true for writing. If students don’t learn how to write and instead rely on an LLM do all the work, then what skill is the technology supposed to help them build upon and master?
In other words, when students use chatbots as a substitute or crutch for learning, they may not learn much.
If LLMs aren’t like calculators, how can chatbots help students learn?
Equating LLMs with calculators is, to put it bluntly, a flawed argument. And yet, it’s not unreasonable to hypothesize that chatbots, when designed properly, can play a helpful role in education. To illustrate, there’s evidence that the technology can complement the learning process—say, as a virtual tutor.
I’ve been testing this use case on my own while attempting to learn a language. Recently, I started studying Neapolitan (a regional language in southern Italy that’s different from standard Italian). So I tried using the “Study and learn” mode of CoPilot’s chatbot to help me remember new words while translating phrases—in this example, from Neapolitan to Italian.
As a virtual tutor, the LLM helped me think through the translation by leveraging my knowledge of standard Italian—rather than just giving me all the answers. And helping me think through the translation in this way did create a helpful learning experience. Rethinking and redesigning LLMs in this way—as virtual tutors—could lead to some promising applications of chatbots in education.
Perhaps a crucial question to ask is whether we should design these chatbots to enable human literacy … or to automate and replace it entirely. I’m rooting for the former over the latter.
Related posts
What is functional literacy, and why does our high-tech society need it?
Why more digital technology hasn’t bridged the digital divide
Why the medium is the message: laptops and learning