A new study from Georgia State University found that community college students receiving targeted, personal text messages from an artificial intelligence chat bot were more likely to complete tasks critical to staying enrolled.
The randomized controlled trial, conducted during the 2020–21 academic year, included about 11,000 students attending Perimeter College, a community college that is part of Georgia State University. Students received personalized text messages—complete with emojis—on their phones reminding them of important deadlines in their academic schedules, telling them how to pay outstanding balances or apply for financial aid, and directing them to campus services. The chat bot also responded to questions from students about campus services and supports, financial aid, and other topics, drawing on thousands of built-in answers to commonly asked questions.
Administrators say the findings reveal valuable lessons about when and how to deliver effective nudges, a student engagement tactic that’s been both embraced and hotly debated on campuses across the country. The hope is the research will help fine-tune the ways colleges use this technology to keep students on track, especially low-income and first-generation students, as the use of chat bots spreads.
If used sparingly and strategically, “the chat bot can cut through the communication clutter,” said Timothy Renick, executive director of the National Institute for Student Success at Georgia State. He believes academia turning its back on nudging, rather than testing “what works and what doesn’t,” would be a “serious mistake.”
The research on the chat bot trial was conducted by postdoctoral research associate Katharine Meyer and Lindsay Page, the Annenberg Associate Professor of Education Policy at the Annenberg Institute at Brown University, which focuses on education research and reform. Meyer is presenting the findings this morning at the Association for Education Finance and Policy annual conference.
The scholars found that timely reminders to students to finish specific tasks were most effective. For example, students receiving messages in the fall were 16 percent more likely than their peers to file their applications for financial aid by January. Students receiving early registration reminders were 4.6 percent more likely to register early for their spring 2021 courses. Messages suggesting students attend a “Comeback Camp,” an academic support program for students with low GPAs in fall 2020, significantly boosted attendance. About 40 percent of students who received the texts participated, compared to 24 percent of students who did not.
The study also found that chat bot reminders successfully encouraged students to seek out help to resolve problems that could hinder their enrollment. Messages that nudged students with unpaid balances to meet with an academic adviser upped visits to advisers by at least 13 percent over the course of a semester. Students who received messages about unpaid balances were 36 percent less likely than their peers to be dropped from the rolls because of outstanding debts to the college.
“When … we’re messaging to students about sort of well-defined tasks where they understand the consequences of not handling that task, that’s where we see the most benefit,” Page said.
Renick noted that getting students to overcome these hurdles can be especially impactful at a college like Perimeter, a predominantly Black institution where about 70 percent of students are eligible for the federal Pell Grant and about 85 percent are working while in college.
At a campus with “mostly low-income students, mostly first-generation students, getting them through the bureaucracy is literally half the battle,” he said. “You can really move graduation rates in significant ways.”
Mitchell Stevens, professor of education and co-director of the Pathways Lab at Stanford University, noted that this kind of research is badly needed, as campuses increasingly use products and services from education-technology firms, like chat bots, in their classrooms.
He described the ed-tech industry as a “promise-heavy and information-thin domain” when it comes to student success outcomes.
“This is a world that’s full of lots of firms providing lots of services about which they are making lots of promises,” he said.
Stevens believes more robust scientific research in collaboration with these firms can help demonstrate the value of these products and improve them.
Georgia State began experimenting with chat bots and tracking results in 2016 in response to staggering rates of summer melt—19 percent of students who confirmed they were attending the university didn’t show up to classes in fall 2015. Despite emails and calls to students and events for students and parents, “We were losing hundreds and hundreds of students,” Renick said.
Since then, the university has extended a more generalized chat bot to all bachelor’s-degree students on its Atlanta campus. Researchers at Brown also conducted another study in fall 2021 where half of a 500-person Georgia State government course, known for low grades and withdrawals, received specialized chat bot messages and half did not. Students who got messages—reminding them about assignments and exams, and sometimes offering sample test questions, based on their progress in the class—were 16 percent more likely to earn a B or higher. First-generation students receiving the texts passed at higher rates and earned grades about 11 points higher than similar students not interacting with the bot. Students with low grades in high school who got texts also sought out academic help at higher rates and were more likely to earn a B or higher in the class than their peers.
“If you can lessen the bureaucracy a little bit, not only does it help students graduate or help students retain or help students get better grades in their class, it really helps the students who are low income, first generation, minoritized … and we’ve seen that across the board,” Renick said.
Lindsey Fifield, director of student success communication at Georgia State, said the chat bot is designed to be user-friendly. Messages address students by name and direct them toward one specific action item based on their personal circumstances. Students receive only up to three messages per week. The language the chat bot uses is “much more casual” than messaging from universities tends to be, avoiding bureaucratic terms such as “FAFSA verification” or “satisfactory academic progress.”
“Those, right out of the gate, are scarier words,” she said. “They’re big. Students don’t always understand what they mean. And even if they do understand what they mean, it’s an intimidating ask.”
Meyer said the key is “coordination within the institution” to make sure that “we hit that sweet spot of sending them the information that they need but not overwhelming them at any given point so they can actually take action on the tasks at hand.”
Given the outcomes of the research, Georgia State plans to include chat bots in more courses, especially STEM classes, and extend chat bot use to all Perimeter students next year.
Using chat bots in academia can be controversial. Nudging as a practice has been criticized in recent years after some studies found it had lackluster outcomes in certain contexts. For example, a study by economists at five universities, published by the National Bureau of Economic Research in 2019, found that nudges from state-level and national organizations to incoming and current college students to submit the FAFSA didn’t budge enrollment or financial aid application rates.
Jinann Bitar, director of higher education research and data analytics at the Education Trust, said research on chat bot interventions is “pretty mixed.” She worries that chat bots built with incomplete data could create more confusion for students and even exacerbate inequities, and some colleges might lack the resources to build carefully designed nudging programs at scale and sustain them.
“A lot of these community colleges that might be best served with this kind of technology are going to have the least capacity for these kinds of interventions,” she said. But she believes the Georgia State research makes a compelling case that chat bots, “if leveraged well,” can help students and free up time for staff members to focus on students with the most need.
Iris Palmer, deputy director for community colleges at New America, a liberal think tank, said chat bots could be especially “powerful tools” for stretched-thin staff members fielding admissions questions at community colleges. The chat bots can also serve students who may be hesitant or uncomfortable about sharing personal information about family hardships or money problems one-on-one with college officials.
“I think we’ve learned a lot about what doesn’t work in nudging,” she said. “Just reminding somebody in a way that’s not customized to them, is not timely and is not from somebody they know and trust, does not work.” But “creating that trusted relationship, even if it’s an automated relationship with a character from the school, does seem to make a difference.”
