As part of a three-week pilot study this quarter, half of the students in STATS 60: “Introduction to Statistical Methods” were assigned a RoboTA, an artificially intelligent robot teaching assistant (TA), to answer their questions by email. The other half of the class emailed their questions to a human TA.

Students emailed different addresses based on the TA they were assigned. The emails were then funneled through a program, stripped of any identifying information and relayed to Lucas Janson, the TA for the class. Janson replied to all the emails without knowing whether they were intended for the human TA or the artificial intelligence (AI) TA. This study is double-blind, so neither the students nor the experimenters have information about the other, which helps prevent bias.

Since the research team has just finished collecting data, the effectiveness of the AI TA has not been determined. However, early findings show that students did not implement a Turing test (when humans try to determine if AI is real or not). There were also technical issues with the funneling program for the emails.

Because the double-blind study concluded and the students were debriefed, the same study cannot be conducted at Stanford again. The next step in this AI TA research will need to be done at a different university, which will remain anonymous to maintain the double-blind study.

The idea for the RoboTA came to STATS 60 professor Mike Baiocchi in 2012 after a student sent him an email after a quiz with the words “I feel stupid asking this.” Baiocchi started brainstorming ideas on how to make students feel more comfortable asking him and the TAs questions about the material.

“We’re trying to understand the range of questions that they’d be willing to ask and if that changes, potentially lowering imposed barriers,” Baiocchi said.

After taking some time off from teaching to do research, Baiocchi read a study in 2016 about the interaction between veterans and Siri and how veterans were more open to talk about PTSD with non-human conversational agents. Baiocchi then met with Andreas Paepcke, who works with user interfaces in the computer science department and helped out in the veterans study because Baiocchi wanted to see if he could create a TA featuring AI.

“[Paepcke] was wondering if we poured [AI] over to an educational setting if students would be more comfortable talking about or exposing weakness in their knowledge to an artificial intelligence [robot] relative to a person,” Baiocchi said.

Before creating a full-blown program, the research team had to study how students would interact with an AI TA to create a program that would respond to those interactions. The pilot study concluded with a debriefing on Nov. 17.

 

Contact Stephanie Brito at sbrito ‘at’ stanford.edu.