Utilizing Log-Based and Neurophysiological Measures to Understand Engagement and Learning with Intelligent Tutoring Systems

Computer-based intelligent tutoring systems (ITSs) are a key to learning in many educational scenarios, but not all students engage effectively with them. Student engagement with ITSs is typically assessed via log data (e.g., problem-solving time), which only partially captures its multidimensional nature. In the current project, we studied how students engage with an intelligent tutor using a combination of log-based, electroencephalography (EEG), and eye-tracking measures. A total of 56 high school students participated in a school-based study with a pretest-intervention-posttest design. During the intervention, students watched short videos and solved chemistry problems using an ITS. Preliminary analysis shows that log-based measures substantially improved model fit, explaining 26% more variance in posttest scores than a model using only pretest scores. Within the model, error rate and hint request rate were significant predictors, while self-reported effort and time spent on problems were not. Hint requests were negatively associated with learning outcomes, emphasizing the need to assess how students use hints and other assistive features of ITSs. Adding EEG and eye-tracking measures did not significantly improve the overall performance of the model. However, a more fine-grained, problem-by-problem analysis revealed that frontal EEG alpha power significantly predicted the number of errors on individual intervention problems. In the future, this finding may support the development of EEG-informed ITSs that can provide more personalized assistance. More generally, these results highlight the value of multimodal measures of engagement in educational technology research.