Course Builder AI
Problem
Participants frequently hesitate before taking action, revisit the same input fields, and move back and forth between sections. Some skip steps entirely. These patterns appear throughout the flow, from creating a project to generating content.
Participants repeatedly returned to input fields.
Six of eight participants hesitated before regenerating content because they were unsure how it would affect their existing work.
Participants were unsure whether their work was saved.
Research
I conducted eight in-person eye-tracking sessions with first-time users and paired each with a retrospective think-aloud (RTA) to understand how they made decisions and where confusion occurred during tasks.
Participants completed the System Usability Scale after each session. The product received a SUS score of 61.3, suggesting that participants did not feel fully comfortable using it.
Hotjar revealed the same interaction patterns, showing where existing users hesitated, retried actions, or tried to figure out what to do next.
Core Insight
The AI Course Builder generates a draft from user input, while detailed editing happens later in the CMS. Across the workflow, participants encountered the same underlying problem: the system did not make its state visible. They could not tell what to provide, how their input would be used, whether work had been saved, or what an action would change. As a result, they relied on trial and error.
Step 01
Participants struggled to differentiate between the “Description” and “Learning Objectives” fields. In gaze replay, users repeatedly copied text between the two, indicating uncertainty about what each field required. This hesitation slowed down their progress.
Design Decision 01
It’s like being asked to draw something without knowing what is expected. Most people do not feel stuck because they lack creativity, but because they do not know what to draw. The existing AI Course Builder creates a similar experience: participants are given input fields, but it is unclear what information they should provide or how it will be used.
To address this, I added four types of guidance to help participants understand each field and reduce cognitive load.
What are users filling out right now?
How much should they write?
What information belongs here?
How will this affect the generated course?
Design Decision 02
Participants’ search for a save option and hesitation to regenerate content signaled a need to preserve existing work. Because AI is non-deterministic, the same input does not guarantee the same output, making previous content important as a reference.
I made saved progress visible through notifications and added version history so users could return to an earlier point before taking action.
Make saved progress and current status visible.
Let users return to an earlier version.
Impact
“We were taking notes the whole time. This was really helpful.”
Conclusion
Triangulating multiple data sources helped validate consistent patterns in user behavior. By combining eye-tracking, RTA, and behavioral data, I identified the underlying reasons for hesitation and confusion rather than relying on a single signal. This shifted the focus from isolated usability issues to the broader problem of system visibility.
Testing was conducted in isolation without the full CMS context, which may have influenced how participants behaved during the study. Future work would validate these findings in a CMS-integrated environment and explore how different interaction models support both generation and editing.
