UIST 2026 · Detroit, MI, USA

Programming by Teaching

Authoring Agent Workflows through Voice Conversation

Takato Yamazaki, Nao Yoshida, Keita Higuchi

SB Intuitions, Tokyo, Japan

0:00 / 0:00

Abstract

Agent workflow automation is increasingly accessible to users without programming experience, but much of the knowledge needed to author workflows is implicit. Programming by Demonstration captures what users do on screen but not the rules behind their actions. We propose Programming by Teaching, where users teach a task to an AI through real-time voice conversation while navigating the application, as they would teach a colleague. The AI listens, takes notes, and generates an agent workflow. We implemented both modes in the same Chrome extension and compared them in a within-subjects study with 28 participants, primarily non-programmers. Teaching passed held-out edge-case tests at 87.2% compared to 46.7% for demonstration (p = .002). Teaching reduced cognitive load (p = .04), scored higher on both pragmatic and hedonic quality (p ≤ .001), and made it easier to incorporate rules not visible in the UI (p = .003). The hardest part of demonstration was not building the workflow but editing it afterward; teaching sidesteps this by front-loading specification into conversation.

How it works

The user works through the task on screen and talks the AI through it, as they would show a new colleague. The AI listens, keeps notes the user can check and correct, and turns them into a workflow when the user is ready.

Left: a browser expense report form. Middle: a conversation timeline in which the user states receipt and approval-routing rules aloud while clicking form fields, and the AI replies and updates its notes. Right: the generated workflow, which checks for a receipt and either asks the requestor for it or fills the form with three-tier approval routing.
A user teaches an expense report task by working in the form and explaining its rules aloud. The AI turns its notes into a workflow that also handles a case the user never demonstrated.

User study

We compared Teaching with Programming by Demonstration, where the AI learns from what the user does on screen. Both modes shared the same workflow generation. 28 participants, 18 of whom had never programmed, automated office tasks in a simulated corporate portal with both modes. Each workflow was then tested on new emails with cases the participant had not walked through.

Programming by Demonstration

Recorded actions 1Typed the amount 2Picked Director 3Clicked Submit Generated workflow Fill in form Route to Director Submit for every amount

The AI watches what you do. It records that you picked the director, but not why, so the workflow does the same for every request.

Programming by Teaching

See this amount? If it’s over ¥300K, it goes to the director. Notes Over ¥300K → Director Otherwise → Manager Generated workflow Fill in form Over ¥300K? Director Manager Submit

You explain the rule while you work. The AI writes it into its notes, and the workflow follows it.

Results

Held-out tests passed

Teaching passed 34 of 39 held-out tests and demonstration 21 of 45 (p = .002). The gap was largest for participants who had never programmed.

DemonstrationTeaching

Workload and user experience

Teaching also lowered workload, scored higher on user experience, and made it easier to add rules that are not visible in the interface.

DemonstrationTeachingMedianMiddle 50% of responses
Each violin shows the spread of the 28 participants' ratings; the bar covers the middle half and the dot marks the median.

Takeaway

Specify first, verify later

Recording a demonstration is the easy part. The effort lands afterward, in finding which rules are missing and where to repair them. Teaching moves that effort up front, into the conversation.

“Finding what to fix” was the hardest part. — P16, P22, P24

“Writing everything into the editor was exhausting.” — P12

Make the AI's understanding visible

The main concern with teaching was not distrust but visibility: participants could not always tell how much the AI had understood. Notes that users can check and correct help; editable rule lists and version diffs are a clear next step.

“I couldn't tell how much it really understood.” — P20

Teaching helps most where editing is hardest

The largest gains were for participants who had never programmed, the group that struggled most with repairing generated workflows. Teaching does not replace demonstration; it complements it.

“I could work as if I were teaching a person.” — P2

BibTeX

@inproceedings{yamazaki2026programmingbyteaching,
  title     = {Programming by Teaching: Authoring Agent Workflows through Voice Conversation},
  author    = {Yamazaki, Takato and Yoshida, Nao and Higuchi, Keita},
  booktitle = {Proceedings of the 39th Annual ACM Symposium on User Interface Software and Technology (UIST '26)},
  year      = {2026},
  address   = {Detroit, MI, USA},
  publisher = {ACM},
  doi       = {10.1145/3830398.3830516}
}