HeyTeddy: programming hardware using only voice commands

HeyTeddy is a novel conversational prototyping tool for physical computing. It allows users to program and execute code in real time on an Arduino board using voice commands instead of writing actual code. Using dialogues, HeyTeddy also guides users with tutorial-like assembly instructions and provides additional information about components. Finally, it uses its agency to suggest possible debugging strategies.

Article | Spring 2020

Physical computing prototyping has become increasingly popular among technical non-experts such as students and creators, yet prototyping interactive systems still remains a laborious task. Past research has shown that the main difficulty novice users face is the integration of two domain-specific tasks: 1) physically placing electronic components and wires to assemble circuits and 2) programming software for embedded systems such as the Arduino platform.

To ease those difficulties, various hardware prototyping tools have emerged in forms such as modular electronic toolkits that lower the barrier to circuit assembly and testing tools to enrich debugging information. Proposed solutions include simplified prototyping tools, which consist of standalone physical blocks linked to programmed behaviors, and better diagnostic tools that provide users with detailed information about the internal state of the electrical circuit. However, these systems are either limited in what users can achieve or still require users to deal with hardware and software separately.

 

A team of researchers in the Department of Industrial Design at KAIST led by Yoonji Kim and Prof. Andrea Bianchi developed the first general-purpose conversational programming environment for physical computing prototyping that simplifies the process of prototyping without limiting what users can accomplish. HeyTeddy parses, interprets, and executes the spoken commands in real-time, and the results are physically reflected in the hardware components. For instance, the “write high” command enables the LED drive by configuring the I/O pins to operate as digital output for which the internal state is set to high (e.g. 5V logic level). Thus users are not required to write, compile, or handle errors in the software, and they do not have to manually upload the code to the hardware. In addition, HeyTeddy supervises user choices, prevents logical errors (e.g. writing analog values to digital pins), guides users on the requirements for building a circuit, and provides an opportunity to test individual components within a context. Users do not need to remember the commands or different options associated with each command. Indeed, HeyTeddy is capable of capturing the users’ intent from the conversation and behaves accordingly. Moreover, the system can autonomously infer the correct options from the context when ambiguities arise. HeyTeddy was published in the Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT) in December 2019.