Designing rule-based conversational agents with behavioral programming: a study of human subjects

Purpose: In this work, the authors propose to harness the advantages of behavioral programming as a new technique for designing rule-based conversational agents. Design/methodology/approach: To examine the study’s hypotheses, the authors perform a first-of-its-kind user study through which the authors examine how potential designers, both expert designers, computationally-oriented designers, and otherwise, leverage behavioral programming (BP) and dialog graphs for designing conversational agents (CAs). The authors also use two standard CA settings common in the literature: designing a CA representative for a user in an online dating service and a non-character player in a role-playing game (RPG). Findings: The study’s results indicate that BP can be successfully utilized by computationally-oriented designers, with or without prior knowledge in CA design, and can facilitate the design of better CAs (i.e. more accurate and more robust). However, to capitalize on these potential advantages, designers may be required to devote more time to the design process and are likely to encounter higher temporal demand levels. These results suggest that BP, which was initially proposed and evaluated in the general context of software design, can constitute a valuable alternative to the classic rule-based CA design technique commonly practiced today. Research limitations/implications: An important limitation of this study is the relatively small participant pool. While the authors do plan to extend this study in the future, the current coronavirus disease 2019 (COVID-19) situation makes it ever more complex to conduct formal user studies of this kind. It is, however, important to note that despite the low number of participants, many of the results are found to be statistically significant. Practical implications: The authors plan to continue this line of work and conduct human studies for additional design techniques in other popular agent-based settings. Specifically, the authors seek to explore how people of different backgrounds should design agents for various tasks such as automated negotiation (e.g. how should a person design a representative agent to negotiate on her behalf?) and social choice (e.g. how should a person design a voting bot to represent her in online voting systems?). Originality/value: People are increasingly interacting with conversational agents in various settings and for a variety of reasons, as the market size of those agents keeps on growing every year. Through a first-of-its-kind human study (N = 41), consisting of both expert designers, computationally-oriented designers, and otherwise, the authors demonstrate a few key advantages and limitations of BP in the realm of conversational agents and propose its consideration as an alternative to the classic dialog graph technique.