Dialogue Systems for Embodied Agents

We are a group of researchers working in computational linguistics and the design and implementation of multi-modal dialogue systems for embodied agents, at the University of Edinburgh's School of Informatics.

The aim of the group is to build general purpose dialogue interfaces which support multi-modal (i.e. speech and gesture) dialogues with embodied devices, applications, or services -- for example mobile robots, vehicles, and automated homes.

Dialogues with embodied agents present particular practical and theoretical problems, which our research addresses:

  • situated dialogue: actions, states, and events in the physical environment are part of the dialogue context

  • multi-level architectures: the integration of low-level robot sensing and control with high-level dialogue management

  • multi-modality: physical actions, gestures, and interactions with graphical representations can be both inputs and outputs of the dialogue

  • "personality": if robot and human are co-present, how can we design the dialogue system to exhibit "person-like" dialogue behaviours such as reaction to gestures, turn-taking, attention, non-verbal feedback?

  • learning and adaptivity: the dialogue manager must be able to re-plan its next moves, based on unexpected user input or environmental events. The system should be able to learn user preferences and optimal dialogue strategies.

  • multiple topics: dialogues may involve several interleaved "threads", due to the unpredictable dynamics of complex physical situations

  • constraint-management dialogues: it is natural for users to use constraint specifications in natural language command-and-control contexts. The system should be able to interpret and reason about such interactions.

  • evaluation methodologies: existing HCI evaluation paradigms must be extended to cover novel embodied conversational settings.

Our systems are able to handle "unscriptable" dialogues where there is no finite state transition network describing a conversation, and no clear end state for a dialogue. This distinguishes them from restrictive dialogue systems in the "form-filling" paradigm (such as many travel-planning systems), and those in which a state transition network suffices to control dialogue flow.

People

Past and Current Projects

Members of the group have experience in a variety of projects at the interface of dialogue systems and embodied agents:
  • Dialogue with a Mobile Robot: Godot
    University of Edinburgh
Godot
  • Dialogues in the Home Machine Environment: D'Homme
    EU Project: Jan 2001 - Dec 2001
    University of Edinburgh
DHOMME
  • Instruction-Based Learning for Mobile Robots: IBL
    EPSRC: March 2000 - Feb 2003
    University of Edinburgh, University of Plymouth
IBL
  • Embodied Believable Agents: MagiCster
    EU Project: Oct 2000 - Nov 2003
    University of Edinburgh
MagiCster
  • Dialogues with a Robotic Helicopter: WITAS
    Jan 2000 - Dec 2002
    Stanford University, Linköping University
WITAS
  • Learning and Reactivity in Dialogue Systems
    Jan 2003 - Dec 2005
    University of Edinburgh, Stanford University

Software and Tools

Our systems use a common software base DIPPER: Dialogue Prototyping Equipment & Resources based on the Open Agent Architecture. We use the Nuance speech recogniser, and Festival, the Nuance Vocalizer and rVoice for speech synthesis. We also use components such as Gemini (SRI's Natural Language parser and generator), the UNIANCE GSL compiler, the TrindiKit for dialogue management, and several theorem provers including Bliksem, Spass, JTP, Mace, Otter.

Corpora

The IBL Corpus comprises of spoken instructions (English, 24 different speakers) to a mobile robot and is available here.

Selected Publications

  • Johan Bos, Ewan Klein, Oliver Lemon, Tetsushi Oka (2003): DIPPER: Description and Formalisation of an Information-State Update Dialogue System Architecture. 4th SIGdial Workshop on Discourse and Dialogue, Sapporo, Japan. Download: pdf

  • Johan Bos, Ewan Klein, Tetsushi Oka (2003): Meaningful Conversation with a Mobile Robot. Proceedings of the EACL. Download: pdf

  • Johan Bos, Tetsushi Oka (2002): An Inference-based Approach to Dialogue System Design. COLING 2002. Proceedings of the 19th International Conference on Computational Linguistics. Pages 113-119. Download: pdf

  • Christian Theobalt, Johan Bos, Tim Chapman, Arturo Espinosa-Romero, Mark Fraser, Gillian Hayes, Ewan Klein, Tetsushi Oka, Richard Reeve (2002): Talking to Godot: Dialogue with a Mobile Robot. Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2002). Pages 1338-1343. Download: pdf

  • Stanislao Lauria, Guido Bugmann, Theocharis Kyriacou, Johan Bos, Ewan Klein (2001): Training Personal Robots Using Natural Language Instruction. IEEE Intelligent Systems, September/October 2001. Pages 38-45. Download: pdf.

  • Oliver Lemon, Alexander Gruenstein, and Stanley Peters (2002): Collaborative Activities and Multi-tasking in Dialogue Systems, Traitement Automatique des Langues (TAL), special issue on dialogue, 43:2, 131-154.

  • Oliver Lemon, Alexander Gruenstein, Alexis Battle, and Stanley Peters (2002): Multi-tasking and Collaborative Activities in Dialogue Systems, in proceedings of 3rd SIGdial Workshop on Discourse and Dialogue.

  • Oliver Lemon, Alexander Gruenstein, Lawrence Cavedon, and Stanley Peters (2002): Collaborative Dialogue for Controlling Autonomous Systems. In proceedings of AAAI Fall Symposium.

  • Oliver Lemon, Anne Bracy, Alexander Gruenstein, and Stanley Peters (2001): Information States in a Multi-modal Dialogue System for Human-Robot Conversation. In Proceedings of Bi-Dialog, 5th Workshop on Formal Semantics and Pragmatics of Dialogue, pages 57 - 67.

  • Oliver Lemon, Anne Bracy, Alexander Gruenstein, and Stanley Peters (2001): A Multi-Modal Dialogue System for Human-Robot Conversation. In proceedings of NAACL 2001

  • Oliver Lemon, Anne Bracy, Alexander Gruenstein, and Stanley Peters (2001): The WITAS Multi-Modal Dialogue System I. In proceedings of EuroSpeech 2001

Johan Bos, 5 Feb 2003