Multimodal user interaction is a discipline that draws upon a diverse range of signal processing research and technologies to address a challenging long-term goal: human-computer interaction that is truly natural. This presentation will provide an overview of the state of the art in multimodal user interaction research, design and applications, and will cover some of the current issues that motivate future research in the area.

In this presentation, key issues for multimodal research and interface design will be raised. Robustness of modalities based upon audio, video and other real time sensor inputs is one concern that has engaged the signal processing research community for years, and will continue to be a critical factor in the adoption of multimodal technologies. Multimodal systems can offer redundancy of input information that can be exploited to improve robustness. Another issue facing the practical use of multimodal interaction is the need for higher-level semantic fusion of different modality inputs. When complementary information from two or more different types of input sensors arrives at or near the same point in time, the multimodal system needs to combine the information in a meaningful and consistent way. Additionally, the presentation of output information in a manner that is perceptually optimized for the user is a research challenge that must account for the availability of output devices and their constraints, user preferences, application metadata, and relevant content. We will also address the issue of standardization of a multimodal architecture, which would allow researchers to create generic components that could be re-used across a wide variety of application domains. Finally, we will give a brief introduction to the current multimodal activities at NICTA.