Multimodal interfaces expand the communication channel between the system and the user allowing users to express themselves more naturally and interact with complex information with more freedom of expression. One of the many cited advantages of multimodal interfaces is their ability to facilitate effortful complex tasks over unimodal interfaces. These strategies often result in changes to the way multimodal constructions are planned and executed.

Cognitive load refers to the amount of mental effort imposed by a particular task and has been associated to the limited capacity of working memory. I will start with an overview of the state of the art in cognitive load measurement. Recent research has shown that users' multimodal constructions exhibit significant changes as they self-manage their cognitive load when faced with tasks of increasing complexity. Our research focuses on extending the accepted benefits of multimodal interaction by using it to detect fluctuations in cognitive load will be stressed. The primary advantage of this approach is that cognitive load can be determined implicitly by monitoring variations of specific multimodal features during day to day tasks. Such unobtrusive measures may help determine user's cognitive load in real time and adapt information content selection and presentation (multimodal output generation) accordingly, in order to ensure optimal user performance.

In this talk, I will describe an experiment designed to identify the relationships between combined speech and manual gesture input structures and users' cognitive load. The two input modalities are very familiar to users and psychologically closely interrelated, both in terms of planning and execution. Assessing a user's cognitive load implicitly through their multimodal behaviour requires identifying a number of indices that reliably reflect fluctuations. Our hypothesis is that variations in redundant and complementary multimodal constructions can reflect cognitive load changes experienced by the user.

The feasibility of using rates of redundant constructions or even complementary constructions in multimodal input as an index of cognitive load is supported by the results of our study. I will illustrate multimodal patterns that may be monitored to detect cognitive load variations based on symptomatic behavioural features. I will conclude with a discussion on the enormous impact such methods may bring to the design of human computer interaction systems, but highlight the current limitations of the pattern acquisition methodology. Directions for future work will also be addressed.