Affective computing has emerged with the aim to enable affective human-computer interaction by designing machines and interfaces that will sense, recognise, understand and interpret human emotional states via language, speech, facial and bodily gesture and respond accordingly. Although much progress has been achieved in the last decade, one major present limitation of affective computing has been that most of the research on emotion recognition has focused on one single sensorial source, or modality, at a time and especially the face display. While it is true that the face is the main display of a human’s affective state, other sources can improve the recognition accuracy. As natural human-to-human interaction is multimodal, the single sensory observations are often ambiguous, uncertain, and incomplete. Despite this fact, the research community has only recently started proposing emotion recognition systems using affective multimodal data.

This talk will introduce recent advances in multimodal affect recognition by focusing on visual modalities. The talk will start by defining affect and emotions and provide a brief historical background of the research field. The problem domain of multimodal affective computing will be discussed next, by focusing on background research, data collection, data annotation, synchrony between modalities, data integration or fusion, information complementarity or redundancy, and information content of modalities. A number of representative systems introduced within the last 5 years analysing monomodal face or body display will be presented. The talk will then cover the representative systems recognising affective bimodal/multimodal data from visual modalities. The limitations of the current systems will be summarized and the features of an 'ideal' multimodal affect analyser will be discussed in order to provide an insight for the future of affective computing.