See our new website at research.ict.csiro.au

skil - Multimodal Instructions and Feedback

We are investigating new forms of computer-supported training, integrating recent advances in new technologies such as virtual environments and natural language. Virtual environments provide a user interface that is based on a spatial and physical paradigm, it guarantees that the perception of the user is plausible and that the user is enabled to act intuitively. With natural language technologies, we extend the virtual environment in order to provide appropriate additional information during the course of the user interaction. The resulting information-rich environment provides not only explicit information about objects in the virtual environment, but also responds to the actions of the user. This enables the environment to reason further about the user interaction based on an explicit representation of the context. The application domain for this approach is currently training or tutoring.

In a training session, i.e., the collaboration between the trainer and the trainee, these tasks form the basis for the generation of the appropriate instructions and feedback. We represent the tasks as explicitly, as hierarchically decomposed task (figure 1), to enable processing and reasoning.  Furthermore, we introduced two layers in the Delivery Platform Myriad for planning the collaboration and the individual turns of the participants, and make extensive use of the explicit representation of the tasks of the participants. Further work has been done to process and detect the user’s actions in the haptic virtual environment as input for this collaboration (user actions like, approach, point, push and pull, etc.). A tutorial reflection, as part of the task of the trainer, exhibits adaptive and appropriate instructions and feedback, as this is a major precondition of effective training and optimal learning.

In our vertical prototype for an information-rich virtual environment (Figure 2), the user action becomes processed through sensors and other layers. At the collaboration management layer, the goals of the participants are planned and executed. The goal of the trainer is embodied as instructions and feedback  to the trainee (user).  The processing refers to the context model where information about the environment is explicitly accessible.

Figure 2 - The architecture the proposed layered processing during the interaction.