Matt Adcock and Chris Gunn, CSIRO ICT Centre, Canberra Australia

In a typical video-conference situation, users can only interact with the physical objects that are local to their side of the communication channel. But there are many physical tasks, such as machine maintenance or medical treatment, which could benefit from the direct spatial guidance from a remote expert. Previous systems have shown that projected augmentation onto the physical objects in the workspace of a ‘worker’ can allow a remote ‘helper’ to assist through pointing, drawing or gesturing [5], [1], [2].

However, these systems tend to fail when the helper sketches some annotations onto an object that is subsequently moved. Similarly, the systems don’t keep annotations ‘stuck’ in place if the projector and camera move within the workspace (as is the case for a wearable system).

The RIDES system [3] attempted to address the movable camera issue by approximating a medical patient lying on a bed as a static 2D plane. While this allowed the camera to move to known pan/tilt positions, any movement by the patient resulted in mismatched annotations.

The camera on the worker’s helmet captures video frames of the scene which are streamed to the helper’s computer using Motion JEPG. The video is displayed to the helper on their graphics tablet. The helper can sketch, freehand, directly onto the video and the normalized screen coordinates of their pen position are sent back over the network to the application running on the worker’s laptop.

The application running on the worker’s laptop also receives, but does not display, the video frames collected by the camera. It employs the ARToolKitPlus library to optically track any physical objects in the video stream that have fiduciary markers attached. A ‘hidden’ Virtual Reality 3D scene is constructed to mirror the real world using the marker tracking data and X3D models of the tracked physical objects. Also in this scene is a ‘viewpoint’ positioned to correspond to the location of the worker’s laser projector (with respect to the worker’s camera). This is similar to the Shader Lamps technique [4].

As the stream of data containing the helper’s sketching is received, a ray is cast from the virtual camera location and the point of collision with any (currently tracked) virtual object is calculated. This ray represents the helper’s line of sight through the pen tip to the drawing point on the object. Successive surface collision points are used to create a 3D line set representing thehelper’s annotation.

Finally, the virtual scene is rendered via the ‘viewpoint’, but with only the annotation lines visible. The rendering is displayed ‘full screen’ by the worker’s pico projector and can be seen by both theworker and the helper on the surface of the physical object.

For the helper, the experience of drawing on the video of the worker’s physical environment is similar to a common desktop paint application. As such, we have also implemented the ability for the helper to switch into an ‘eraser’ mode and remove any specific annotation, or delete all annotations at once.

To support multiple dynamic objects, the 3D line sets are assigned to the virtual model they were created against. When a sketch moves from one object to another, a new line set is produced.