Colloquium by Dr. Michael Jenkin about "Talking with robots at depth"

Talking with robots at depth

Abstract: The underwater environment provides a range of interesting applications for human-robot teams. A critical issue for any human-robot interaction at depth is the development of an appropriate communication mechanism between humans and robots. Robots have a wide range of options for communication, including the use of displays and lights, to communicate to human team members. For human to robot communication a number of different approaches are possible, but perhaps the most natural for humans is the use of well understood and accepted hand signals to communicate with robots.  How can we leverage standard SCUBA communication signals to talk to robots underwater?

Perhaps the most straightforward mechanism would be to train some network to recognize these gestures. Given a gesture language of 30 or so standard communication symbols it  would be expensive and perhaps impractical to develop a hand-labelled dataset of these hand gestures to support a machine learning-based approach. To avoid the cost of hand labelling such a large dataset, here we automate the process of collecting a labelled dataset through the use of a simple model trained on a hand-labelled dataset that only identifies salient objects(divers, their heads and hands), and then use a weakly supervised learning process to label a complex set of diver gestures. The result of this process is a system that can recognize a large number of diver hand gestures. Performance of the resulting system is compared against a hand-labelled set of diver gestures. We demonstrate the performance of this approach using a large dataset of underwater gestures and expressions.

This work is in collaboration with Robert Codd-Downey, a PhD student at York University.

Dr. Michael Jenkin
Lassonde School of Engineering
York University