I spent the summer of 2004 near Kyoto, Japan, working in the Department of Humanoid Robotics and Computational Neuroscience at ATR.

• My supervisor was Gordon Cheng.
• The program was jointly sponsored by the NSF and the JSPS.

At ATR, I implemented a system that allows DB, a hydraulic humanoid robot, to recognize pointing gestures by humans and then to point in the same direction. The robot first finds a human in the environment by searching for a face and tracking it over time through the use of a skin color model. Using this same color model, DB locates the human's arms by looking for skin blobs to the left and right below the detected face. By fitting ellipsoids to these blobs, the robot can determine the general direction in which the arm is pointing. If the shape and location characteristics of the arms are found to be within expected bounds, and the arms are not hanging vertically at the person's side, the robot estimates a location in which one of the arms is likely pointing. The robot then moves its own eyes and arm toward that point.

The machine perception techniques I used were quite robust in locating and tracking a human being's head and arms, and in accurately determining the direction the human was pointing (in image, rather than world, coordinates; this is a limitation of not having used stereo vision to determine depth). The system's weakness was jitter in the motor control, which made tracking of visual features difficult, though this is a problem that could be solved rather easily.

Some videos of the results:

• DB's eye camera view with movement off. A person's head and arms are located and tracked, and the direction of pointing indicated by a white X (the right arm is favored when both arms are pointing).
• DB's eye camera view with movement on. Jitter in the eye motion makes tracking less robust.
• DB following my pointing (1)
• DB following my pointing (2)

See my web site.