Cornell professor Ashutosh Saxenam is developing a context sensitive robot that is able to understand natural language commands, from different speakers, in colloquial English. The goal is to help robots account for missing information when receiving instructions and adapt to their environment.
Tell Me Dave is equipped with a 3D camera for viewing its surroundings. Machine learning has enabled it to respond to entire commands with flexibly defined actions. Its computer “brain” has been fed simulated video simulations of actions, accompanied by voice commands from different speakers of different dialects and accents. The robot matches instructions to a range of potential actions and determines the the right one using the context of other words in the command and environment details.
At this time, Tell Me Dave can correctly follow human voice instructions 64 percent of the time.