The latest standard in robotics

But getting to the point where robots can work in the unstructured environment of our homes (as opposed to industrial settings) would take a major technological leap and a massive coordination of efforts from roboticists around the globe. The living room has been called the last frontier for robots—but first, the robotics community needs some standards that everyone can agree on.

Enter a suitcase-sized box containing 77 objects. It contains things like hammers, a cordless drill, a can of Spam and a nine-hole peg test. As ordinary as they may seem, these carefully curated household items could be the future of a new kind of standardization for robotics. Known as the Yale-CMU-Berkeley (YCB) Object and Model Set, the intent is to provide universal benchmarks for labs specializing in robotic manipulation and prosthetics around the world.

Dollar, an associate professor of mechanical engineering & materials science, came up with the idea about two years ago. He wants to bring a level of specificity and universality to manipulation tasks in robotics research.

For instance, a research paper today might describe a particular task as "robotic hand grasps hammer." Are we talking about a big hammer or a little one? We don't know, and that's a problem if you work in a robotics lab looking to replicate the research. With the YCB Set, everyone's on the same page—in this hypothetical case, by working with the same 23.45-ounce Stanley hammer included in the set.

In addition to the objects, the project also provides five examples of manipulation tasks (such as pouring water from a pitcher to a mug, or setting the table) and benchmarks for each. A website for the project also allows other laboratories to expand on these tasks by contributing their own protocols and benchmarks. When laboratories work solely by their own standards and protocols, Dollar said, there's often an unconscious bias toward that lab's particular strengths. Universal standards would provide a more impartial way to evaluate results.

The YCB Set arrives at a time when the robotics field has reached a critical point. Robots currently do well in structured environments, such as factory settings, where they perform and repeat a very limited number of tasks. "In a structured environment, a robot sees exactly the same object in exactly the same place," Dollar said. "It's a relatively straightforward thing to get robots to operate in those environments because you just have to program it to do one thing. And you can always program something to do one thing well."

But Dollar and other roboticists have something more challenging in mind for their creations. "People in the robotics community today are thinking about robots that can work in daily environments, and in the home," he said. "That's sort of the flip side of assembly lines."

Standards have long been a crucial part in the advancement of science. Until the 19th century, the schedules of individual communities were governed by municipal clocks. Today, thanks to globally coordinated time (and increasingly accurate atomic clocks), we have personal GPS systems and driverless cars.

For centuries, people used their hands and feet to measure the lengths and heights of things. When things got standardized around the world, the International Committee for Weights and Measures stored metal rods in a climate-controlled vault in Paris, each serving as the standard bearer for a particular unit of measurement. In more recent years, those metal objects have been usurped by even more precise standards based on the speed of light (in which we come back again to atomic clocks and the standardization of time).

In a sense, the 77-item box is the robotics equivalent of the Paris vault or the atomic clocks, and may usher in an era when laboratories better communicate to advance the field at a faster pace. It's a critical step, since things get tricky as robots move away from the assembly line.

Dollar specializes in robotic manipulation, or grasping. As humans, we often take for granted the complexity of something as seemingly simple as picking up a fork and using it. To build robots that can perform not just one of these tasks, but many, individual labs can no longer work as isolated villages operating on their own measurements. They need a universal standard.

That's where the box of 77 items comes into play. The objects are the sort of things you find around the house. Certainly, it's easy enough for a robotics lab to find their own objects to manipulate. But for the research to move forward, the results of that lab's work has to be comparable to other labs.

"When we have a new idea for a new component or hand idea, we want to test it out and see how well it works," he said. "With quantitative evaluation, we can see how things stack up compared to other ideas." There have been other attempts to standardize manipulation tests, but Dollar said they don't capture the high level functionality that the YCB Set demonstrates.

It's only recently, roboticists say, that such standards would have much purpose. The field simply wasn't sophisticated enough until recent years to benefit from such standardization. It's a different story now, though, as integrated systems require the work of multiple disciplines to create a robot that can do something like put away dishes.

"As robots move out of the lab and into the real world, it gets harder to understand their capabilities and limitations," said Robert Howe, the Abbott and James Lawrence Professor of Engineering at Harvard. "In a factory where everything is carefully arranged, you can rigorously test how they work, but in my kitchen I have 20 kinds of coffee mugs. So it's a big puzzle how to characterize and compare robots. The approach that Aaron is taking is a promising one."

Howe notes that even a seemingly simple grasping task requires very advanced engineering. You need to plan the hand and the arm so that it doesn't knock over other things, the contacts must be carefully controlled—and then you have to wrap up all these coordinated elements into a single system that works fast enough to be useful. His lab is concerned with tactile sensing, which is one piece of the puzzle, but the same task could also require the input of computer vision specialists.

"That's why the YCB Set is clever," he said. Now a lab can score how well they do on a certain task, and other labs can try to match or beat that score. After Dollar had the idea for the standardized set, he brought on board two former colleagues in the robotics community, Dr. Siddhartha Srinivasa from Carnegie-Mellon University and Dr. Pieter Abbeel of UC Berkeley ("These are people I knew I could work well with and make something happen").

And he assigned Berk Calli, a postdoctoral associate in his lab, to take the lead on the project. Calli, who came to Yale in 2014, said the lack of reproducibility in robotics is a problem long recognized among researchers in the field. It's very rare, he said, for a paper to compare just two algorithms from other labs.

"If you can get five or 10 groups using one single protocol to compare their algorithms, that would be a huge step," he said. "It will be a huge thing in terms of quantification and comparison in robotics, because this has never been done before."

It's gotten to the point, Calli said, that the field doesn't have much choice but to take on the matter of standardization. "There's like a pool of algorithms and no one knows which performs the best. And we cannot proceed further without knowing what is working and what is not."

Ideally, the YCB Set will take on a life of its own. The objects and example tasks provided are just a beginning. Manipulation research progresses quickly and covers a wide range of technical interests and research approaches, so the five manipulation tests Dollar and his team provide are only examples of protocols that labs can use with the objects.

That's why on the YCB Object and Model Set website, the research team has also provided a framework for other labs to contribute their own manipulation tests and benchmarks. There, researchers can see protocols from other labs and have a forum for discussion.

"The main thing is just getting other researchers to propose their own protocols and get people to utilize them," Dollar said.