Teaching trucks to see

By Daniel Oberhaus

Felix Heide's research is creating camera systems to improve highway safety. Photos by Frank Wojciechowski

For the past year and a half, a small fleet of semi-trucks has been roaming the streets of Albuquerque, New Mexico.

To the untrained eye, these trucks run by Torc Robotics, an independent subsidiary of Daimler Truck, look perfectly ordinary, no different from the thousands of others that pass through this transportation hub every day. But look a little closer and you’ll see that the truck’s driver isn’t actually touching the wheel. The truck is driving itself, expertly navigating the city’s traffic on short test drives.

Surveys have consistently shown that an overwhelming majority of Americans are still nervous about sharing the road with robots, but Felix Heide, an assistant professor of computer science at Princeton, said he wasn’t afraid at all the first time he rode in one of these autonomous trucks. If anything, he felt safer knowing the vehicle was navigating using a sophisticated imaging system instead of inputs from a human driver. He, after all, helped build it.

“It’s a super exciting feeling to have this technology you developed in the lab with all these really rigorous tests finally running in a truck,” said Heide. “This will save a lot of lives.”

In 2016, Heide cofounded a startup called Algolux, based in Montreal, to commercialize next-generation imaging systems that he had been researching for years. As part of this effort, Heide and his team at Algolux began collaborating on a research effort with Torc Robotics that explored how Algolux’s imaging systems could boost safety and efficiency in transportation. In February, Torc took the relationship a step further by acquiring Algolux for an undisclosed sum, which means that Heide’s technology will start being used in operational autonomous test trucks in the not-too-distant-future.

“This was the logical extension of a tight ongoing partnership,” said Heide. “Now we have the opportunity to combine Torc’s strong automated driving system, Daimler Trucks’ engineering and manufacturing capabilities, and our AI and vision capabilities, which are really the three core pillars that you need in order to build a massively scalable autonomous vehicle system.”

Computer scientists have been chasing the dream of self-driving cars for decades, but the technological building blocks only recently moved from academic labs to industrial products. Google’s first experimental self-driving car hit public roads in 2009, and since then many of the world’s leading technology companies — including Apple, Uber and Tesla — have been locked in a fierce competition to establish themselves as the leader in autonomous vehicle technology.

Heide is certain that if autonomous vehicles are ever going to take over our roads, it will require significantly better vision systems, which his technology provides.

“Most existing systems don’t work well in harsh conditions or if there’s things like lost cargo or a fallen motorcyclist on the road,” said Heide. “We need to make them smarter and build robust vision systems so they work in all conditions.”

Autonomous vehicles typically use radar, laser scanning (LiDAR), computer vision, or some combination of these systems to navigate the world. While these systems work well enough in optimal conditions — daytime, good weather, highway driving — they struggle to handle night driving, bad weather, and unusual circumstances. In the case of a dense fog, for example, the light emitted by an autonomous vehicle’s imaging system will be scattered by the fog in such a way that the camera’s sensors may fail to detect an object in the road or hallucinate an object that isn’t actually there. To solve this problem, Heide and his collaborators had to take an entirely new approach to autonomous vehicle imaging systems.

Today, most existing imaging systems used in self-driving cars and trucks are developed by isolated groups of experts each working on one piece of the puzzle: Some may focus on the optics, others on the sensors, others on the algorithm to interpret the images. Heide, by contrast, developed a holistic approach to building autonomous vehicle imaging systems that uses AI to customize each component of the imaging system in the context of the whole. The result is that he has been able to develop cameras that are tailored to specific tasks and dramatically outperform existing autonomous vehicle systems under poor conditions. These cameras can safely steer vehicles through rain, fog and snow, they can detect objects hundreds of feet in front of them at night, and they can even use radar to detect objects around corners.

“This approach of using AI to create trainable models of the entire imaging and image analysis chain allows us to treat these camera systems as systems we can train and evolve so they are optimized for specific tasks,” Heide says. “This is a fundamentally different approach to designing cameras that allows us to put these superhuman vision capabilities in autonomous vehicles.”

Much of the foundational academic work on the imaging technologies now used by Algolux was done in Heide’s lab at Princeton, but he wasn’t satisfied with confining his creations to benchtop experiments. He founded Algolux to forge a pathway to get the technology out of the lab and into the real world where it could make a material difference on the lives of millions of people. “You want to make sure that solving these really challenging engineering problems makes a real impact,” Heide said. “If you want to do that in an industry setting, the only way to do it is to build the technology at scale. And that’s what we’re trying to do here.”

The decision to partner with Torc to embed Algolux’s camera systems into semi-trucks was a natural jumping-off point. Whereas many other automobile manufacturers are focused on adding autonomous systems to personal vehicles, bringing the technology to large trucks promised a big upside — both in terms of economics and safety. The U.S. trucking industry is still struggling to find enough drivers who are willing to handle long-haul routes across the country. The U.S economy depends on these truckers to deliver everything from construction materials to Amazon packages, and Heide hopes that Algolux’s technology can be a win-win solution that both improves the economics of the trucking industry while making the job safer for drivers and the commuters they share the road with.

“We see both a really big business case and a big societal win, which is why we’re so committed to solving this problem. These improvements in autonomous driving will benefit us all,” Heide said.