Machine Heart Report
Editor: Zhang Qian
People can become familiar with a road frequently, and self-driving cars should also be able to do so.
Self-driving cars rely on various sensors to "see" the world, and then use artificial neural networks to process the data from the sensors. They are different from humans because humans have memory and become familiar with a road after walking a few times, but for self-driving cars using artificial neural networks, this road is new every day. This can become a problem in bad weather, where sensors tend to be less reliable.
To alleviate this problem, researchers from Cornell University the Ann S. Bowers School of Computer and Information Sciences and the School of Engineering published three research papers at CVPR 2022. The core idea is to create " memory" and use these memories during subsequent drives.
Paper 1 is titled "HINDSIGHT is 20/20: Leveraging Past Traversals to Aid 3D Perception". The first author is doctoral student Yurong You, and the senior author is Kilian Weinberger, Bowers CIS Professor of Computer Science at Cornell University.
Paper link: https://arxiv.org/pdf/2203.11405.pdf
Professor Weinberger believes, “The core of the above question is, can we learn from repeated experiences? For example, for the first time, a car’s laser scanner was used from a distance When you see an oddly shaped tree, you might mistake it for a pedestrian, but once you get close enough, the object category becomes clear, so the next time you drive by the same tree, even in fog or fog. In the snow, you also hope that the car has learned to correctly identify the tree."
"In fact, you rarely have the opportunity to drive on a road that no one has driven recently. There is always someone who has passed here recently, so gather experience and It seems like a natural thing to take advantage of," said paper co-author Katie Luo.
Led by doctoral student Carlos Diaz-Ruiz, the team drove a car equipped with a lidar sensor on a 15-kilometer loop in and around Ithaca 40 times over 18 months, collecting Information about the environment (highway, city, campus), weather (sunny, rainy, snowy) and different times of the day along the way.
This information forms a data set named "Ithaca365", the details of which can be found in Paper 2 "Ithaca365: Dataset and Driving Perception under Repeated and Challenging Weather Conditions".
Paper link: https://openaccess.thecvf.com/content/CVPR2022/papers/Diaz-Ruiz_Ithaca365_Dataset_and_Driving_Perception_Under_Repeated_and_Challenging_Weather_CVPR_2022_paper.pdf
"This research tackles a key challenge for autonomous vehicles. —Adverse weather conditions," co-author of Ithaca365 Diaz-Ruiz said, "If the road is covered with snow, humans can rely on memory, but artificial neural networks cannot rely on memory, which puts them at a disadvantage."
The "HINDSIGHT" in the title of paper 1 is a method that detects when a car passes an object. A method of calculating object descriptors using neural networks. It then compresses these descriptions, called SQuaSH (Spatial-Quantized Sparse History) features, and stores them on a virtual map, similar to the "memory" stored in the human brain.
The next time it traverses the same location, the self-driving car can query the local SQuaSH database of each lidar point along the way and "recall" what it learned last time. This database is continuously updated and shared among vehicles, thereby enriching the information available to perform identification.
“This information can be added as features to any lidar-based 3D object detector,” You Yurong said. “The detector and SQuaSH representation can be jointly trained without any additional supervision or human annotation, but this requires cost. A lot of time and effort."
While HINDSIGHT still assumes that the artificial neural network has been trained to detect objects and has the added ability to create memories, Paper 3 "Learning to Detect Mobile Objects from LiDAR Scans Without Labels" goes a step further and proposes a A method called MODEST (Mobile Object Detection with Ephemerality and Self-Training).
Paper link: https://openaccess.thecvf.com/content/CVPR2022/papers/You_Learning_To_Detect_Mobile_Objects_From_LiDAR_Scans_Without_Labels_CVPR_2022_paper.pdf
In this paper, the author lets the self-driving car learn the entire perception process from scratch. Initially, the artificial neural networks in the vehicles never touch any objects or streets. After traversing the same path multiple times, it can learn which parts of the environment are stationary and which objects are moving. Slowly, it will teach itself what other traffic participants are and what safety factors can be ignored.
The algorithm can then reliably detect these targets, even on roads not included in the initial repeated traversal.
The researchers hope that both approaches will significantly reduce the cost of developing self-driving cars, which still rely heavily on expensive human-labeled data, and make such cars more efficient by learning to navigate in the most commonly used locations.
The first author of Papers 1 and 3 is Cornell University doctoral student You Yurong (he also participated in Paper 2). He graduated from the ACM class of Zhiyuan College of Shanghai Jiao Tong University. He joined Lu Cewu's laboratory during the summer of his sophomore year and started research on computer vision, and reinforcement learning. He also went to the Stanford University AI Laboratory in the summer of his junior year. During the summer vacation, I went to Cornell University for a scientific research internship in related fields. Later, I was admitted to Cornell University and California Institute of Technology with full scholarship for doctoral degrees in computer science.
In the end, he chose Cornell University to pursue a Ph.D. under the tutelage of Kilian Q. Weinberger, professor of computer science, and Bharath Hariharan, assistant professor of computer science, focusing on computer science, machine learning and other directions.
Reference link: https://news.cornell.edu/stories/2022/06/technology-helps-self-driving-cars-learn-own-memories
https://news.sjtu.edu.cn/ztzl_qczj/20180628/ 78193.html
