Convolutional Neural Network (CNN) with a large number of learnable parameters and multiplication operations has shown excellent performance in image classification, object detection, semantic segmentation and low-level image tasks, but the resulting power consumption is too high. The application of CNN on portable devices such as mobile phones and cameras. Therefore, some recent studies have focused on exploring efficient methods to reduce computing costs.
At present, there are a variety of algorithms for obtaining high computational efficiency deep neural networks, such as the weight pruning method of removing unimportant parameters or filters from the pre-trained neural network with minimal loss of accuracy, and by imitating the output distribution of the teacher model Directly learn the knowledge distillation method of the student model. Another research approach to obtain efficient neural networks is to reduce the bit width of weights and activation values to reduce memory usage and power consumption. There are many such methods, which can also greatly reduce the computational complexity, but the performance of the generated network is still lower than the CNN method.
In response to these problems, Hanting Chen et al. proposed an Adder Neural Network (ANN) that does not require a large number of floating-point number multiplication operations, which not only achieves better performance than low-bit-width binary neural networks, but is also useful for future deep learning hardware accelerators. The design has a profound impact.
In the paper "Kernel Based Progressive Distillation for Adder Neural Networks" accepted by the NeurIPS 2020 academic conference in Huawei Noah's Ark Laboratory and the University of Sydney, the researchers used a nuclear-based progressive distillation method to build better performance The additive neural network. The researcher said that this research makes the performance of ANN surpass the same structure of CNN, thus achieving better performance with less power consumption. This research will also benefit applications such as smart phones and the Internet of Things. The latest issue of the
machine heart NeurIPS online sharing invited Xu Yixing, a researcher at Noah's Ark Laboratory, to explain this cutting-edge research in detail.
Sharing topic: Core-based additive neural network progressive distillation method
Sharing guests: Xu Yixing, researcher at Noah’s Ark Laboratory, Master of Intelligent Science Department of Peking University, Bachelor of Zhejiang University. Under the tutelage of Professor Xu Chao from Peking University and Professor Tao Dacheng from the University of Sydney, he had an internship at MSRA. His research interests mainly include computer vision, machine learning and deep learning, and he has published several papers in academic conferences such as NIPS, CVPR, ICML, AAAI, and IJCAI. At present, the main research directions are additive neural network, neural network model miniaturization, neural network automatic search and semi-supervised learning.
Sharing summary: Deep Convolutional Neural Network (CNN) has achieved success in many computer vision tasks. However, to ensure performance, CNN contains a large number of multiplication operations. The recently proposed additive neural network (ANN) replaces the convolution operation with an addition operation without multiplication, which greatly reduces the operating power consumption of the network and the required chip area. However, the accuracy of ANN is comparable to that of CNN with the same structure. There is still a certain gap. This lecture introduces a kernel-based progressive distillation method to improve the classification performance of ANN. Experiments show that on multiple standard image classification data sets such as CIFAR-10, CIFAR-100 and ImageNet, ANN can achieve performance beyond the same structure CNN, laying the foundation for further application of ANN in practice.
Live time: November 25, Beijing time 20:00-21:00
Paper link: https://arxiv.org/pdf/2009.13044.pdf
Join the mobile group and watch the live broadcast
"Mobile group" is the heart of the machine The artificial intelligence technology community initiated will continue to provide online and offline activities such as technical open courses, paper sharing, and interpretation of popular topics, and provide daily selected papers and tutorials, intelligent technology research weekly reports in the community, and "mobile group" Activities such as talent services and industrial technology docking will also be organized from time to time, and all technical practitioners in the AI field are welcome to join. Add the heart of the machine assistant (syncedai5), note "2020", addThis live broadcast group.
2020 NeurIPS MeetUp
Affected by the new crown epidemic, NeurIPS 2020 is held online. Although this can save you a lot of expenses including registration, air tickets and accommodation, it is still a pity that you can't participate in this annual academic conference offline and have close exchanges with academic experts.
Machine Heart is officially authorized by NeurIPS, and will hold 2020 NeurIPS MeetUp in Beijing on December 6th. It will set up Keynote, paper sharing and Poster link , and invite top experts, paper authors and on-site audiences to communicate.
