Xinzhiyuan reported
Source: Algorithms and the beauty of mathematics
(ID: MathAndAlgorithm)
[Xinzhiyuan Guide] Recently, Hinton’s Capsule paper finally lifted the veil of mystery, and because of this paper, he was Published on the front page headlines of major media.
In the paper, Capsule is defined by Hinton as a group of neurons: 's activity vector represents the instantiation parameters of a specific entity type.
His experiments show that the multi-layer Capsule system of discriminative training shows the most advanced performance on the MNIST handwritten data set, and the effect of identifying highly overlapping numbers is much better than that of CNN.
This paper will undoubtedly be the highlight of the NIPS conference in early December this year.
However, Hinton was well prepared for the warm-up of this paper.
A month ago, in an interview with the media in Toronto, the great god Hinton categorically declared that he would abandon backpropagation and let the entire artificial intelligence be recreated from scratch. The unidentified media immediately became trapped.
In August, Hinton also used a speech on "What are the problems with convolutional neural networks?" Converging the outputs of adjacent feature detectors of different types" is very problematic. In
's speech at the time, Hinton didn't mention the different views of Yann LeCun, the father of CNN. After all, the current CNN blindly pursues the recognition rate and has limited help in the "understanding" of the image content.
To further advance artificial intelligence, so that it can understand the content of images and build abstract logic like a human brain. It is definitely not enough to recognize the order of pixels. It is necessary to find a way to represent the content in a good way... This means new methods and technologies.
And the current deep learning theory, since Hinton Great God (trained with restricted Boltzmann machine first, and then tuned with supervised back propagation algorithm) was established in 2007, in addition to the neural network structure With minor revisions, a lot of progress is focused on gradient flow.
is just like the example cited by Zhihu Big V "SIY.Z" in "Analysis of Hinton's Recent Capsule Plan". (Https://zhuanlan.zhihu.com/p/29435406)
sigmoid will saturate, causing the gradient to disappear. So there is ReLU. The negative half axis of
ReLU is a dead zone, which causes the gradient to become zero. So there is LeakyReLU, PReLU.
emphasizes the stability of the gradient and weight distribution, and thus has ELU and the newer SELU.
is too deep, the gradient cannot be passed, so there is a highway.
simply doesn't even need the parameters of the highway, and directly changes the residuals, so ResNet is available.
forcibly stabilizes the mean and variance of the parameters, so there is BatchNorm.
adds noise to the gradient flow, so there is a dropout. The gradient of
RNN is unstable, so a few passes and gates are added, so LSTM is available.
LSTM is simplified, with GRU. There is a problem with the JS divergence of
GAN, which will cause the gradient to disappear or become invalid, so WGAN is available.
WGAN has a problem with the gradient clip, so there is WGAN-GP.
and the essential changes, especially for the dynamic visual content, three-dimensional vision and other problems that CNN cannot solve at present... It may really be possible to find another way by conducting more basic research.
This is of course hard work. If Hinton personally manages it, success will destroy the backpropagation algorithm and deep learning theory that he is famous for, and failure will repeat the mistakes of Einstein's "cosmic constant" in his later years.
Therefore, Li Feifei greatly appreciates his courage here:
Now Capsule's paper has just come out, The great gods of deep learning did not rush to comment on it, and the foreign media in the middle of the night have not yet published articles about it. Even Hacker News, which has been slobbering in the technical circle, is quiet today.
However, what is certain is that Capsule's further effects will definitely appear at the NIPS conference a month later.
As for the subsequent impact of Hinton’s move on deep learning and the entire artificial intelligence community, the great gods, including Yann LeCun, may not dare to come to conclusions. Let’s wait for time to verify whether Hinton’s painstaking efforts are worth it. .
This is just as the great God Hinton said in an interview with Wu Enda:
If your intuition is accurate, then you should stick to it, and you will be able to achieve something in the end; on the contrary, if your intuition is not good, it doesn’t matter if you don’t stick to it. . Anyway, you can’t instinctively find a reason to stick to them.
Of course, the battalion commander certainly believed in the intuition of the Great God Hinton, and he also hoped that artificial intelligence could go further at the current level. Although the meaning of
is different, Hinton’s move reminded the battalion commander of Lord Kelvin, who was also in the ancient years. His 1900 speech on the "two dark clouds" of physics was a "prophecy":
"ultraviolet" "Disaster" allowed Planck, who was nearly a year old, to create a precedent for quantum mechanics, and "ether drift" made Einstein, who had just graduated, started thinking about special relativity, and the edifice of classical physics collapsed.
So, is it a "dark cloud" floating above the artificial intelligence? Is it a new era? let us wait and see.
reference link:
https://zhuanlan.zhihu.com/p/29435406
