In 1982, American scientist John Hopfield invented a neural network and added many restrictions to allow the neural network to maintain memory during changes in order to learn. In the same year, Finnish scientist Teuvo Kohonen proposed a self-organizing map (Self-Organizing Map) based on the unsupervised algorithm vector quantization neural network (Learning Vector Quantization Network), hoping to reduce the complexity of the problem by shortening the Euclidean distance between the input and the output. Correct relationships are learned from the network. In 1987, American scientists Stephen Grossberg and Gail Carpenter proposed the Adaptive Resonance Theory Network (Adaptive Resonance Theory) based on their earlier theories, which means that known information and unknown information are "resonant" to infer from the known information. Unknown information realizes "analogous learning". Although these networks have added keywords such as "self-organizing", "adaptive" and "memory", their learning methods are not efficient and require continuous optimization and design based on the application itself. In addition, the memory capacity of the network is very large. Small and difficult to apply in practice.
In 1986, computer scientists David Rumelhart, Geoffrey Hinton and Ronald Williams published the backpropagation algorithm (Backpropagation), which solved the problem of neural network learning in stages. Through the chain rule of gradient, the difference between the output result of the neural network and the real value can be fed back to the weight of each layer through the gradient, so that each layer function is trained like a perceptron. This is Geoffrey Hinton's first landmark work. Today he is an engineering researcher at Google and has won the Turing Award, the highest honor in the computer field. He once said in an interview: "We don't want to build a model that simulates the way the brain works. We look at the brain and think, since this model of the brain works, then if we want to create some other models that work, we should Find inspiration from the brain. The backpropagation algorithm simulates the feedback mechanism of the brain.
Later in 1994, while working as a postdoctoral fellow in Geoffrey Hinton's group, computer scientist Yann LeCun combined the neural cognitive machine and the backpropagation algorithm to propose the convolutional neural network LeNet for identifying handwritten postal codes, and achieved 99% automatic accuracy. recognition rate and can handle almost any handwriting form. This algorithm was a huge success at the time and was used in the US postal system.
In 2017, the ImageNet image classification competition announced the completion of its last session. But this does not mean that deep learning has ceased. On the contrary, the research and application of deep learning have broken away from the previous stage of "classification problems" as the research theme and entered a stage of extensive development. At the same time, the number of submissions to international conferences related to deep learning has increased exponentially year by year, which also shows that more and more researchers and engineers are devoted to the development and application of deep learning algorithms. The development of deep learning in recent years has shown the following trends.
First, from a structural point of view, the types of neural networks will become more diverse. Among them, the Generative Adversarial Network (Generative Adversarial Network), which can perform the reverse process of convolutional neural networks, has developed rapidly since it was proposed in 2016 and has become an important "growth point" of deep learning. Since deep learning algorithms can extract features from original information (such as images), the reverse process is also logically feasible, that is, using some messy signals to generate corresponding images through a specific neural network. Therefore, computer scientist Ian Good fellow proposed a generative adversarial network. In addition to the generator that can generate images, this network also provides a discriminator. During the training process, the generator tends to learn a generated image that is difficult for the computer to distinguish and is extremely close to reality, and the discriminator tends to learn a powerful ability to judge real images and generated images. The two are confrontational learning. The more realistic the generated pictures are, the more difficult it will be for the discriminator to distinguish; the stronger the discriminator is, the more powerful the generator will be to generate new, more realistic pictures. Generative adversarial networks are widely used in areas such as face generation and recognition, image resolution improvement, video frame rate improvement, and image style transfer.
Second, the research questions tend to be diverse. On the one hand, some concepts in other branches of machine learning, such as reinforcement learning and transfer learning, have found new positions in deep learning. On the other hand, the research on deep learning itself has also developed from "engineering trial and error" to "theoretical derivation". Deep learning has been criticized for its lack of theoretical foundation, relying almost entirely on the experience of data scientists during the training process. In order to reduce the impact of experience on the results and reduce the time of selecting hyperparameters, in addition to modifying the original classic network structure, researchers are also fundamentally modifying the efficiency of deep learning. Some researchers are trying to connect with other machine learning methods (such as compressed sensing, Bayesian theory, etc.) to transform deep learning from engineering trial and error into theoretically guided practice. There are also studies that are trying to explain the effectiveness of deep learning algorithms rather than just treating the entire network as a black box. At the same time, researchers are also establishing another machine learning problem for hyperparameters, namely meta learning, to reduce the difficulty and randomness of the process of selecting hyperparameters.
Third, as a large number of research results are released, more algorithms are applied to products. In addition to some small-scale companies that have successively developed image generation applets, large companies are also competing to seize the high ground of deep learning. Internet giants Google, Facebook and Microsoft have successively established deep learning development centers. Chinese Internet companies Baidu, Alibaba, Tencent, JD.com and ByteDance have also established their own deep learning research centers. Some unicorn companies based on deep learning technology, such as DeepMind, SenseTime, Megvii, etc., have also stood out from a large number of competitors. Since 2019, deep learning research in the industry has gradually shifted from focusing on paper publication to implemented projects. For example, Tencent AI Lab optimizes video playback, and the pulmonary nodule screening produced by Yitu has been piloted in some hospitals.
Fourth, with the gradual popularization of 5G technology, deep learning will be embedded into daily life together with cloud computing. The reason why deep learning technology has been difficult to implement is the lack of computing resources. The cost of a supercomputer equipped with a graphics card can reach 500,000 yuan, and not all companies have sufficient funds and talents to make full use of these equipment. With the popularization of 5G technology and the blessing of cloud technology, companies can obtain computing resources directly from the cloud at low cost through leasing. Companies can upload data to the cloud and receive calculation results back from the cloud in near real-time. A slew of emerging startups are figuring out how to take advantage of this infrastructure: they have assembled a team of computer scientists and data scientists to provide deep learning algorithm support and hardware support to other companies. This allows some industries that previously had little to do with computer technology (such as manufacturing, service industries, entertainment industries, and even the legal industry) to no longer need to define problems and develop solutions on their own, but to conveniently enjoy computer technology through cooperation with algorithm companies. Professional support from the technology industry makes it easier to obtain the empowerment of deep learning.
Development status and trends of deep learning
