There Will Be a Scientific Theory of Deep Learning (arxiv.org)

Article summary

The article discusses the emergence of a scientific theory of deep learning, which aims to characterize properties and statistics of the training process, hidden representations, and performance of neural networks. It identifies five growing bodies of work that contribute to this theory, including solvable idealized settings, tractable limits, and simple mathematical laws. The authors argue that this theory is best thought of as a mechanics of the learning process, which they term 'learning mechanics'. This theory is expected to provide a deeper understanding of deep learning and its applications.

Main themes

• Deep Learning Theory
• Neural Network Mechanics
• Learning Dynamics
• Scientific Theory of Deep Learning
• Machine Learning History

What commenters say

• The development of deep learning was accelerated by the combination of exponentially more compute power and larger, high-quality datasets.
• The concept of a transformer could have been used on slower hardware earlier, but it would not have achieved the same results due to limited data and compute resources.
• Some argue that the brain's learning process is more similar to reservoir computing than deep learning, and that end-to-end loss optimization is not a biologically plausible mechanism.
• Others propose that the brain's learning process may involve a combination of innate structure and deep learning, with the brain trying to approximate backpropagation with local rules.
• The importance of architecture in neural networks is debated, with some viewing it as a choice of tradeoffs that facilitate learning, while others see it as fundamental to the learning process.
• The development of scalable open-source deep learning frameworks has been a crucial factor in the advancement of deep learning research.
• Some argue that true intelligence requires a high threshold of compute capacity and the ability to deal with complex learning environments, while others propose that injecting sensible inductive bias into models is key to achieving effective learning.