Were RNNs all we needed? (arxiv.org)

Article summary

The article revisits sequence modeling from a historical perspective, focusing on Recurrent Neural Networks (RNNs) and their potential to be simplified and parallelized during training. The authors propose minimal versions of LSTMs and GRUs, which use fewer parameters and achieve competitive performance on various tasks. The paper suggests that RNNs can be trained as efficiently as modern alternatives, but their performance is only competitive at small scale. The authors' goal is to explore the potential of RNNs in the context of current deep learning architectures.

Main themes

• RNNs vs Transformers
• Sequence modeling
• Parallelization and efficiency
• Deep learning architectures
• Model scalability

What commenters say

• The simplification of RNNs by removing certain dependencies on the hidden state allows for parallel training and competitive performance on small-scale tasks.
• The performance of RNNs is limited to small-scale tasks and may not extend to larger models or context sizes.
• The choice of architecture is not as important as the dataset and the ability to embed it on a curve, as long as the curve is sufficiently expressive.
• Finding the most efficient architecture is crucial due to the high cost of running GPUs at scale and the need for continuous retraining.
• Some argue that the focus on RNNs is misguided and that new approaches are needed for few-shot learning and out-of-distribution patterns.
• Others believe that the key to progress lies in computational efficiencies and the ability to retrain models continuously, rather than in new architectures.
• There is a trade-off between model expressiveness and computational or memory efficiency, and the best architecture will depend on the specific constraints of the problem.
• The development of new architectures is influenced by the attention and resources devoted to them, and RNNs may have been overlooked in favor of Transformers.