Talkie: a 13B vintage language model from 1930 (talkie-lm.com)

Article summary

The article introduces Talkie, a 13B vintage language model trained on pre-1931 English text, and explores its capabilities and limitations. The model is designed to simulate conversations with someone from the past and can be used to study the evolution of language and ideas. The authors also discuss the challenges of training vintage language models, including data quality and temporal leakage. The model is available for use and collaboration, with plans to scale up and improve its performance.

Main themes

• vintage language models
• historical text analysis
• conversational AI
• temporal leakage
• data quality
• AI development

What commenters say

• The idea of creating a conversational AI that can simulate talking to someone from the past is not new and has been envisioned by pioneers in the field, including Steve Jobs.
• Recreating a historical figure like Aristotle through a language model is impossible due to the lack of available data and the complexity of reversing entropy.
• Some commenters argue that training a language model on historical data can help reduce modern bias and emulate the voice and worldview of the era, while others are skeptical about the model's ability to truly understand the context.
• The ability of a vintage language model to come up with new ideas or inventions, such as General Relativity, is seen as a strong test of its true cognition and lack of training data contamination.
• There are differing opinions on the potential of language models to simulate human-like conversation, with some seeing it as a solved problem and others arguing that it is still an unsolved problem due to safety and reliability concerns.
• The use of vintage language models can provide a unique perspective on the evolution of language and ideas, and can be a useful tool for researchers and historians.