How fast is N tokens per second really? (mikeveerman.github.io)

Article summary

The article discusses the concept of tokens per second in the context of language models and provides a tool to visualize and understand the speed of token generation. It highlights the difference in perceived speed between code and text, as well as the impact of context size on token generation. The tool allows users to experiment with different speeds and modes, including code, text, and reasoning models. This helps to internalize the meaning of tokens per second and understand the limitations of language models.

Main themes

• Token generation speed
• Language models
• Context size impact
• Code vs text
• Model limitations
• Agentic workflows

What commenters say

• Some users find the tool helpful in understanding the speed of token generation and its implications for their work.
• The speed of token generation is not the only factor to consider, as quality of output and context size also play a significant role.
• Fast token generation can enable new workflows and use cases, such as parallel requests and branching strategies, but may also introduce new challenges and limitations.
• The value of intelligence over speed is debated, with some users prioritizing the quality of output over the speed of generation.
• The tool's visualization of token generation speed can be striking, but some users may not notice a significant difference between different modes and speeds.
• The current state of language models is compared to the dial-up era of the internet, implying that there is still significant room for improvement in terms of speed and efficiency.
• The importance of considering decoding speed, prefill speed, and context size when evaluating the performance of language models is emphasized.
• Some users are interested in exploring the potential of agentic workflows and parallel processing with fast token generation, while others are more focused on the quality and reliability of the output.