Are OpenAI and Anthropic losing money on inference? (martinalderson.com)

Article summary

The article analyzes the costs of running AI inference at scale, specifically looking at the costs of input processing and output generation. It suggests that input processing is relatively cheap, while output generation is more expensive. The author estimates that the cost of input processing is around $0.001 per million tokens, while output generation costs around $3 per million tokens. This cost asymmetry has implications for the profitability of different AI applications.

Main themes

• AI Inference Costs
• Input vs Output Processing
• Profitability of AI Applications
• Cloud Computing Economics
• Machine Learning Business Models

What commenters say

• The cost of training AI models is a significant factor that should be included in any analysis of the profitability of AI companies.
• The marginal cost of inference is the most relevant factor in determining the profitability of AI applications, and training costs are not directly relevant to this calculation.
• The economics of AI inference are similar to those of cloud computing, with significant margins to be made by companies that can scale efficiently.
• The cost of AI inference will continue to decrease as technology improves, making it more profitable for companies to offer AI services.
• Some commenters argue that the article's analysis is flawed because it only considers the cost of inference and not the cost of training AI models.
• Others argue that the cost of training AI models is a one-time cost that can be amortized over many users, and that the marginal cost of inference is the key factor in determining profitability.
• There is disagreement about whether the article's conclusions about the profitability of AI applications are realistic, with some commenters arguing that the costs of training and deploying AI models are too high.
• Some commenters suggest that the AI industry is likely to see significant consolidation and competition in the coming years, which could impact the profitability of AI companies.