Claude Code daily benchmarks for degradation tracking (marginlab.ai)

Article summary

The article introduces a daily benchmark tracker for Claude Code's performance on SWE tasks, aiming to detect statistically significant degradations. The tracker uses a curated subset of SWE-Bench-Pro and runs daily evaluations with the latest Claude Code release and the SOTA model. The goal is to provide a resource for detecting degradations, following a postmortem on Claude degradations published by Anthropic in September 2025. The tracker reports daily, weekly, and monthly pass rates with 95% confidence intervals.

Main themes

• Model Performance
• Degradation Detection
• Benchmarking
• LLM Development
• User Experience

What commenters say

• Some users believe that Claude Code's performance has degraded over time, despite the company's assurances that model quality is not reduced due to demand or server load.
• The degradation may be due to changes in the model or infrastructure, rather than user perception or prompting techniques.
• Benchmarks are useful for measuring model performance, but may not capture the full complexity of real-world usage and user experience.
• The company's claims of not reducing model quality are met with skepticism by some users, who suspect that cost-cutting measures may be behind the perceived degradation.
• User perception of model performance can be influenced by various factors, including changes in prompting techniques, workflows, and personal biases.
• Some users argue that the degradation may be a result of the company's efforts to optimize costs and reduce compute allocations, rather than a deliberate attempt to reduce model quality.
• The phenomenon of model degradation may be exaggerated or imagined by users, and more data and evaluation are needed to determine the reality of the situation.