Too much efficiency makes everything worse (2022) (sohl-dickstein.github.io)

Article summary

The article discusses how increased efficiency can sometimes lead to worse outcomes, citing examples from education, science, and economics. This phenomenon is related to Goodhart's law, which states that when a measure becomes a target, it ceases to be a good measure. The article also draws parallels with overfitting in machine learning, where a model becomes too specialized to the training data and fails to generalize well. The author suggests that recognizing and mitigating this issue can help improve outcomes in various fields.

Main themes

• Efficiency and outcomes
• Goodhart's law
• Overfitting in machine learning
• Proxy metrics
• System robustness
• Optimization trade-offs

What commenters say

• The pursuit of efficiency can lead to diminishing returns and negative consequences when taken too far.
• Goodhart's law is a fundamental principle that applies to many fields beyond economics, including machine learning and social systems.
• The concept of overfitting in machine learning is not directly equivalent to the issues with efficiency and proxy metrics in other fields.
• Optimizing for the wrong metric is a distinct problem from excessive efficiency, and both require careful consideration of robustness and trade-offs.
• Some argue that the article's critique of efficiency is misplaced, and that the real issue is optimizing for the wrong metric or using a flawed model.
• Others suggest that recognizing the limitations of metrics and models is crucial to avoiding unintended consequences and finding useful solutions.
• The importance of slack and redundancy in systems is highlighted as a way to mitigate the risks of over-optimization and increase robustness.
• There is a need for a more nuanced understanding of efficiency and optimization, one that balances competing goals and considers the potential downsides of extreme efficiency.