GLM 5.2 beats Claude in our benchmarks (semgrep.dev)

Article summary

GLM 5.2, an open-weight model from Zhipu AI, has been found to outperform Claude in benchmarks, achieving a 39% F1 score in IDOR detection. The model was tested in a simple Pydantic AI harness with a standard IDOR prompt, without the scaffolding of a purpose-built harness. This result surprised the testers, as GLM 5.2 was not expected to perform as well as it did. The test was intended to compare the performance of different models in vulnerability detection, with and without the support of a harness.

Main themes

• AI model performance
• Vulnerability detection
• Open-weight models
• Export controls
• AI regulation
• Cybersecurity

What commenters say

• The US government may impose export restrictions on open-weight models like GLM 5.2, potentially limiting their availability and use.
• The development and use of open-weight models could undermine the financial interests of companies like OpenAI and Anthropic, leading to attempts to restrict their use.
• Restricting access to advanced AI models could put US companies at a disadvantage, as attackers will still be able to access and use these models, regardless of the law.
• The real motive behind attempts to restrict open-weight models may be profit, rather than safety or security concerns.
• The use of open-weight models could lead to a cat-and-mouse game between regulators and developers, with models being shared and used through alternative channels like peer-to-peer networks.
• The development of AI models is a global effort, and restricting access to these models could hinder progress and innovation in the field.
• The US government's actions on AI regulation may be driven by a desire to protect the interests of US companies, rather than a genuine concern for safety or security.
• The use of open-weight models could democratize access to advanced AI capabilities, allowing smaller companies and individuals to compete with larger players in the field.