Stealing Reasoning Traces from Proprietary LLM APIs (stolen-thoughts.com)

Article summary

Researchers have discovered a method to extract reasoning traces from proprietary large language models (LLMs) by replaying encrypted traces in weaker models, allowing them to recover the original model's hidden reasoning in plaintext. This method works across models from different providers, including OpenAI, Anthropic, and Google. The attack exploits the fact that LLM providers return encrypted chain-of-thought blocks to clients, which can be replayed across sessions and models.

Main themes

• LLM security
• Reasoning trace extraction
• Model vulnerabilities
• Encryption weaknesses
• AI model interactions

What commenters say

• The ability to replay reasoning traces across models and sessions is a security vulnerability that can be exploited to extract sensitive information.
• Using a per-user encryption key could have prevented the attack, but it would not have stopped plain-text trace extraction attacks.
• Server-side storage of reasoning traces is not a viable solution due to scalability and latency issues, as well as zero data retention requirements.
• The attack is possible because weaker models have weaker safeguards against reasoning replaying, allowing them to decrypt and recite the injected reasoning.
• The providers' decision to allow model switching mid-conversation may have contributed to the vulnerability, as it requires the ability to replay reasoning traces across models.
• The attack does not involve decrypting the payload, but rather getting the server to decrypt it and then prompting the model to repeat back the reasoning in plain text.