What happened after 2k people tried to hack my AI assistant (fernandoi.cl)

Article summary

The author created a website where people could try to hack their AI assistant, Fiu, by sending emails to see if they could get it to reveal the contents of a secrets file. Over 2,000 people sent 6,000 emails, but none were successful in getting Fiu to leak the secrets. The experiment was run with a powerful model, Claude Opus 4.6, which is specifically trained for resistance to prompt injection. The author notes that while the experiment was successful, it does not necessarily mean that the model is completely secure.

Main themes

• AI security
• Prompt injection
• Model robustness
• Experiment design
• Language model vulnerabilities

What commenters say

• The experiment's results are not conclusive, as a small sample size and known exfiltration vector may not reflect real-world scenarios.
• The model's ability to resist prompt injection does not necessarily mean it is usable or useful in practice.
• The experiment's design, including the use of a powerful model and a specific set of rules, may have influenced the results and limited the ability to draw broader conclusions.
• The fact that no one was able to successfully extract the secrets does not mean that the model is completely secure, as more sophisticated attacks or longer-term testing may still reveal vulnerabilities.
• The usability of the model is a separate concern from its security, and a model that is secure but unusable is not useful.
• The experiment highlights the importance of testing AI models with diverse and realistic scenarios to ensure their robustness and security.
• The results of the experiment may not be generalizable to other models or scenarios, and more research is needed to fully understand the security implications of AI models.