I built a vulnerable app and spent $1,500 seeing if LLMs could hack it (kasra.blog)

Article summary

The author built a vulnerable app to test if large language models (LLMs) could hack it, spending $1,500 on the experiment. The app had a secure API but used Firebase as the data layer, which was the target of the exploit. The author tested various LLMs, including GPT, Deepseek, and Claude, with mixed results. The experiment aimed to reproduce a common class of exploits found in Firebase and Supabase apps.

Main themes

• LLM security testing
• Vulnerable app exploitation
• Firebase security
• AI model limitations
• Guardrails and constraints

What commenters say

• The guardrails in some LLMs, such as Anthropic, can prevent them from solving problems, even if it's for legitimate purposes like pentesting.
• The restrictions in LLMs may be a business decision to upsell more advanced models or features, rather than a purely security-driven choice.
• Some argue that LLMs should not have guardrails that prevent them from being used for certain tasks, as it can limit their usefulness and create an exclusionary system.
• Others believe that guardrails are necessary to prevent malicious use of LLMs and that vendors have the right to choose what capabilities to offer to their customers.
• The discussion around LLMs and security raises questions about the balance between usefulness and safety, and who should determine what capabilities are allowed or restricted.
• There is a concern that the current approach to LLM security may lead to a situation where only certain individuals or organizations have access to advanced models, creating a power imbalance.