Testing application security vulnerabilities using agentic Large Language Models
A developer spent fifteen hundred dollars evaluating whether LLM agents could successfully identify and exploit custom application vulnerabilities. While they solved basic issues, they struggled with complex, multi-step logic flaws. Use structured pentesting suites for automated security evaluation.
Why it matters
You can use structured agent loops to quickly audit basic security flaws, but you must enforce strict token budget limits to avoid unexpected API bills.
TL;DR
- 01Set a hard budget limit on security-oriented agent loops to prevent recursive call inflation
- 02Isolate test databases and environments completely when allowing agentic tools to execute write operations
- 03Audit system and controller files individually rather than scanning broad codebases in single context windows
Experiment Parameters
The developer conducted 10 runs for each model. The experiment totaled $1,500 in costs. Models were tested on a custom React Native/FastAPI application designed with a specific Firebase access control flaw.
Performance Discrepancies
Results varied significantly between models. deepseek-v4-pro achieved a 3/10 solve rate. The study highlighted that while models effectively identify common injection patterns, they struggle with logical 'broken access control' flaws, often fixating on the wrong stack components.
Cost and Guardrails
Agentic loops were highly inefficient. Models often became stuck in repetitive action loops, burning tokens while failing to pivot their strategy. The developer noted that 'high thinking' models consumed significant tokens without a proportional increase in success rate.