Testing application security vulnerabilities using agentic Large Language Models
A developer spent fifteen hundred dollars evaluating whether LLM agents could successfully identify and exploit custom application vulnerabilities. While they solved basic issues, they struggled with complex, multi-step logic flaws. Use structured pentesting suites for automated security evaluation.
Why it matters
You can use structured agent loops to quickly audit basic security flaws, but you must enforce strict token budget limits to avoid unexpected API bills.
When developing web applications with tools like Cursor or Claude Code, security testing is often treated as an afterthought or left to traditional static application security testing tools. This analysis details an experiment where a custom vulnerable web application was targeted by LLM agents to evaluate their automated pentesting capabilities. The author set up specific challenges ranging from classic SQL injections to complex logical bypasses, aiming to understand if agentic frameworks can replace human security researchers.
The architecture utilized various API models, coordinating multiple LLM calls to execute autonomous hacking tasks. Instead of simple prompting, the setup employed agentic loops where the model could write code, execute payloads against target servers, analyze responses, and adapt its strategy dynamically. By spending fifteen hundred dollars, the developer collected empirical data on success rates, cost-per-vulnerability, and critical bottlenecks.
Under the hood, the experiment highlights a clear boundary in agent capabilities. Standard injection attacks (SQL injection, Cross-Site Scripting) and outdated library vulnerabilities were solved quickly because their signatures are heavily represented in pre-training corpora. However, logical vulnerabilities—such as bypassing rate limits through race conditions or manipulating session states—consistently defeated the agents. The primary issue was context fragmentation: as the agent generated more tools calls, the environment state expanded beyond what the context window could coherent evaluate, leading to expensive, repetitive action loops.
For developers looking to use LLM agents for security testing, this means you should not rely on them for end-to-end logical assessments. Instead, integrate them directly into your continuous integration and continuous deployment pipelines using highly scoped prompts. For example, write custom system instructions that direct your agent to audit a single, specific controller file for access control bugs, rather than asking it to scan the entire repository at once.
The experiment also shows that costs escalate rapidly during long-running agent loops. Without strict execution limits, agents will continuously try variations of the same failed payload, burning tokens at a high price point. Always implement maximum iteration guardrails and strict system prompt length constraints to prevent exponential API bills.
While LLM agents are highly efficient at detecting low-hanging syntactic and structural security flaws, human-guided exploration remains essential for uncovering deep logical and architectural vulnerabilities.
Key takeaways
- 01Set a hard budget limit on security-oriented agent loops to prevent recursive call inflation
- 02Isolate test databases and environments completely when allowing agentic tools to execute write operations
- 03Audit system and controller files individually rather than scanning broad codebases in single context windows