Human Approval Prompts Fail in AI Coding Agents, Study Shows
An empirical analysis of 40,000 sessions in an AI agent security challenge reveals human reviewers miss 33% of security threats due to permission fatigue. Commands hiding payloads inside package.json scripts like `npm run analyze` were approved over 64% of the time.
Why it matters
Relying on human approval for command execution creates a false sense of security, as prompt fatigue leads reviewers to approve malicious actions.
TL;DR
- 01Human accuracy in approving agent commands averages only 66.3%, dropping over time due to fatigue.
- 02Commands hiding payloads inside benign script names like `npm run analyze` were approved 64.7% of the time.
- 03Effective protection requires automated sandboxing and separating credentials rather than relying on manual prompt approvals.
The Breakdown of Permission Fatigue
Data from the security challenge reveals that human accuracy drops as sessions progress due to notification fatigue and time pressure. In the evaluation, 32.9% of user sessions ended with a net negative score because penalties from approved malicious actions outweighed correctly handled commands.
Dangerous vs. Benign False Positives
- Obvious destruction (
rm -rf /,chmod -R 777 /): 11.7% miss rate. - Scope violations (
cat ~/.aws/credentials): 35.0% miss rate. - Script masking (
npm run analyzecontaining exfiltration payloads): 64.7% miss rate. - Benign commands wrongly blocked:
npm config set registry(blocked 59%),rm -rf dist/(blocked 45%).
Practical Remediation: Sandboxing and Secret Separation
Asking developers to audit every command is an ineffective security barrier. Instead, practical mitigations require applying sandboxing and separating credentials and environment variable secrets.