Google ToolGrad Generates Agent Datasets Answer-First Using Textual Gradients
Google introduced ToolGrad at ACL 2026, a synthetic data framework that generates verified API execution chains before generating matching user prompts. Fine-tuning an open Gemma-3-12B model on just 500 ToolGrad trajectories enabled it to achieve an 83.1 score on the Berkeley Function Calling Leaderboard, matching Gemini 2.5 Pro and outperforming GPT-5.
Impact: High
Why it matters
You can generate high-quality agent tool-use datasets with near-100% pass rates and fine-tune compact local models to match top frontier APIs.
TL;DR
- 01Answer-first generation eliminates DFS trial-and-error, reaching near-100% data pass rates for tool-use datasets.
- 02Textual gradients from parallel execution reports guide valid long-horizon API chaining.
- 03Gemma-3-12B fine-tuned on 500 ToolGrad samples scored 83.1 on BFCL, outperforming GPT-5 and matching Gemini 2.5 Pro.
Key facts
- ToolGrad-12B BFCL Score
- 83.1 (self-reported)
- Gemini 2.5 Pro BFCL Score
- 83.2
- Claude 4.5 Opus BFCL Score
- 82.8
- GPT-5 BFCL Score
- 74.4
- Data Generation Pass Rate
- Near 100%
- ToolBench API Database
- 16,000+ real-world APIs
The Answer-First Inversion
Conventional tool-use dataset generation relies on query-first depth-first search (DFS) exploration across vast tool pools. This approach is inherently inefficient: agents waste significant token budgets wandering dead-end API paths. ToolGrad, presented at ACL 2026 by Google XR researchers, reverses the pipeline by generating the ground-truth tool-use chain first. Because a concrete, executed sequence of tool calls provides deterministic parameters and outputs, annotating the matching user query requires only a single Large Language Model (LLM) step, achieving an almost 100% generation pass rate.
Textual Gradients and Four-Module Architecture
ToolGrad builds on the concept of textual gradients from TextGrad, treating descriptive natural language execution feedback as directional signals to iteratively construct multi-API workflows. The framework operates across four core modules:
- API Proposer: Filters a broad API repository (such as ToolBench's 16,000+ real-world APIs) down to viable candidates that extend the current chain.
- API Executors: Executes selected API candidates in parallel to produce detailed runtime reports.
- API Selector: Evaluates runtime logs, identifies the best-performing API call as a textual gradient, and appends it to the trajectory.
- LLM Updater: Adjusts the synthetic prompt and model response to mirror the newly extended API sequence.
Benchmark Results on BFCL
Using gemini-2.5-flash-lite as a generator, the researchers built ToolGrad-500, a compact dataset of 500 trajectories, and fine-tuned Gemma-3 models (1B, 4B, and 12B). Evaluated on the Berkeley Function Calling Leaderboard (BFCL) against out-of-distribution tools, ToolGrad-12B scored 83.1. This score surpassed gpt-5 (74.4), claude-4.5 Opus (82.8), open-weight baselines like ToolACE and Hammer-2.1-7B, and matched gemini-2.5-pro (83.2), confirming that student models can outperform their teacher when trained on verified execution chains.
✓ When to use
- Generating synthetic function-calling fine-tuning datasets for local models like Gemma-3.
- Orchestrating long-horizon agent trajectories across multi-step external API dependencies.
- Distilling tool-calling capabilities from expensive proprietary models into compact open-weight models.
✕ When NOT to use
- Scenarios where tool APIs cannot be safely executed or sandboxed during synthetic data generation.
- Simple single-turn prompt-response tasks requiring no external tools or multi-step execution.
What to do today
- Audit existing synthetic agent data pipelines to test generating verified API execution traces before prompting.
- Benchmark local Gemma-3-12B against proprietary APIs on internal tool schemas using the BFCL test protocol.
- Incorporate runtime execution feedback (textual gradients) to prune dead-end trajectories in agent fine-tuning runs.
Sources