Benchmarking Gemini's tool-calling limits (+ the cost of "cheaper" models)

Key takeaways

• Scale your toolsets confidently: If you are building on Gemini 3 Flash, you don't need to over-engineer complex, dynamic tool-filtering or routing layers for toolsets under 100. The model handles extreme tool density and deep sequential execution with near-total reliability.

• Reliability is a cost optimization: Minimizing retries and broken turns is just as critical to cloud budget efficiency as minimizing input/output token costs. High accuracy prevents context-window compounding.

At Fullstory, we are constantly pushing the boundaries of what our in-app AI agents can do. Every new capability means another registered tool, and as your function-calling surface grows, you hit a classic developer dilemma: How many registered tools are too many?

Conventional wisdom suggests that as you bloat an LLM’s context with more and more function definitions, its accuracy will degrade. It gets distracted, suffers from "loss in the middle," or hallucinates bad parameters.

To find the breaking point for our workflows, we designed a stress-test benchmark to measure tool-selection accuracy against increasingly massive toolsets. What we discovered completely flipped our assumptions about model selection. Performance was only half the story. The other half was the hidden economics of production token costs.

How we benchmarked tool-calling accuracy

The goal was simple: test how well a model can find the exact tool it needs as the total number of registered tools grows from 1 to 100.

Benchmark at a glance
Models: Gemini 3 Flash and Gemini 2.5 Flash Lite. Tool counts: 1 to 100 registered tools. Trials: 5 at every tool count, for every phase, except the 5-step test, which we ran on Gemini 3 Flash only. Phases: one call, two calls in order, five calls in order

We ran three distinct phases of the experiment, conducting 5 trials for every single tool count size:

1. The single-turn challenge: The agent is given a prompt where it is expected to call one specific tool on its very first turn out of a pool of N tools.

2. Two-turn challenge: The agent has to successfully call exactly two correct tools in a specific sequence.

3. Five-turn challenge: To test the model's limits and reasoning stamina, the agent is required to successfully execute exactly five correct tool calls in a strict order. If a single tool call in the sequence is wrong or skipped, the entire trial is marked as a failure.

We pitted Google’s next-generation Gemini 3 Flash against Gemini 2.5 Flash Lite to see how a premium powerhouse compares to an efficiency-optimized baseline at handling dense toolsets.

Results: Gemini 3 Flash tool-selection accuracy at 100 tools

When we visualized the data, the contrast between the two models was staggering.

1. Gemini 2.5 Flash Lite: the erratic baseline

When we ran the initial tests on Gemini 2.5 Flash Lite, the graphs showed massive volatility. As the toolset grew past 13 tools, the single-turn success rate began swinging wildly between 40% and 100%. In the multi-step challenge, it suffered severe degradation at higher tool volumes, dropping to a 0% success rate as we approached 100 registered tools.

2. Gemini 3 Flash: high-stamina precision

We expected to see a steady degradation in success rate as the tool count approached 100, especially as we increased the sequential step count. Instead, Gemini 3 Flash obliterated the benchmark:

• Single-step: Achieved a flat 100% success rate across almost the entire spectrum, barring a single anomalous dip at the 63-tool mark, where the model opted for a text response instead of invoking the tool.

• Two steps: Maintained a pristine 100% accuracy rate from 1 tool all the way to 100 tools.

• 5 steps: We assumed a 5-step strict sequence would finally break the model. While not entirely flawless, Gemini 3 Flash showed staggering resilience. Out of 500 total trials, it blinked only twice: once during a trial with 71 registered tools and once during a trial with 81 registered tools (dropping to an 80% success rate for just those two configurations). Aside from those two isolated blips, it maintained a perfect 100% accuracy rate across the rest of the 100-tool spectrum.

The hidden cost of failed tool calls is the "history tax"

This experiment leads to a critical financial takeaway for engineering teams scaling AI agents.

When choosing an LLM, it’s easy to look strictly at price-per-token metrics and assume the cheaper model is always the most cost-effective choice. But raw token pricing ignores a brutal reality of multi-turn conversations: the compounding cost of context.

A failed tool call costs you more than one wasted turn. The tokens generated during that failed execution, including the model's incorrect output, the system's error message, and the prompt asking it to retry, all get appended to the conversation history. Because LLMs reprocess the entire context window on every subsequent turn, you pay for those failed-turn tokens over and over again for the remainder of the conversation.

Modeling the breakeven curve

To understand how this impacts the bottom line, we mapped out a financial simulation comparing the effective cost of a premium model against a lower-cost alternative, using Gemini 3 Flash and Gemini 2.5 Flash Lite as our baseline targets.

C_total = C_baseline + Σ_i=1^F [(T_fail × P_output) + (T_fail × R_i × P_input)]

Where T_fail represents the bloated tokens from a bad call, R_i is the number of remaining turns left in the conversation after failure i, P_input is the input token price, P_output is the output token price, and C_baseline is the cost of the conversation if every tool call had succeeded.

This formula gives you the effective cost of running an agent with the financial penalties of erroneous tool calls baked right in. By running this calculation for both your cheap and expensive models, you can map out their true points of comparison. Finding the exact number of failures (F) at which C_{total_cheap} crosses C_{total_expensive} gives the exact breakeven point, where a higher volume of failed tool calls makes the "cheaper" model effectively more expensive.

When you factor in deep multi-step workflows like our 5 Step test, an unreliable model cascades into multiple failures early on, making it likely that you cross this financial breakeven point.

How to calculate your own effective agent cost

Because tool density, system prompts, and model pricing vary, the breakeven point is dynamic. If you are deciding between two models for your agent architecture, you should run this math yourself using a simple three-step framework:

1. Run the density benchmark: Test your specific toolset size on both models across a statistically significant sample size to find each model's real-world tool failure rate (%).

2. Calculate the history tax: Estimate your average conversation length (total turns) and measure the token overhead generated per failed call.

3. Compute effective cost: Factor the compounding token costs into your projected volume. If your cheaper model’s failure rate sits above its calculated breakeven line for your average conversation length, the premium model is actually your low-cost option.

That effective cost number is what your AI ROI math actually rests on, not the sticker price per million tokens.

Build denser agents without the reliability tax

By benchmarking our agents against realistic worst-case scenarios, we can confidently build richer, feature-dense agent experiences at Fullstory without sacrificing reliability or breaking the bank.