Routing Before Reasoning
Scaling tool use in LLM agents by retrieving a bounded candidate set before expensive reasoning.
- Status
- Published · 2026
- Role
- Independent Researcher
- Timeline
- Sep 2026
- Stack
- Jev · gpt-5.6-sol · Python · deterministic tool benchmark
01Research question
Tool-using agents are often given every available tool schema at inference time. That preserves the full capability set, but the expensive reasoning model must repeatedly process a growing catalog even when only one tool is relevant.
This study asks whether capability retrieval can be separated from reasoning: a lightweight router first ranks compact tool summaries, then the downstream model receives full schemas for only the top-k candidates. It measures where that separation saves context and cost, where it introduces new failures, and whether router confidence can choose candidate-set size dynamically.
02Experimental design
The controlled benchmark contains 100 deterministic mock tools and 78 fixed, primarily single-step tasks. Each task is evaluated in nested toolspaces from 5 to 100 tools, keeping its required tool present while adding a frozen sequence of near misses and domain-interleaved distractors.
The baseline sends every full tool definition to gpt-5.6-sol. Routed conditions use Jev to rank compact summaries, then expose only the top 1 or top 5 full schemas to the same downstream model. A fixed-N ablation at 100 tools tests k = 1, 3, 5, and 10. Execution success, routing recall, downstream selection, tokens, priced cost, latency, and failure stage are recorded from immutable experiment manifests.
03Scaling result
Full-toolspace reasoning remained capable as the catalog grew: execution success was 0.769 at 100 tools. The clearer pressure was economic. Mean downstream context increased from 282 tokens at 5 tools to 2,982 at 100, while priced inference cost rose from $0.00145 to $0.01224 per attempt under the paper's frozen pricing version.
With top-5 pre-routing, downstream context stayed near 290 tokens at 100 tools and cost was $0.00157 per attempt, including router inference. Execution success was 0.756. That point estimate is close to the baseline, but the paired bootstrap interval is too wide to support an equivalence claim.
04Key insight
Retrieval relocates toolspace complexity; it does not remove it. A small candidate budget makes routing coverage decisive. A larger budget recovers more required tools, but gives the reasoner more distractors and more context to process.
At 100 tools, k = 3 and k = 5 reached the same strict recall while producing different execution outcomes. Binary Recall@k therefore does not fully describe the prompt the downstream model sees: candidate identity, ordering, and distractor composition can still change selection and execution.
05Limits and next step
The benchmark uses deterministic mocks, a modest task set, primarily single-step requests, one main router family, and one downstream reasoning model. It does not evaluate authentication, changing external state, multi-call recovery, or natural-language answer quality. Primary cells also have one repetition, so the reported intervals do not capture provider or run-to-run variance.
The tested confidence policy was not reliable enough to allocate candidate budgets: some incorrect top-1 routes carried high confidence, and the low-confidence escalation branch was unused. A stronger next step is to learn escalation from richer signals and validate it across larger catalogs, multi-step tasks, and repeated runs.