Skip to content

Tool Defs total is a never-pruned union across sessions, then multiplied by toolsRatio #797

Open

Description

@jjjermiah

A session with 34 tool definitions showed 62K for Tool Defs. Those 34 tools measure 13,062 with MC's own estimator. The rest of the number is tools that were not in the request.

Two effects stack.

1. The measurement set is a lifetime ledger. recordToolDefinition() (packages/plugin/src/features/magic-context/tool-definition-tokens.ts:170) stores each tool under ${providerID}/${modelID}/${agentName}, keyed by toolID. There is no session in the key, and the module docstring (:4-8) says so on purpose. getMeasuredToolDefinitionTokens() (:255) sums the whole map, and nothing removes a toolID when that tool stops being registered. loadToolDefinitionMeasurements() (:128) reloads every row from tool_definition_measurements at startup, so the stale set also survives restarts. Rows come from the tool.definition hook in index.ts:671-689.

Both the per-flight measurement and the hook attribution look correct. The problem is that the total is a union over every session, directory and OpenCode version that ever ran on that model and agent.

2. A calibration ratio multiplies the union. estimateFinalWireInputTokens() (packages/plugin/src/hooks/magic-context/final-wire-token-estimate.ts:159-162) returns Math.round(measuredToolDefinitions * calibration.toolsRatio), and the sidebar uses that value (packages/plugin/src/plugin/rpc-handlers.ts:464-465). For the 62K session toolsRatio is 1.54, so 40,271 measured becomes 62,000 displayed.

Numbers from my store, agent orchestrator:

bucket rows row sum rows in the session their sum
model A 108 40,271 34 13,062
model B 137 33,373 34 13,062

The 74 extra rows in model A include lsp, task and todowrite (built-ins OpenCode 2 no longer ships, last measured 2026-09-23), bash (AFT's earlier name for shell), openchamber, and tools from unrelated repos: rdepot_*, k8s_math_api, regress_*, pr_*, k6_*, splunk_*, positron_*. Rows keep being added today (a k6_* row at 23:59 last night, from a work repo), and bucket size tracks age: the oldest pair holds 171 rows, a pair created a week ago holds 74.

To reproduce:

select provider_id, model_id, agent_name, count(*), sum(token_count)
from tool_definition_measurements
group by 1, 2, 3;

Then compare one bucket against the tool ids the sidebar session actually sends (OpenCode 2 with Code Mode on: execute plus the tools registered with codemode: false).

Suggested fix. Measure the set that is serialized for the flight. OpenCode 2 hands plugins the exact post-split tools record on the session.context hook, so the payload set is available without a second mechanism. If keeping tool.definition, prune ids not seen in the last N flights of that key and total the pruned set, so removing a tool also shrinks the number.

Environment: plugin 0.49.23, OpenCode 2.1.4, darwin arm64, doctor PASS 18 / WARN 1 / FAIL 0.

One unrelated thing seen while running doctor there: it reports Local embedding runtime unverified: no installed plugin tree found to inspect on OpenCode 2 because getOpenCodePluginCacheRoots() looks under ~/.cache/opencode/packages/..., which is the OpenCode 1 layout. OpenCode 2 installs to ~/.cache/opencode/npm/<name>@<spec>/<timestamp>/node_modules/..., so the probe never finds a tree.

Activity

  1. magic-alfonso commented on Oct 27, 2026

    @magic-alfonso

    Thanks for the precise trace and the numbers. The code confirms it. recordToolDefinition keys measurements by provider, model and agent with no session in the key, writes them through to tool_definition_measurements, and nothing removes a tool id that stopped being sent. getMeasuredToolDefinitionTokens then sums the whole map.

    The total isn't only shown in the sidebar. The same measured value feeds Magic Context's own usage estimate for the next request, so a stale union can make a session look fuller than it is. We're checking exactly which decisions that affects before choosing the fix. Measuring the set actually serialized for the flight, as you suggest, is the direction we'd take. We'll also check the OpenCode 2 plugin cache path that doctor probes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions