[Feature Request] Configuration to Limit Input Tokens per Request #2637

Is your feature request related to a problem? Please describe.

Yes. Currently, when in a long chat session, opencode appears to send the entire chat history with every new prompt. For very long sessions (e.g., sessions with over 1 million tokens of history), this results in significant and often unnecessary costs.

For many tasks, only the most recent portion of the conversation is relevant for generating the next response. For example, I may only need the last 50,000 tokens of context, not the full 1 million. As it stands, I am paying for the processing of the entire chat history on every single request, which is financially wasteful and increases API latency. There is currently no way to control this behavior without manually clearing the session, which causes the entire local context to be lost.

Describe the solution you'd like

I would like a configuration setting in opencode.json that allows users to specify a maximum number of input tokens to be sent to the model for any given request.

For example, a user could set:

{
  "maxInputTokens": 50000
}

When this setting is active, the opencode client would intelligently truncate the chat history before sending it to the API, ensuring that only the most recent and relevant parts of the conversation (up to the specified token limit) are included in the payload.

This would give users direct control over their API costs and latency without forcing them to lose the valuable context available locally in the terminal.

As LLM models continue to increase their maximum context window, I imagine the need for this type of feature will only increase over time.

Describe alternatives you've considered

1. Manually clearing the chat history (/clear): This is a destructive action. It forces the user to erase valuable context that they may want to scroll back and refer to locally. The goal is to preserve the local chat history while limiting the data sent over the API.
2. Continuing as is: This is not a sustainable option for users who engage in long, complex coding sessions, as costs can quickly become prohibitive.

Additional context

This feature would be a critical improvement for power users and anyone on a budget. It provides essential control over token consumption, which is a key factor in using any LLM-based tool.

This type of context management is a standard feature in many other LLM interfaces (for example, aider has a max_input_tokens advanced model setting), and adding it to opencode would make it an even more powerful and economically viable tool for professional developers. Thank you for considering this feature.

Slight variation

Perhaps a better name should be maxChatInputLength. And instead of using tokens, we would just do some kind of simple truncation of the chat session by character length. All of the context related to the system prompt, tools, and what-not can be kept at 100%, but the chat would be truncated. This variant captures the spirit of the idea, which is to limit the massive amounts of waste that are typical for many of our agentic sessions which require only a shallow context.