Skip to content

Tower mode: a worker dies on its first 429 and the tower keeps spawning more #4682

Open

Description

@mikael-titinovskii

What happens

In tower mode, a spawned worker that gets a 429 gives up almost at once and disappears from the roster. The main agent doing similar work visibly retries first. The tower also keeps spawning workers into the same rate limit, and a worker that hit it is just marked dead, so I end up resuming agents by hand.

What I expected

A worker that hits a 429 gets retried, or at least is not marked dead before the normal retry path runs. And the tower should hold off spawning for a while instead of pushing more requests at a provider that is already saying no.

How to reproduce

  1. Turn on tower mode (KIMI_CODE_EXPERIMENTAL_TOWER=1).
  2. Plan a mission and spawn 3-4 workers.
  3. Use a model whose key is close to its limit (a shared key or a free tier is enough).

The workers that get 429s die one by one, the tower does not slow down, and the limit keeps firing.

Why

Three things add up:

  • packages/agent-core-v2/src/features/tower/towerRateLimitService.ts already has reportRateLimited() and reportSuccess(), and acquire() already reads the pause they set. Nothing in production calls the two reporters, only the unit test does. So the 60 s spawn pause and the smaller concurrency budget never fire, and TowerStatus always reports blockedUntil: null.
  • packages/agent-core-v2/src/features/tower/tools/spawn/spawnTool.ts runs a worker once. If the run fails, the task settles failed and the death is stored. There is no retry. Swarm already handles this: features/swarm/session/agentRunBatch.ts spots the rate-limit error, backs off and requeues the agent.
  • My [loop_control] max_attempts_per_step is 3, which is the old default. With 3, retries are 500 ms and 1 s apart, so even a worker that does retry is gone about 1.5 s after the first 429. A 429 storm lasts minutes.

Setup

kimi-code 2.4.1 on Windows 11 (10.0.26200) x64, model openrouter/deepseek/deepseek-v4.1-flash-1017, max_attempts_per_step = 3, max_running_tasks = 4.

Fix

I have one working locally: wire the two reporters to turn failures, so a rate-limited turn sets the pause and a successful turn clears it, and retry a rate-limited worker in place with 3 s / 6 s / 12 s backoff and a cap of 3 retries before falling back to today's death path. Both come with tests. I would like to open a PR once you are ok with the direction.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions