You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Tower mode: a worker dies on its first 429 and the tower keeps spawning more #4682
In tower mode, a spawned worker that gets a 429 gives up almost at once and disappears from the roster. The main agent doing similar work visibly retries first. The tower also keeps spawning workers into the same rate limit, and a worker that hit it is just marked dead, so I end up resuming agents by hand.
What I expected
A worker that hits a 429 gets retried, or at least is not marked dead before the normal retry path runs. And the tower should hold off spawning for a while instead of pushing more requests at a provider that is already saying no.
How to reproduce
Turn on tower mode (KIMI_CODE_EXPERIMENTAL_TOWER=1).
Plan a mission and spawn 3-4 workers.
Use a model whose key is close to its limit (a shared key or a free tier is enough).
The workers that get 429s die one by one, the tower does not slow down, and the limit keeps firing.
Why
Three things add up:
packages/agent-core-v2/src/features/tower/towerRateLimitService.ts already has reportRateLimited() and reportSuccess(), and acquire() already reads the pause they set. Nothing in production calls the two reporters, only the unit test does. So the 60 s spawn pause and the smaller concurrency budget never fire, and TowerStatus always reports blockedUntil: null.
packages/agent-core-v2/src/features/tower/tools/spawn/spawnTool.ts runs a worker once. If the run fails, the task settles failed and the death is stored. There is no retry. Swarm already handles this: features/swarm/session/agentRunBatch.ts spots the rate-limit error, backs off and requeues the agent.
My [loop_control] max_attempts_per_step is 3, which is the old default. With 3, retries are 500 ms and 1 s apart, so even a worker that does retry is gone about 1.5 s after the first 429. A 429 storm lasts minutes.
Setup
kimi-code 2.4.1 on Windows 11 (10.0.26200) x64, model openrouter/deepseek/deepseek-v4.1-flash-1017, max_attempts_per_step = 3, max_running_tasks = 4.
Fix
I have one working locally: wire the two reporters to turn failures, so a rate-limited turn sets the pause and a successful turn clears it, and retry a rate-limited worker in place with 3 s / 6 s / 12 s backoff and a cap of 3 retries before falling back to today's death path. Both come with tests. I would like to open a PR once you are ok with the direction.
Activity