You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Ingester stuck in LEAVING when its ring entry has 0 tokens #7899
Describe the bug #7122 changed the LEAVING recovery check in Lifecycler.initRing to len(instanceDesc.Tokens) != 0 https://github.com/cortexproject/cortex/blame/master/pkg/ring/lifecycler.go#L805C55-L805C63, so ingesters that lost only some of their tokens can rejoin. An ingester whose entry is LEAVING with 0 tokens still skips that branch and keeps LEAVING with 0 tokens. autoJoin never runs, because it only runs from PENDING, and /ready keeps returning this instance owns no tokens. Restarting the pod or forgetting the instance doesn't get it out of this state.
Expected behavior
A LEAVING entry with 0 tokens should go back to PENDING so autoJoin can generate new tokens.
Proposed Fix
if instanceDesc.State == LEAVING {
if len(instanceDesc.Tokens) == 0 {
// No tokens to keep; restart the lifecycle so autoJoin generates new ones.
instanceDesc.State = PENDING
} else if i.autoJoinOnStartup {
instanceDesc.State = i.getPreviousState()
} else {
instanceDesc.State = PENDING
}
}
Activity
Ring: don't publish instance state before tokens are generated
Ring: don't publish instance state before tokens are generated