You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A follower can forward a write to itself forever: the HTTP resolver has the ambiguity #6183 fixed for client routing, on the one path that redirects automatically #6191
Found while implementing #6183 (PR #6187). That issue fixed the client-routing resolver: an address two peers both resolve to identifies neither, so it is no longer advertised. The HTTP resolver next to it has the identical flaw, and it feeds a path that redirects without a human in the loop.
The defect
RaftHAServer.resolveHttpAddress (ha-raft, line 1816) falls back to deriveHttpAddressWithWarning (line 1907), which combines the peer's Raft host with this node's HTTP port. On a cluster whose nodes differ by port rather than by host - several nodes on one machine - every peer derives to the same address, so a follower resolves the leader's HTTP address to its own endpoint.
RaftReplicatedDatabase.forwardCommandToLeaderViaRaft (line 2585) then does:
The POST lands back on this node. PostCommandHandler runs the command, reaches RaftReplicatedDatabase.command, sees !isLeader(), and forwards again - to itself. Nothing breaks the cycle; it is bounded only by HTTP timeouts and the HTTP worker pool. Every hop consumes a request thread on a node that is already failing to make progress.
Every write forwarded from a follower is affected, not only DDL: the branch is queryEngine.isExecutedByTheLeader() || analyzed.isDDL() || !analyzed.isIdempotent().
Why this is not already covered
The comment above the forward reads:
// If this node becomes the leader while waiting, getLeaderHttpAddress() returns its own address and the POST to self executes locally.
That describes the benign case - this node became the leader, so the self-POST executes locally and terminates. It reads like coverage of "the address resolved to me", and it is not: in the ambiguous case this node is still a follower when the request arrives.
The guard already exists elsewhere in this module.ArcadeStateMachine.triggerSnapshotDownload() (line 2587) does exactly the comparison the forward path is missing:
if (leaderHttpAddr.equals(localHttpAddr)) {
// "Refusing a snapshot resync: the resolved leader address %s is this node's own." (issue #6111)return;
}
So the same hazard is guarded on the resync path and unguarded on the forward path - a divergence rather than an oversight, which is the shape this project usually wants removed rather than documented.
Reachability
Needs a cluster whose peers share a host and whose http port is not declared in arcadedb.ha.serverList. That is narrower than the #6183 case, because http is the third positional field and the docs call it required for forwarding, so most deployments declare it. But host:raftPort alone is documented as valid (Trailing fields are optional), and localhost:2434,localhost:2435,localhost:2436 is the obvious thing to type for a local three-node cluster. On Kubernetes, where hosts differ, the derive is correct and nothing changes.
Suggested fix
Refuse rather than loop, and say why - the same shape as #6111's resync guard and #6183's routing guard:
In forwardCommandToLeaderViaRaft, if the resolved leader address equals this node's own HTTP address and this node is not the leader, throw ServerIsNotTheLeaderException instead of POSTing. The caller then gets the typed, actionable error the HTTP and gRPC layers already know how to report, rather than a hang. Keep the existing behaviour when this node is the leader (the local execution the current comment describes).
Consider hoisting the check into getLeaderHttpAddress()/resolveHttpAddress so triggerSnapshotDownload's hand-rolled copy can go away and any future caller inherits it. This needs care: unlike the routing table, these addresses also feed cluster reporting (getReplicaAddresses(), getStats()) and peer-to-peer snapshot transfer, where a best-effort address is still worth showing and nothing auto-redirects onto it. Follow-ups from #6091: a derived routing address can point a client back at the refusing node, and only graphBatchLoad names the leader #6183 deliberately left those alone for that reason. A resolver-level guard would have to distinguish "advertise" from "dial", or the check stays at the two dial sites.
Whichever way, log the one-time WARNING naming arcadedb.ha.serverList's http field, next to the derive warning that already fires.
A regression test can follow Issue6183AmbiguousRoutingIT: an in-process cluster with undeclared HTTP ports, a write issued on a follower, asserting it fails fast with a typed error instead of timing out.
Found while implementing #6183 (PR #6187). That issue fixed the client-routing resolver: an address two peers both resolve to identifies neither, so it is no longer advertised. The HTTP resolver next to it has the identical flaw, and it feeds a path that redirects without a human in the loop.
The defect
RaftHAServer.resolveHttpAddress(ha-raft, line 1816) falls back toderiveHttpAddressWithWarning(line 1907), which combines the peer's Raft host with this node's HTTP port. On a cluster whose nodes differ by port rather than by host - several nodes on one machine - every peer derives to the same address, so a follower resolves the leader's HTTP address to its own endpoint.RaftReplicatedDatabase.forwardCommandToLeaderViaRaft(line 2585) then does:The POST lands back on this node.
PostCommandHandlerruns the command, reachesRaftReplicatedDatabase.command, sees!isLeader(), and forwards again - to itself. Nothing breaks the cycle; it is bounded only by HTTP timeouts and the HTTP worker pool. Every hop consumes a request thread on a node that is already failing to make progress.Every write forwarded from a follower is affected, not only DDL: the branch is
queryEngine.isExecutedByTheLeader() || analyzed.isDDL() || !analyzed.isIdempotent().Why this is not already covered
The comment above the forward reads:
That describes the benign case - this node became the leader, so the self-POST executes locally and terminates. It reads like coverage of "the address resolved to me", and it is not: in the ambiguous case this node is still a follower when the request arrives.
The guard already exists elsewhere in this module.
ArcadeStateMachine.triggerSnapshotDownload()(line 2587) does exactly the comparison the forward path is missing:So the same hazard is guarded on the resync path and unguarded on the forward path - a divergence rather than an oversight, which is the shape this project usually wants removed rather than documented.
Reachability
Needs a cluster whose peers share a host and whose
httpport is not declared inarcadedb.ha.serverList. That is narrower than the #6183 case, becausehttpis the third positional field and the docs call it required for forwarding, so most deployments declare it. Buthost:raftPortalone is documented as valid (Trailing fields are optional), andlocalhost:2434,localhost:2435,localhost:2436is the obvious thing to type for a local three-node cluster. On Kubernetes, where hosts differ, the derive is correct and nothing changes.Suggested fix
Refuse rather than loop, and say why - the same shape as #6111's resync guard and #6183's routing guard:
forwardCommandToLeaderViaRaft, if the resolved leader address equals this node's own HTTP address and this node is not the leader, throwServerIsNotTheLeaderExceptioninstead of POSTing. The caller then gets the typed, actionable error the HTTP and gRPC layers already know how to report, rather than a hang. Keep the existing behaviour when this node is the leader (the local execution the current comment describes).getLeaderHttpAddress()/resolveHttpAddresssotriggerSnapshotDownload's hand-rolled copy can go away and any future caller inherits it. This needs care: unlike the routing table, these addresses also feed cluster reporting (getReplicaAddresses(),getStats()) and peer-to-peer snapshot transfer, where a best-effort address is still worth showing and nothing auto-redirects onto it. Follow-ups from #6091: a derived routing address can point a client back at the refusing node, and only graphBatchLoad names the leader #6183 deliberately left those alone for that reason. A resolver-level guard would have to distinguish "advertise" from "dial", or the check stays at the two dial sites.arcadedb.ha.serverList'shttpfield, next to the derive warning that already fires.A regression test can follow
Issue6183AmbiguousRoutingIT: an in-process cluster with undeclared HTTP ports, a write issued on a follower, asserting it fails fast with a typed error instead of timing out.