objectstore-plugin: snapshot credentials make one STS call per actor and hit the STS quota at scale #3156

On GCS, the credential exchange downscopes the snapshot transfer account's token once per actor, and each downscope is an STS token exchange. STS allows 6,000 token exchanges per project per minute by default. In a 200,000-actor benchmark, bursts of cold suspends exceeded that, STS refused the exchanges with 429 quota_exceeded, and the suspends failed.

The per-actor token cache does not help with this. Each actor's first transfer is a miss, whatever the cache size.

A small test program reproduced the refusal on a two-node cluster: STS refused calls after about 6,000 in a minute and accepted them again as the minute rolled over.

Proposal

Mint the downscoped tokens locally, with client-side Credential Access Boundaries:

1. Exchange each source token at STS once, for an intermediary token and a session key.
2. For each actor, encrypt its Credential Access Boundary with the session key and append it to the intermediary token. No STS call.

STS then sees one call per source token refresh, about every 20 minutes, instead of one per actor. The rules each token carries do not change, and GCS enforces them the same way.

Google documents client-side token exchange and implements it in its Java library. The Go libraries do not support it yet. Google's STS team said the format is stable for implementations outside its libraries, as long as they follow the same logic. It needs tink-go, since STS sends the session key as a Tink keyset.

A Go prototype passed against real GCS: each locally minted token could write and delete under its own actor's prefix and was refused on another actor's.