Repository navigation
[Bug] COS STS default policy uses an invalid wildcard owner UID #3983
Description
Search before asking
I searched the existing issues and found nothing similar.
Fluss version
main (development)
Please describe the bug
When Fluss obtains Tencent Cloud COS federation credentials without an explicitly configured fs.cosn.userinfo.security_token_policy, COSSecurityTokenProvider builds the default resource as:
qcs::cos:<region>:uid/*:<bucket>/<prefix>*
Tencent Cloud COS does not accept uid/* as the bucket owner segment for object access. The owner must be the bucket APPID, which is the numeric suffix of a standard COS bucket name. The generated temporary credentials are therefore accepted by STS but COS requests fail with HTTP 403.
Expected resource format:
qcs::cos:<region>:uid/<appid>:<bucket>/<prefix>*
This can be reproduced with a standard bucket such as example-1250000000: construct the provider without a custom policy, obtain the federation token, and access the configured remote-data prefix. STS returns credentials, but COS denies the operation because the resource owner is invalid.
No credentials, internal endpoints, or account identifiers are included in this report.
Solution
Parse the numeric APPID suffix from the COS bucket name and use it in the default bucket-scoped policy. If the bucket name does not end in -<APPID>, fail with an actionable message instructing the user to configure a custom policy rather than silently generating an unusable one. Add unit tests for both paths.
Are you willing to submit a PR?
Yes, I will submit a PR with the fix and focused unit tests.
Activity
damokelis commentedon Aug 13, 2026
During live Fluss-to-Paimon 2.0 validation, I found a second independent 403 cause in the same COS STS path:
hadoop-cos 3.3.5drops the session token by convertingCOSSessionCredentialstoBasicCOSCredentials. Hadoop fixed this in HADOOP-19648.PR #3984 now also upgrades
hadoop-costo 3.5.0 and adds aCosNFileSystem.initialize()regression test that checks the credentials received by the COS SDK through Hadoop's retry proxy.A shaded build was deployed to a test cluster. It downloaded Fluss KV snapshots from COS without 403, completed 14 consecutive Flink checkpoints, and committed multiple Paimon 2.0 snapshots to COS over a 15-minute observation window.
damokelis commentedon Aug 13, 2026
Follow-up from the implementation and live validation found a second independent cause of COS STS 403 responses:
hadoop-cos 3.3.5convertsCOSSessionCredentialstoBasicCOSCredentials, dropping the session token before COS requests. HADOOP-19648 fixes this inhadoop-cos 3.5.0.PR #3984 now covers both causes:
hadoop-costo 3.5.0 and preserve temporary session credentialsCosNFileSystemThe updated change was validated end to end with a fresh 31-table MySQL full snapshot plus continuous CDC into Fluss, followed by Paimon 2.0 lake tiering to COS. The fresh database produced non-empty
snapshot-1andLATESTobjects, including verified snapshot metadata with 1,463 and 5,637,613 records. No real HTTP 403/Forbidden/AccessDenied response occurred after deploying the fix.