You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Iceberg scan fails with ExpiredToken once REST-vended credentials expire #7387
With an Iceberg REST catalog that vends short-lived S3 credentials, the native Iceberg scan (spark.comet.scan.icebergNative.enabled=true) signs every S3 request with the s3.* credentials vended when the driver loaded the table. Comet never refreshes them. A scan that is still reading after they expire fails with ExpiredToken.
Spark's own read of the same table keeps working, because Iceberg's S3FileIO refreshes the credentials through the client.refresh-credentials-endpoint the catalog returns.
Nothing in the error points at spark.sql.catalog.<catalog>.s3.comet.credential.provider.class, which is the only way to avoid this today.
Steps to reproduce
Configure an Iceberg REST catalog that vends S3 credentials with a short lifetime and returns client.refresh-credentials-endpoint (for example Unity Catalog, about one hour).
Enable Comet with the native Iceberg scan.
Run a query whose scan reads for longer than the credential lifetime.
Expected behavior
The native scan refreshes the vended credentials the way S3FileIO does, with no extra configuration.
Additional context
Spark 4.0.1, Comet 1.1.0. #7363 adds a provider that refreshes and selects it automatically when the catalog returns a refresh endpoint.
Activity