You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Queries on Iceberg metadata tables (.partitions, .snapshots, .all_manifests, .entries, .files, .refs) fall back to Spark. Table-maintenance jobs and partition-listing queries read these, and the operators above the scan stay in Spark.
Describe the potential solution
Read the common metadata tables natively, starting with .partitions, .snapshots and .all_manifests. Check how much iceberg-rust's metadata table support already covers.
Additional context
The native Iceberg scan already supports the _file, _pos, _spec_id and _partition metadata columns. This issue is about the metadata tables, which CometScanRule.isIcebergMetadataTable sends to Spark. #2033 made these queries fall back cleanly. #6719 covers data tables whose names end in a metadata-table name.
Activity