You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Iceberg writes for CTAS / RTAS on Spark 3.4 #7428
On Spark 3.4, CREATE TABLE AS SELECT and REPLACE TABLE AS SELECT into Iceberg plan AtomicCreateTableAsSelectExec / AtomicReplaceTableAsSelectExec, and the staged table writes inline. These writes are not eligible for native Iceberg writes, so the write and the operators above the last shuffle stay in Spark.
Describe the potential solution
Plan the staged CTAS/RTAS write on Spark 3.4 through the same native Iceberg write path used for AppendData, and commit through the staged table.
Additional context
The Iceberg guide notes that CTAS/RTAS on Spark 3.4 writes inline today.
Activity