Repository navigation
Support extended partition cols for listing table. #18482
Description
Is your feature request related to a problem or challenge?
When scanning partitioned files, there are scenarios where runtime-generated values (not persisted in the files) need to be attached to each RecordBatch.
Currently, when a partition contains multiple files, loadNextBatch has no context about which file it is returning rows from.
This makes it impossible to append per-file runtime data to the resulting RecordBatch.
We’d like a way to extend the file schema and stream with additional columns—similar to how table_partition_cols are added from directory structure.
Example
Partition directory: /data1/
Files:
/data1/file1
/data1/file2
/data1/file3
File schema: { row_id: Int32, b: Int32 }
Runtime metadata:
file1 -> cumulative_total_rows = 5
file2 -> cumulative_total_rows = 7
file3 -> cumulative_total_rows = 17
Derived schema:
{ row_id: Int32, b: Int32, cumulative_total_rows }
Example expression:
row_id + cumulative_total_rows
Describe the solution you'd like
Extend the ListingTable and ListingOptions to support user-provided extended columns (extended_cols), which are appended to each file’s stream and schema—analogous to table_partition_cols.
- Add extended_cols to ListingOptions, defined as:
extended_cols: HashMap<String, HashMap<String, ScalarValue>>
where:
- outer key = column name
- inner key = file name
- value = runtime constant for that file
- These values should be made available in the scan output (similar to partition columns), allowing expressions to reference them.
Describe alternatives you've considered
- Expose ObjectMeta to PhysicalExprAdapter, allowing it to append file metadata (e.g., file name) to the stream.
- Then a MemTable with file_name → extended_col mappings could be joined to enrich data.
Any alternative mechanism that makes per-file runtime context accessible during scan would work.
@alamb @timsaucer Any thoughts?
Activity
alamb commentedon Nov 5, 2025
Thanks for filing this ticket @animodak7
There are a few "vritual" columns I have heard people ask about before:
Here are some related discussions:
I think the filename is (or could be) made available today
Row numbers need support in the arrow parquet reader (though I am talking with @vustef about this shorly)
alamb commentedon Nov 5, 2025
Here is another related PR in iceberg-rust
_filecolumn iceberg-rust#1824feniljain commentedon Nov 26, 2025
Hey @alamb 👋🏻
Sorry, if its an obvious question, but I dont understand where exactly did you mean to expose the above? Do you mean on stream level or somewhere else?
alamb commentedon Nov 26, 2025
I mean that the ListingTable already has the information on what file is being scanned, so with just code in DataFusion we could add support for scanning filenames
Things like row numbers and row group numbers need prquet reader support
However, thanks to @vustef once we upgraade to arrow 57.1.0 we'll have access to the row numbers from the parquet reader
alamb commentedon Feb 3, 2026
I filed an epic to track related items