Repository navigation
Add with_virtual_columns to ParquetSource for reading virtual columns #20132
Description
I have found that arrow-rs has supported reading virtual columns, including RowGroupIndex and RowNumber, since v57.0.0.
I propose to add a method with_virtual_columns to ParquetSource to let user decide which virtual columns should be read.
The virtual columns, like RowNumber, could be used to implement prewhere, late materialized topk, and secondary indexes and more fancy stuff.
virtual columns support in arrow-rs: apache/arrow-rs#8715
Reacted by Osman Telli
Activity
[-]Add with_virtual_columns to FileScanConfigBuilder for reading virtual columns[/-][+]Add with_virtual_columns to ParquetSource for reading virtual columns[/+]jkylling commentedon Feb 3, 2026
Very timely that I was already looking into this 😅 Have a look at #20133 I suspect we will need some additional changes in
arrow-rs.alamb commentedon Feb 3, 2026
I think it is time to make an epic to try and coordinate this work (adding metadata/virtual columns to the ListingTable/Parquet reader)
See
mbutrovich commentedon May 5, 2026
I'd like to open a PR that lands a subset of the work @jkylling prototyped in #20133, as physical-layer plumbing below this epic.
What I'd build. Extend
TableSchemawith avirtual_columns: Vec<FieldRef>list. Thread it intoParquetOpenerviaArrowReaderOptions::with_virtual_columns(...), handling the three arrow-rs edges (predicate-pushdown schema, projection mask, output stream schema) the same way #20133 did. No changes toListingTable, SQL planning,information_schema, or any_metadatasurface.Approach. The core logic would follow @jkylling's #20133 with credit in the commit message and PR description, narrower in scope and focused on what downstream physical-plan consumers need today.
Why this is distinct from #20071. The UDF-rewrite pattern in #20071 works for file-constant metadata (
input_file_name, modification time) because the opener substitutes a literal. It cannot express reader-generated virtual columns like row number or row group index, whose values change per row and are produced by the parquet reader itself viaArrowReaderOptions::with_virtual_columns. The two approaches are complementary, not overlapping.Why it does not preempt this epic. Any UX you settle on (relation-scoped hidden columns,
_metadatastruct, scalar UDFs) will need some way for the opener to produce these columns.TableSchema::with_virtual_columnsis that seam. It makes no commitment to visibility, collision, or naming policy.Who is ready to consume it today.
_tmp_metadata_row_indexis requested. Comet builds physical plans directly and does not need the SQL UX.#20071 landed the same shape of change (minimal opener-boundary plumbing, deferring UX to this epic) with approval from @alamb.
alamb commentedon May 5, 2026
This sounds like wishful thinking -- I didn't have a chance to move it over the line
alamb commentedon May 5, 2026
I personally suggest trying to break up the work into the smallest PRs possible to make it plausible to review
mbutrovich commentedon May 5, 2026
IMO #22026 is as small as it could possibly be as a first step, even adding an allow-list for only one of arrow-rs's supported virtual columns, expecting more could be in follow-up PRs with their own tests.