Repository navigation
metadata column support #13975
Description
Is your feature request related to a problem or challenge?
spark support metadata column
then we can treat rowid as metadata column.
Describe the solution you'd like
add metadata column in TableProvider trait
Describe alternatives you've considered
No response
Additional context
No response
Reacted by Curtis Nishikino and Alex Hall
Activity
nishikinocurtis commentedon Jan 6, 2025
+1, also encountering cases where parsing and executing queries referring to rowid is required.
Both logical planning and physical planning (related to
scan()s) would also need enhancement to 1. parse queries refer to metadata columns and 2. if necessary produce corresponding execution plan to pull metadata out.I'd be interested in contributing this feature if more detailed discussion regarding these surrounding implementation is made.
alamb commentedon Jan 12, 2025
@nishikinocurtis -- perhaps you can look at the proposed PR / API here:
alamb commentedon May 1, 2025
I think this is duplicated by #15173 so let's continue the discussion there
location,size,last_modified) inListingTableProvider#15173niebayes commentedon Feb 3, 2026
I think the metadata column is one of the virtual columns and arrow-rs has supported reading virtual columns since v57
alamb commentedon Feb 3, 2026
Indeed -- I started organizing outstanding tickets / issues into an epic so we can more easily coordinate the work