You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Spark] Let VortexOutputWriter store footer metadata #11136
VortexWriter.Builder can store user metadata segments in the file footer, but VortexOutputWriter doesn't expose this. A caller that writes Vortex files from Spark outside the commit protocol, and needs its own footer keys, has to maintain a separate writer that duplicates VortexOutputWriter's row-to-Arrow conversion and batching.
Proposed solution
Add a VortexOutputWriter constructor that takes Map<String, byte[]> metadata and passes it to VortexWriter.Builder.metadata.
The existing constructor delegates with no metadata, so Spark's write path doesn't change.
Values are read back verbatim by NativeFiles.readMetadata.
Additional context
We write event files directly from mapPartitions, one file per partition directory. Each file records the layout version, schema version and bucket count it was written with, so a reader can refuse files it doesn't understand. Today that needs a custom writer whose only difference is the footer metadata.
Activity
feat(spark): let VortexOutputWriter store footer metadata (#11137)