Skip to content

[Spark] Let VortexOutputWriter store footer metadata #11136

Open

Description

@ChristopherWirt

Problem or motivation

VortexWriter.Builder can store user metadata segments in the file footer, but VortexOutputWriter doesn't expose this. A caller that writes Vortex files from Spark outside the commit protocol, and needs its own footer keys, has to maintain a separate writer that duplicates VortexOutputWriter's row-to-Arrow conversion and batching.

Proposed solution

Add a VortexOutputWriter constructor that takes Map<String, byte[]> metadata and passes it to VortexWriter.Builder.metadata.

  • The existing constructor delegates with no metadata, so Spark's write path doesn't change.
  • Values are read back verbatim by NativeFiles.readMetadata.

Additional context

We write event files directly from mapPartitions, one file per partition directory. Each file records the layout version, schema version and bucket count it was written with, so a reader can refuse files it doesn't understand. Today that needs a custom writer whose only difference is the footer metadata.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions