Repository navigation
Implement faster single batch encoding/decoding for use in shuffle #1189
Description
What is the problem the feature request solves?
Because Spark shuffle is block-based rather than streaming, it is necessary to serialize single batches along with schema information. We currently use Arrow IPC to do this, but this is not efficient. A crude prototype in PR TBD shows that we can get much better performance with a custom implementation of single batch serde.
Describe the potential solution
No response
Additional context
No response
Activity