You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Spark] Expose a write summary from VortexOutputWriter #11125
VortexOutputWriter.close() finishes the file but throws away the VortexWriteSummary that VortexWriter.finish() returns. To learn anything about the file it just wrote, a caller has to open it again.
Proposed solution
After a successful close(), expose a summary of the write:
file size and row count, from VortexWriter.finish();
number of batches;
peak Arrow bytes of one batch;
peak bufferedBytes() reported by the native writer.
Additional context
Callers can collect file stats from tasks (sizes, rows per file) to check layout and skew, without listing and reopening files.
Activity