Repository navigation
GroupedHashAggregateStream::emit throws ArrowError: column types must match schema types #24069
Description
Describe the bug
While testing DataFusion Comet, we found that DataFusion throws an exception:
ArrowError: column types must match schema types,
expected List(Struct("colA": Boolean, ...))
but found List(Struct("colA": non-null Boolean, ...)) at column index 1and the corresponding call stack is GroupedHashAggregateStream::emit() ← spill() ← try_update_memory_reservation()
Also raised the issue in Comet: apache/datafusion-comet#5239
To Reproduce
No response
Expected behavior
No response
Additional context
No response
Activity
alamb commentedon Aug 8, 2026
This looks like a regression to me so adding to the release
55.0.0(Jul / Aug 2026) #22393FYI @2010YOUY01 perhaps this is related to the aggregate refactor
[-]`GroupedHashAggregateStream::emit` throws ArrowError: column types must match schema types[/-][+]Fix `GroupedHashAggregateStream::emit` throws ArrowError: column types must match schema types[/+][-]Fix `GroupedHashAggregateStream::emit` throws ArrowError: column types must match schema types[/-][+]`GroupedHashAggregateStream::emit` throws ArrowError: column types must match schema types[/+]Add end-to-end reproducer for aggregating batches stricter than the d…
alamb commentedon Aug 11, 2026
Update it is not clear to me that this is a regression or not. I am working on a reproducer to be sure
collect_set(struct)aggregate crashes on spill emit datafusion-comet#5239 (comment)Add end-to-end reproducer for aggregating batches stricter than the d…
alamb commentedon Aug 11, 2026
I made a reproducer here:
And it fails on both 54 and main (55) so I have determined it is not a regression
fix: adapt input batches with stricter nested nullability to planned …
[branch-55] fix: adapt input batches with stricter nested nullability…