You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Note that in Spark, the array_agg is an alias of collect_list, link here.
Also note that, Datafusion also support array_agg, however, there seems to a difference in behaviour and syntax with Spark.
For eg. Datafusion support ORDER BY within array_agg, link, and can provide deterministic ordering. Spark on the other hand, doesn't support ORDER BY within array_agg and does not ensure deterministic ordering. Spark doc explicitly mentions this for all 2 functions - The function is non-deterministic because the order of collected results depends on the order of the rows which may be non-deterministic after a shuffle., link.
changed the title [-][datafusion-spark]: Implement collect_list/array_agg and collect_set[/-][+][datafusion-spark]: Implement collect_list/array_agg[/+]on Oct 5, 2025
Activity
[-][datafusion-spark]: Implement collect_list/array_agg and collect_set[/-][+][datafusion-spark]: Implement collect_list/array_agg[/+]alamb commentedon Oct 14, 2025
Thanks! I added this to the epic:
datafusion-sparkSpark Compatible Functions #15914cht42 commentedon Jan 8, 2026
take
feat(spark): Implement collect_list/collect_set aggregate functions (#…
feat(spark): Implement collect_list/collect_set aggregate functions (a…