Skip to content

[datafusion-spark]: Implement collect_list/array_agg #17923

Description

@rishvin

Is your feature request related to a problem or challenge?

Implement collect_list, array_agg equivalent of spark.

Note that in Spark, the array_agg is an alias of collect_list, link here.

Also note that, Datafusion also support array_agg, however, there seems to a difference in behaviour and syntax with Spark.

For eg. Datafusion support ORDER BY within array_agg, link, and can provide deterministic ordering. Spark on the other hand, doesn't support ORDER BY within array_agg and does not ensure deterministic ordering. Spark doc explicitly mentions this for all 2 functions - The function is non-deterministic because the order of collected results depends on the order of the rows which may be non-deterministic after a shuffle., link.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Activity

  1. added theissue type on Oct 5, 2025
  2. changed the title [-][datafusion-spark]: Implement collect_list/array_agg and collect_set[/-] [+][datafusion-spark]: Implement collect_list/array_agg[/+] on Oct 5, 2025
  3. alamb commented on Oct 14, 2025

    @alamb
    Contributor
  4. cht42 commented on Jan 8, 2026

    @cht42
    Contributor

    take

  5. added a commit that references this issue on Jan 10, 2026
    821d410
  6. added a commit that references this issue on Mar 24, 2026
    f3b06fe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions