Skip to content

Saving the groups generated from groupby operation #5674

Closed as not planned
Closed as not planned

Description

@digital-idiot

Problem
Group by is an expensive operation. Therefore I want to store my dataset to disk in the form of groups from group by operation. My use case is concerned with the groups, for example I want to take advantage of the lazy loading and only want to load selected groups into memory and process them.

Preferred Solution

  • An additional parameter to pass the groups to cache when writing the dataset to disk
    Or
  • A separate function to write the dataset as a collection of groups to file.

Alternatives considered
Treating each group as separate dataset and writing each of them to separate file. This is not suitable if number of groups is large and each group is relatively very small.

Additional context
It would also be great if groupby operation is natively supported for multiple coordinates.

Activity

  1. max-sixty commented on Sep 18, 2024

    @max-sixty
    Collaborator

    Thanks for the suggestion. This didn't get traction and we're trying to keep issues < 1000, so I'll close, but feel free to suggest again, possibly with some motivating examples.

  2. dcherian commented on Sep 18, 2024

    @dcherian
    Contributor

    Funnily enough, the shuffle_by op here is one way to accomplish this: #9320 . Chunk boundaries will line up with group boundaries, so with the right data format this will work well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions