Skip to content

[Epic]: Google Summer of Code 2025 Correlated Subquery Support #16059

Closed
Closed
[Epic]: Google Summer of Code 2025 Correlated Subquery Support#16059
Labels
EPICA larger project, actively underway, with sub tasksenhancementNew feature or request

Description

@irenjj
irenjj
opened on May 15, 2025

This ticket tracks the progress for a 2025 Google Summer of Code (GSOC) sponsored project on Correlated Subquery Support

Project Documentation

Is your feature request related to a problem or challenge?

DataFusion currently has limited support for correlated subqueries. This project aims to implement comprehensive support for correlated subqueries in Apache DataFusion by applying Hyper's 'Unnesting Arbitrary
Queries' framework.

Timeline:

Except from the Official GSOC Timeline:

  • May 8 - June 1: Community Bonding Period | GSoC contributors get to know mentors, read documentation, get up to speed to begin working on their projects
  • June 2: Coding officially begins!
  • July 18: Mid term evaluation
  • August 25: Final week
  • Sep 8: Final evaluations due / wrap up

Work

Epics tracking technical work:

Other potential future work

Related work:

Related documentation

Newer research that might be interesting

Activity

  1. irenjj
    added
    enhancementNew feature or request
    on May 15, 2025
  2. alamb
    changed the title [-][Epic]: Enhance Correlated Subquery Support[/-] [+][Epic]: Google Summer of Code 2025 Correlated Subquery Support[/+] on May 15, 2025

  3. alamb commented on May 15, 2025

    @alamb
    alamb
    on May 15, 2025
    Contributor

    @irenjj could you attach your GSOC proposal to this issue so it is public? I think you did quite a good job on that and it would be great to make it more public

  4. alamb commented on May 18, 2025

    @alamb
    alamb
    on May 18, 2025
    Contributor

    Let's plan to use this ticket for high level project planning / coordination, and will use other tickets for specific technical work

    I update the description on this ticket with some more detail and the GOSC timeline, along with some other related work I know about. If anyone else has suggested readings we can add to the list that would be great

    @irenjj and I had a discussion today and covered the following items:

    1. As described above, @irenjj plans to first focus on support for multi-level subqueries [dbolympics] Support Nested correlated subquery error with a depth exceeding 1 #15558 (hopefully we'll be able to collaborate with @suibianwanwank and @duongcongtoai for this)
    2. It is likely that supporting multiple subquery decorrelation (e.g. arbitrary levels of correlation) will require a more general subquery decorrelation framework (which we will discuss in more detail on related tickets)
    3. One of the major lessons for working with DataFusion and open source is figuring out how to work with the broader community (among other things, you never know who is reading an issue or who might be interested in an update)
    4. We also hope to write a blog post about correlated subquery support, as the final report: Blog about DataFusion correlated subquery support #16084

  5. alamb commented on Jul 25, 2025

    @alamb
    alamb
    on Jul 25, 2025
    Contributor

    👋 @irenjj and @duongcongtoai , I hope things are well with you

    It seems as though you have been working on @duongcongtoai 's fork which is a great idea: https://github.com/duongcongtoai/arrow-datafusion/pulls?q=is%3Apr

    Would it be possible to summarize how things are going and what you are planning to do in the near future?

  6. irenjj commented on Jul 25, 2025

    @irenjj
    irenjj
    on Jul 25, 2025
    ContributorAuthor

    👋 @irenjj and @duongcongtoai , I hope things are well with you

    It seems as though you have been working on @duongcongtoai 's fork which is a great idea: https://github.com/duongcongtoai/arrow-datafusion/pulls?q=is%3Apr

    Would it be possible to summarize how things are going and what you are planning to do in the near future?

    It's runnable and have passed half of tests in subquery.slt in my local branch, but haven't submitted the PR yet. and most operators are supported now(except union and cte), currently we are focusing on metadata layer issues.

  7. alamb commented on Jul 25, 2025

    @alamb
    alamb
    on Jul 25, 2025
    Contributor

    Awesome -- thank you @irenjj

    Maybe once you have a PR, you can ping me and I can give it a high level look and hear what you plan next.

  8. Standing-Man commented on Jul 29, 2025

    @Standing-Man
    Standing-Man
    on Jul 29, 2025
    Contributor

    Hi @alamb and @irenjj, I would like to understand how to implement the Correlated Subquery feature in DataFusion. Is there a timeline organized by feature milestones and corresponding Pull Requests that I can refer to in order to study the relevant code?

  9. alamb commented on Jul 29, 2025

    @alamb
    alamb
    on Jul 29, 2025
    Contributor

    Hi @Standing-Man -- that is a great question. @irenjj perhaps this would be a good exercise for the next weekly update -- make an "Epic" style description of the steps needed and how many you have completed / are working on

  10. irenjj commented on Aug 10, 2025

    @irenjj
    irenjj
    on Aug 10, 2025
    ContributorAuthor

    Hi @Standing-Man here is a runnable version #17110 (but still has some issues), and i wrote a simple doc to explain how it works.
    cc @alamb @duongcongtoai

  11. alamb
    added
    EPICA larger project, actively underway, with sub tasks
    on Aug 12, 2025

  12. alamb commented on Sep 10, 2025

    @alamb
    alamb
    on Sep 10, 2025
    Contributor

    @irenjj now that GSOC 2025 is coming to a close, perhaps you can briefly summarize the state of the project and we can close down this ticket?

  13. irenjj commented on Sep 11, 2025

    @irenjj
    irenjj
    on Sep 11, 2025
    ContributorAuthor

    Here is a brief summarize of the project:

    COMPLETED WORK
    Simple implementation of decorrelation framework - algorithm implementation for converting correlated subqueries with three main components:

    • DependentJoinRewriter - Converts subqueries into dependent joins
    • DependentJoinDecorrelator - Removes correlations using advanced unnesting
    • Duplicate elimination optimizer(WIP) - Removes redundant operations

    REMAINING WORK

    • Fix corner cases in existing tests - Ensure all previously passing tests in subquery.slt work with the new decorrelation framework, including edge cases that currently fail
    • Add configuration variable - Implement a config variable to manually enable/disable the decorrelate_general functionality, allowing users to toggle between old and new subquery processing
    • Add complex test cases - Implement support for more sophisticated scenarios that the original system couldn't handle (e.g., TPC-DS benchmark queries and other complex correlated subquery patterns)

  14. alamb commented on Sep 12, 2025

    @alamb
    alamb
    on Sep 12, 2025
    Contributor

    the PR to add correlated subquery support is here:

    I think with that let's claim this project is done.

    Thank you for all your effort @irenjj

  15. alamb
    closed this as completedon Sep 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    EPICA larger project, actively underway, with sub tasksenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions