Discussion: DataFusion Improvement Proposal (DIPs) Process?

@viirya

Is your feature request related to a problem or challenge?

Sometimes, I feel that some important proposals in DataFusion lack sufficient context, or that the relevant context is scattered across various issues and PR comments. This makes it difficult to fully understand the proposals or to trace their motivations and evaluate their soundness. As a result, we sometimes see large PRs — hundreds or even thousands of lines — that are based on these proposals, making the review process even more challenging. Only the author or those who were involved in the initial discussions seem to be in a position to effectively review them.

For example, Spark has the SPIP (Spark Project Improvement Proposal) mechanism, where contributors submit formal documents for review when proposing significant changes. These documents typically consolidate the technical details, motivation, and background of the proposal into a single place. This approach helps the community better understand and participate in discussions around major changes.

I wonder if it would be beneficial for DataFusion to adopt a similar lightweight proposal process for major design changes — something that allows ideas and context to be collected and reviewed before implementation begins. It could help improve transparency, facilitate broader community involvement, and make the review process more accessible.

If the full SPIP process — including voting and formal approval — feels too heavy or unnecessary for our context, perhaps we could at least establish a lightweight template for major change proposals. This template could include sections for motivation, background, technical details, and other relevant context. Having a consistent format would make it easier for the community to follow and engage with significant design discussions.

My opinions:

Finding the outstanding proposals and discussions is difficult. They are all public but there is lots of them going on
The context for proposals is often scattered across issues and PRs
It is hard to know when "enough" communication has been done for a proposal to move forward and when it needs more work
Improving the communication around major changes is becoming more important as the project grows and we have more users and contributors

For example, there are several recent discussions that could benefit from this more formal proposal process, including but not limited to the discussion itself above

Plan to replace SchemaAdapter with PhysicalExprAdapter #16800 itself from @adriangb
Support multiple ordered array_agg aggregations #16625 from @findepi
Document the SQL dialect DataFusion attempts to follow #13704 in general , and Document the SQL dialect DataFusion attempts to follow #13704 (comment) recently with @berkasynnada
[DISCUSSION] Memory accounting model discussion #16841 from @gabotechs,
[EPIC] ScalarUDFImpl::equals default implementation is error-prone #16677 with @findepi

Describe the solution you'd like

Some sort of "process" that

Makes it easy to find outstanding community improvement proposals
Makes it easy to know the steps to create a new improvement proposal
Is documented

Describe alternatives you've considered

Here is a strawman (for discussion) proposal:

Add a new tag in the DataFusion repo ("DIP - DataFusion Improvement Proposal")
Add a new ISSUE_TEMPLATE for proposals issues based on the SPIP one and current DataFusion issue template
Add a section to the site documentation describing the process

I personally worry that DataFusion is not at a point I where formal voting / formal approval would add a lot of value, but I do think formalizing the proposal format and making them easier to find would be beneficial.

I propose starting with more formalization around the communication of proposals and we can add more explicitly approval / consensus standards if and when they become necessary.

Additional context

Here is the documentation for the spark process: The https://spark.apache.org/improvement-proposals.html

I looked through the list of SPIPs in Spark and the few I looked at didn't have huge amounts of discussion. They often linked to a google doc with more details.

@alamb

Is your feature request related to a problem or challenge?

@viirya says in #16800 (comment):

Sometimes, I feel that some important proposals in DataFusion lack sufficient context, or that the relevant context is scattered across various issues and PR comments. This makes it difficult to fully understand the proposals or to trace their motivations and evaluate their soundness. As a result, we sometimes see large PRs — hundreds or even thousands of lines — that are based on these proposals, making the review process even more challenging. Only the author or those who were involved in the initial discussions seem to be in a position to effectively review them.
For example, Spark has the SPIP (Spark Project Improvement Proposal) mechanism, where contributors submit formal documents for review when proposing significant changes. These documents typically consolidate the technical details, motivation, and background of the proposal into a single place. This approach helps the community better understand and participate in discussions around major changes.
I wonder if it would be beneficial for DataFusion to adopt a similar lightweight proposal process for major design changes — something that allows ideas and context to be collected and reviewed before implementation begins. It could help improve transparency, facilitate broader community involvement, and make the review process more accessible.
If the full SPIP process — including voting and formal approval — feels too heavy or unnecessary for our context, perhaps we could at least establish a lightweight template for major change proposals. This template could include sections for motivation, background, technical details, and other relevant context. Having a consistent format would make it easier for the community to follow and engage with significant design discussions.

My opinions:

Finding the outstanding proposals and discussions is difficult. They are all public but there is lots of them going on

The context for proposals is often scattered across issues and PRs

It is hard to know when "enough" communication has been done for a proposal to move forward and when it needs more work

Improving the communication around major changes is becoming more important as the project grows and we have more users and contributors

For example, there are several recent discussions that could benefit from this more formal proposal process, including but not limited to the discussion itself above

Plan to replace SchemaAdapter with PhysicalExprAdapter #16800 itself from @adriangb

Support multiple ordered array_agg aggregations #16625 from @findepi

Document the SQL dialect DataFusion attempts to follow #13704 in general , and Document the SQL dialect DataFusion attempts to follow #13704 (comment) recently with @berkasynnada

[DISCUSSION] Memory accounting model discussion #16841 from @gabotechs,

ScalarUDFImpl::equals default implementation is error-prone #16677 with @findepi

Describe the solution you'd like

Some sort of "process" that

Makes it easy to find outstanding community improvement proposals

Makes it easy to know the steps to create a new improvement proposal

Is documented

Describe alternatives you've considered

Here is a strawman (for discussion) proposal:

Add a new tag in the DataFusion repo ("DIP - DataFusion Improvement Proposal")

Add a new ISSUE_TEMPLATE for proposals issues based on the SPIP one and current DataFusion issue template

Add a section to the site documentation describing the process

I personally worry that DataFusion is not at a point I where formal voting / formal approval would add a lot of value, but I do think formalizing the proposal format and making them easier to find would be beneficial.

I propose starting with more formalization around the communication of proposals and we can add more explicitly approval / consensus standards if and when they become necessary.

Additional context

Here is the documentation for the spark process: The https://spark.apache.org/improvement-proposals.html

I looked through the list of SPIPs in Spark and the few I looked at didn't have huge amounts of discussion. They often linked to a google doc with more details.

Great ! @alamb i think surely it would be very helpful for the future context and to keep the proposals on point

I looked through the list of SPIPs in Spark and the few I looked at didn't have huge amounts of discussion. They often linked to a google doc with more details.

Most discussions happened in the google docs, I think. Especially such docs allow to comment/discuss around specified sentences, API proposal, etc.

I agree that formal voting/approval doesn't make sense yet.

Also having a structured way to propose "larger" changes that incorporates all relevant context for reviewers is definitely a good idea. I think the EPICs that you had are basically this - a focusing a point for a large change that has relevant context + links to the other sub-issues/PRs. (example: #9637)

Also having a structured way to propose "larger" changes that incorporates all relevant context for reviewers is definitely a good idea

+1, this is a good idea. We all prefer concentrated information rather than searching everywhere for things to help us understand a huge PR.

I will try and summarize the contents of this ticket into a PR to the documentation site later this week and will post an update here

Describe the solution you'd like

Some sort of "process" that

Makes it easy to find outstanding community improvement proposals

Makes it easy to know the steps to create a new improvement proposal

Is documented

Those are noble goals, but i would also add the following

it's need to be "executable" (each step of the process needs to be feasible)
it's need to be decisive (reaching a decision)
it's need to be finite (running to an end in bounded time)

last but not least

it needs to involve relevant stakeholders (subject matter experts and heavy users of given functional area)
- of course everyone on the Internet should be able to provide input, but some people are simply more likely to be interested in a topic

Finiteness is expectedly hard to achieve in a distributed community where everyone follows their own timelines, but this should not come at the cost of missing involving relevant people, i.e. people which input can be reasonably expected to be useful.

@nuno-faria

Those are noble goals, but i would also add the following

it's need to be "executable" (each step of the process needs to be feasible)

it's need to be decisive (reaching a decision)

it's need to be finite (running to an end in bounded time)

When I ran an engineering team at a company, I would 100% agree with all these items and insisted on them myself.

However, after several years in an open source project like Arrow and DataFusion I am not sure they are a great match for this community.

I have seen several projects that start as vague statements of goal and then various people provide parts of the implementation / writeups / tickets, etc and it ends up being completed (utf8view being one, porting all functions to be UDFs is another, the most recent Parquet metadata cache from @nuno-faria is shaping up to be similar as well)

The other question is who decides if a particular proposal meets those criteria 🤔 Maybe if we say they are best practice that would be enough

it needs to involve relevant stakeholders (subject matter experts and heavy users of given functional area)

of course everyone on the Internet should be able to provide input, but some people are simply more likely to be interested in a topic

Finiteness is expectedly hard to achieve in a distributed community where everyone follows their own timelines, but this should not come at the cost of missing involving relevant people, i.e. people which input can be reasonably expected to be useful.

In general I agree -- I think the key challenges are:

Identifying who the key people are (some of the core contributors probably know, but figuring this out is a high barrier)
Ensuring stakeholders have enough time to provide input if they want, but do not hold up the process indefinitely due to lack of bandwidth

Since all communications are public, I think any stakeholder who is interested in a feature already has the opportunity to discover and participate. However, I think we could do a better job making it easier for relevant stakeholders to find proposals

I have seen several projects that start as vague statements of goal and then various people provide parts of the implementation / writeups / tickets, etc and it ends up being completed (utf8view being one, porting all functions to be UDFs is another, the most recent Parquet metadata cache from @nuno-faria is shaping up to be similar as well)

Those are examples of implementation projects. These can span unforeseeable amount of time, depending on interested of contributors.

I think DIP is a different category. DIP process should run to an end in bounded time, even if this means "timeout => rejection".
There is no point in starting a DIP in 2019 and all of the sudden deciding on it in 2026.

Since all communications are public, I think any stakeholder who is interested in a feature already has the opportunity to discover and participate

That works provided that all potential stakeholder are listening on a communication channel where DIPs are announced.
What would be such a channel?

Copying from ASF DF Slack

Also pretty nice thing is how to define proposals in clean and understandable way which described here https://github.com/kubeflow/community/tree/master/proposals#kep-format-to-propose-and-document-enhancements-to-kubeflow
The example of recent change is https://docs.google.com/document/d/1xj3K6bOT4f0EPiC4zwr2OsbRkROjIA8u_TBRZRVxrHI/edit?tab=t.0#heading=h.bco7r7uaguxy

This approach definitely takes more time for writing up the change, however it would easier for community to digest the change and make fundamental review on design level rather than going through hundreds/thousands lines of code.

Another nice benefit this design would serve as a solid draft for public article

That works provided that all potential stakeholder are listening on a communication channel where DIPs are announced.
What would be such a channel?

I do not know of anything yet except for github alerts, which I agree are hard to navigate in. In my mind, clearly defining the channel where any potential stakeholder should look would be one of the key improvements of a more formal DIP process

I made a small PR to consolidate the documentation we currently have about proposing new features:

Docs: Consolidate feature proposal content into roadmap #17156

I hope that if that is merged we will then at least have a place to put any new process / writeups

ALso, for my own sanity I tried out adding some new labels to help find proposals

Specifically, I added two labels EPIC and PROPOSED EPIC

So the idea would be:

If you are looking for "in progress" work, look for an EPIC: https://github.com/apache/datafusion/issues?q=is%3Aissue%20state%3Aopen%20label%3AEPIC

If you are looking for active proposals (that are not yet underway / looking for proposals) look for the PROPOSAL EPIC: https://github.com/apache/datafusion/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22PROPOSAL%20EPIC%22

If people like this basic idea, I can write it up more formally. I don't think it replaces the need for a DIP process, but maybe it could help people find the proposals more easily

ALso, for my own sanity I tried out adding some new labels to help find proposals

Specifically, I added two labels EPIC and PROPOSED EPIC

So the idea would be:

If you are looking for "in progress" work, look for an EPIC: https://github.com/apache/datafusion/issues?q=is%3Aissue%20state%3Aopen%20label%3AEPIC

If you are looking for active proposals (that are not yet underway / looking for proposals) look for the PROPOSAL EPIC: https://github.com/apache/datafusion/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22PROPOSAL%20EPIC%22

If people like this basic idea, I can write it up more formally. I don't think it replaces the need for a DIP process, but maybe it could help people find the proposals more easily

I have an idea to include in the process for moving PROPOSED EPIC → EPIC: requiring a sponsoring maintainer—someone who can merge PRs (committer/PMC) and is willing to follow and review major PRs for that EPIC.

In a discussion thread, it can be hard to tell whether someone is simply participating and giving general feedback, or if they are willing to devote significant time to getting the feature completed. Making this a formal step in the process would make it clearer. Without a dedicated sponsoring maintainer, some good PRs can remain unreviewed for a very long time.

In a discussion thread, it can be hard to tell whether someone is simply participating and giving general feedback, or if they are willing to devote significant time to getting the feature completed. Making this a formal step in the process would make it clearer. Without a dedicated sponsoring maintainer, some good PRs can remain unreviewed for a very long time.

this is a really good point.

I took a shot at updating the docs with epics as well as the idea of Sponsoring Maintainer -- i would love any feedback:

docs: Update documentation on Epics and Supervising Maintainers #17505

BTW I am writing up my own thoughts on how large projects get done in DataFusion and how to we can increase that capacity.

@nellshamrell

BTW I think the Rust community in general did a great job with this type of communication via "This week in Rust" https://this-week-in-rust.org/

For example: https://this-week-in-rust.org/blog/2026/02/18/this-week-in-rust-639/

Looking at the commits, https://github.com/rust-lang/this-week-in-rust/commits/main/, this newsletter currently seems to be largely due to the heroic efforts of @nellshamrell

@alamb

@alamb you are very kind (and this made my day) - though it's not a solo effort. My fellow editors listed in our README also commit heroics to get the newsletter out every week <3

Navigation Menu

Repository navigation

Discussion: DataFusion Improvement Proposal (DIPs) Process? #16886

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Describe the solution you'd like

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Discussion: DataFusion Improvement Proposal (DIPs) Process? #16886

Description

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

Adez017 commented on Jul 24, 2025

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

viirya commented on Jul 24, 2025

phillipleblanc commented on Jul 25, 2025

xudong963 commented on Jul 26, 2025

alamb commented on Jul 28, 2025

findepi commented on Aug 4, 2025

Describe the solution you'd like

alamb commented on Aug 4, 2025

findepi commented on Aug 6, 2025

comphead commented on Aug 6, 2025

alamb commented on Aug 6, 2025

alamb commented on Aug 12, 2025

alamb commented on Aug 12, 2025

2010YOUY01 commented on Aug 13, 2025

alamb commented on Aug 13, 2025

alamb commented on Sep 10, 2025

alamb commented on Feb 19, 2026

nellshamrell commented on Feb 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions