Repository navigation
Dataframe interchange protocol #25
Description
This issue supersedes #1 and #14. As agreed in 6 Aug call the first milestone in the definition of the dataframe API will be the part to interchange data. As an sample use case, Matplotlib being able to receive dataframes from different implementations (e.g. pandas, vaex, modin, cudf, etc.).
This work was originally discussed in OSSData, and an initial draft was later proposed here: wesm/dataframe-protocol#1.
The topics to discuss and decide on are next:
- Access dataframe properties/metadata:
- Get number of rows and columns Get number of rows and columns #20
- Get column names Get and set column names #21
- Get column data types
- Selecting a column and accessing the underlying data (I think it'll require a decision on Separate object for a dataframe colum? (is Series needed?) #6, whether we want a separate column/Series object)
- Data types (which are part of the standard, how they are represented, etc.) Data types to support #26
- How downstream libraries will access this API (implemented in the dataframe directly, or returned object via
__dataframe__) - Should row labels be part of the standard?
The procedure to include this part on the standard RFC will be as follows:
- Define the goals, requirements, target audience, scope and use cases, and include them in the RFC
- Discuss and build an standalone document specific to the data interchange based on the above topics
- Review internally, and post publicly for additional feedback
- Update the prototype with the agreed API
- Finalize and approve the API and the prototype, and add them to the RFC document
Reacted by Ralf Gommers and Hyukjin Kwon
Activity
jorisvandenbossche commentedon Aug 20, 2020
There is already quite some discussion about this at wesm/dataframe-protocol#1. Do we want to continue the discussion there, or rather here?
Is what is being discussed over there the basis of the proposal here, or are there clear aspects where we want to deviate?
datapythonista commentedon Aug 20, 2020
Since we're focussing in the interchange protocol, we're using the discussion in Wes' PR as a starting point. I've been using that PR to discuss things previously discussed, but I'm creating issues in this repo for things that weren't.
rgommers commentedon Aug 20, 2020
Probably not. And if there are cases, those should be discussed on
wesm/dataframe-protocol.I think the intent here is to go back and actually start by documenting what was missed in the whole Discourse discussion that started that prototype. Namely purpose & scope, goals/non-goals, etc. That would have made that discussion a whole lot more productive, and the decisions on the prototype easier and more consistent.
The prototype itself seems great, and Wes and a few others had goals very much in line with the broader goals for the dataframe effort within this Consortium. From my perspective that prototype stalled because of (a) muddled discussion and half the participants not understanding the goals, and (b) no funding.
So if we can fix those issues, hopefully we can bring
__dataframe__to life.rgommers commentedon Sep 13, 2022
This is a very old issue, let close it.
I think we're slowly getting there - 4 implementations in the wild now, and it seems to cover what scikit-learn needs in principle.
Thanks all!