Repository navigation
Related topic: dataframe protocol for data interchange/export #1
Description
In March'20 there was a very detailed discussion about introduction a new __dataframe__ protocol: https://discuss.ossdata.org/t/a-dataframe-protocol-for-the-pydata-ecosystem/267. The purpose of it is being able to exchange data between different implementations, or export data to (e.g.) Apache Arrow or NumPy.
There's a strawman implementation at wesm/dataframe-protocol#1.
The discussion went a little all over the place, with many people misunderstanding the main purpose was data exchange rather than providing an API to manipulate or do computations with a dataframe. That would be a much larger topic, and something this consortium aims to deliver an RFC for.
That said, the __dataframe__ topic is very much related, and is also a potentially interesting example of a cross-dataframe-library topic that could really benefit from having a detailed RFC with requirements and use cases. We should consider picking up that topic, and consider lessons from it in community engagement.
Activity
szha commentedon Jun 18, 2020
It seems feasible to build this on top of array exchange. Having clarity first on array exchange seems like a low hanging fruit. To me the
__cuda_array_interface__doesn't seem generalizable to the future accelerators (e.g. it doesn't make sense to make__tpu{1,2,3,4}_array_interface__) so the pydata API definition would be a good opportunity to fix it.rgommers commentedon Sep 9, 2021
Time to close this. A lot of work has been done here; see https://data-apis.org/dataframe-protocol/latest/ and the "interchange protocol" label.