Using Array API functions on DataFrame objects #50

Activity

1. rgommers commented on Aug 23, 2021

   

   rgommers

   on Aug 23, 2021

   Member

   More actions

   In some cases users like to use Array API functions (for example where) on DataFrame objects (in particular Series). Is this something that we would like to support in the API? If not, how would we recommend users approach these kinds of problems.

   We had (/have) a pretty strong consensus that there should not be a separate Series-like object, but only a DataFrame object with a single column. IIRC the key issue is that Series and DataFrame have so much API duplication, for little benefit.

   That's not a complete answer though. Your question can be rephrased as something like:

   • "Can array API functions be applied to dataframes?", or
   • "Is there a connection between the array and dataframe APIs?"

   It in principle makes sense to me to have array API functions work on dataframes with a homogeneous dtype (which includes single-column dataframes). I'm not sure there's a good way to pick and choose what functions in the array API make sense. I can imagine a dataframe library providing the whole array API somehow, or to reuse an existing array library that is a dependency.

   👍React with 👍3Reacted by Leo Fang, jakirkham and Chris Pryer

   ❤️React with ❤️1Reacted by Chris Pryer
2. jakirkham commented on Sep 2, 2021

   

   jakirkham

   on Sep 2, 2021

   MemberAuthor

   More actions

   When discussing this in the call a few weeks back (and please feel free to correct me), we explored a few options. In the end gravitated towards having some way to share/convert data between DataFrames and Arrays with the idea one could then use Array operations on the converted Array. If the underlying library doesn't have an actually Array, they could just return some object that implements the Array API. Here are some related discussions on this topic ( #25 ) ( #39 ) ( #48 )

   cc @jorisvandenbossche (in case I missed anything here 🙂)

   👍React with 👍2Reacted by Leo Fang and Ralf Gommers
3. jbrockmendel commented on Sep 14, 2022

   

   jbrockmendel

   on Sep 14, 2022

   Contributor

   More actions

   is something analogous to __array_ufunc__ or __array_function__ an option? IIUC the numpy devs haven't always been happy with these designs
4. rgommers commented on Sep 14, 2022

   

   rgommers

   on Sep 14, 2022

   Member

   More actions

   is something analogous to __array_ufunc__ or __array_function__ an option?

   The array API standard has __array_namespace__. If Column is array API standard compliant and has an __array_namespace__ method, things will work.

   IIUC the numpy devs haven't always been happy with these designs

   Indeed. __array_namespace__ is based on the design ideas that were trying to address the limitations of __array_ufunc/function__.
5. rgommers

   added

   API design

   on Sep 29, 2022
6. rgommers commented on Apr 27, 2023

   

   rgommers

   on Apr 27, 2023

   Member

   More actions

   We had (/have) a pretty strong consensus that there should not be a separate Series-like object, but only a DataFrame object with a single column.

   We collectively changed our minds on this one. I think that came over time with (a) the realization that we're really building a library author-focused API that is quite different from the public APIs in current dataframe libraries, (b) that there are actually things that a one-column dataframe does not do well enough (e.g. we now are introducing unique values function for Column which is pretty clean; unique for DataFrame would be different and much more complex), and (c) that a column object which is array-like but with solid missing data support and string/datetime/categorical dtypes is a useful primitive.

   So, I think we still want the thing here that @jakirkham originally asked for - but on Column rather than on DataFrame.