Repository navigation
Interchange Protocol: unused nan_as_null keyword? #125
Description
The DataFrame Interchange Protocol has a nan_as_null keyword in __dataframe__ that can be specified by the consumer, i.e. the person/library calling this method. The docstring explains its goal:
dataframe-api/protocol/dataframe_protocol.py
Lines 400 to 403 in d10a096
However, at the moment I think none of the existing implementation we are aware of actually supports this keyword: pandas, vaex, modin, cudf all simply ignore it (and cudf actually uses a wrong default), and pyarrow as well but will raise an error if you pass a non-default value (i.e. True).
The following disclaimer in the docstring seems to have been copied around as some version of this appears in all the listed projects:
`nan_as_null` currently has no effect; once support for nullable extension
dtypes is added, this value should be propagated to columns.
I am not fully sure what this note is meant for (I suppose it originates from the pandas implementation where for nullable extension types this keyword could be implemented? Since the default numpy dtypes already use NaN as null, the keyword wouldn't have any effect for that. But for other libraries that explanation doesn't necessarily hold).
And also, if nobody implemented it, do we actually need it?
Activity
kkraus14 commentedon Mar 31, 2023
I would vote to remove this. This is redundant to
describe_nullon the column level with much more rigid capabilities.rgommers commentedon Mar 31, 2023
Seems okay to remove indeed. The original motivation was that Arrow uses bit rather than byte masks, and those aren't supported by NumPy. So the nan-to-null mapping has to be done somewhere, and it's easier to do on the producer than the consumer side. However, Joris says Pandas already has the code to handle this on the consumer side. So, since no one currently uses
nan_as_null, it seems safe to remove it.[-]Interchange Protocol: usused `nan_as_null` keyword?[/-][+]Interchange Protocol: unused `nan_as_null` keyword?[/+]rgommers commentedon Aug 29, 2023
gh-228, which deprecates the
nan_as_nullkeyword, is now merged. As discussed in gh-226, we should ensure it's removed from all the consumers before removing it from the signature of__dataframe__. So let's leave this issue open, and use it as the reference to point to for PRs where the keyword is removed from consumers which call the__dataframe__method.Remove use of `nan_as_null` from callers of `__dataframe__`
DEPR: remove use of `nan_as_null` from callers of `__dataframe__`