Repository navigation
[Python] Interchange pa.Table's Column.null_count doesn't count NaNs #34774
Description
Describe the bug, including details regarding any error messages, version, and platform.
It seems the interchange Column.null_count() (relevant spec) has erroneous behaviour
>>> import pyarrow as pa
>>> pa.__version__
'12.0.0.dev304' # from https://pypi.fury.io/arrow-nightlies/
>>> df = pa.table([pa.array([float("nan")], type=pa.float64())], ["foo"])
>>> dfi = df.__dataframe__()
>>> col = dfi.get_column(0)
>>> col.null_count
0 # should be 1I assume this is because Arrow does not treat NaNs as nulls, which semantically makes sense, but in the interchange protocol it should—see vaexio/vaex#2120 for a related discussion.
See pandas for expected behaviour
>>> import pandas as pa
>>> df = pd.DataFrame({"foo": [float("nan")]})
>>> dfi = df.__dataframe__()
>>> col = dfi.get_column(0)
>>> col.null_count
1cc @AlenkaF (let me know if not to tag you on things! coincidentally I was working on data-apis/dataframe-interchange-tests#20 today when Ralf commented heh.)
Component(s)
Python
Activity
[-]Interchange `pa.Table`'s `Column.null_count` doesn't count NaNs[/-][+][Python] Interchange `pa.Table`'s `Column.null_count` doesn't count NaNs[/+]AlenkaF commentedon Mar 30, 2023
Do tag me please! =)
Very cool you both jumped on it so quickly hehe.
But we agree with Maarten 😊 vaexio/vaex#2120 (comment)
There would be loss of information for all the libraries that actually differentiate between nan and null, right?
AlenkaF commentedon Apr 2, 2023
Closing this issue as we have received confirmation from the protocol side that
nanandnullcan be treated separately, see discussion in data-apis/dataframe-api#126