Repository navigation
Convert pandas str indexes to numpy StringDType consistently #11785
Description
What is your issue?
Follow-up to #11322. With pandas 3, a default string index (StringDtype(na_value=nan)) only becomes a numpy StringDType coordinate when it's passed as the dim of concat. Everywhere else it becomes object:
import pandas as pd
import xarray as xr
idx = pd.Index(["a", "b"], name="x")
db = xr.DataArray(1)
xr.DataArray([1, 2], coords={"x": idx}, dims="x").x.dtype # object
xr.Dataset(coords={"x": idx}).x.dtype # object
xr.DataArray([1, 2], dims="x").assign_coords(x=idx).x.dtype # object
pd.Series([1, 2], index=idx).to_xarray().x.dtype # object
xr.concat([db, db], dim=xr.Variable("x", idx)).x.dtype # object
xr.concat([db, db], dim=idx).x.dtype # StringDType(na_object=nan)(xarray main, pandas 3.0.6, numpy 2.5.4)
The reason is that PandasIndexingAdapter only maps pd.StringDtype to np.dtypes.StringDType when the dtype is passed explicitly (#11102), and only the concat dim path in _calc_concat_dim_index does that. In all other cases PandasIndexingAdapter falls back to get_valid_numpy_dtype (xarray/core/utils.py), which returns object for any non-numpy dtype.
As discussed in #11322 (comment), the plan is to use StringDType everywhere, following #11098. Since the minimum supported numpy version is now 2.2, StringDType is always available, so no object fallback for older numpy versions is needed. That probably means:
- Map
pd.StringDtypetonp.dtypes.StringDType(na_object=dtype.na_value)inget_valid_numpy_dtype(or wherever the default coord dtype is picked), so thatPandasIndexandPandasIndexingAdapterdon't need the dtype passed explicitly. - Put this mapping in a single shared helper; it currently exists in
PandasIndexingAdapter.__init__and, after Fix concat failure with pandas StringDtype indexes (#11317) #11322, also in_calc_concat_dim_index. - Decide whether non-index variables should follow, e.g. a pandas
strcolumn inDataset.from_dataframe, which currently also becomesobject.
Activity
headtr1ck commentedon Oct 22, 2026
If
StringDTypebecomes the default for pandas strings, the.straccessor needs some work too. I compared every accessor method on the same data as<Uand asStringDType(). Without missing values, almost everything gives the same values, and string-returning methods keepStringDType. These are the gaps (xarraymain, numpy 2.5.4):extractwithdim: elements of aStringDTypearray are plain Pythonstr, soval.dtypein_get_res_multifails. Usingdtype=objectthere would be enough, since the result is cast afterwards anyway.<U/StringDType:_stringifycasts arguments withastype(self._obj.dtype.kind), i.e.astype("U"), which numpy doesn't allow forStringDType.na_objectis lost: several methods cast their result withastype(self._obj.dtype.kind)("T"), which gives the defaultStringDType()(e.g.cat/+,split,partition,extract).na_object=np.nan(what pandas 3strdata maps to) orNone, almost every method fails. Object arrays withNaNfail the same way, so the accessor never supported missing values, but withStringDTypeit becomes much more common.test_accessor_str.pyonly coversnp.str_andnp.bytes_.FIX: Preserve pandas StringDtype on xarray coordinates
FIX: Preserve pandas StringDtype and missing markers for coordinates
FIX: Preserve pandas StringDtype across xarray coordinates and DataFr…
FIX: Preserve pandas StringDtype across coordinates and DataFrame export
FIX: preserve pandas StringDtype in coordinate and DataFrame round-trips
FIX: preserve pandas StringDtype coordinates and unsupported missing …
FIX: consistently preserve pandas StringDtype in xarray coordinates
TST: Align StringDType repr doctests and static typing with CI
FIX: consistently preserve pandas StringDtype in xarray coordinates
TST: Align StringDType repr doctests and static typing with CI