Repository navigation
Representing & checking Dataset schemas #1900
Description
What would be the best way to canonically describe a dataset, which could be read by both humans and machines?
For example, frequently in our code we have docstrings which look something like:
def get_returns(security_ids):
"""
Retuns mega-dimensional dataset which gives recent returns for a set of
securities by:
- Date
- Return (raw / economic / smoothed / etc)
- Scaling (constant / risk_scaled)
- Span
- Hedged vs Unhedged
Dataset keys are security ids. All dimensions have coords.
"""
This helps when attempting to understand what code is doing while only reading it.
But this isn't consistent between docstrings and can't be read or checked by a machine.
Has anyone solved this problem / have any suggestions for resources out there?
Tangentially related to python/typing#513 (but our issues are less about the type, dimension sizes, and more about the arrays within a dataset, their dimensions, and their names)
Activity
shoyer commentedon Feb 9, 2018
I think the right word for this may be "schema". For applications and models (rather than data analysis), these sort of conventions can be super-valuable. I like the idea of declarative spec that can be validated.
Just googling around, I came up with pandas-validator: https://github.com/c-data/pandas-validator
max-sixty commentedon Feb 9, 2018
Right! 🤦♂️
Interesting, thanks.
Do you think this fits into a 'function which validates', rather than a Mypy-like type annotation? I think ideally there would be a representation of the schema that could work with both, so maybe this isn't the important question atm.
max-sixty commentedon Feb 9, 2018
And let me know if there are already textual schema definitions from other libraries that you think are good, before we go and build our own (we don't work with any netCDF-like files so don't have that context)
shoyer commentedon Feb 9, 2018
ncdump -h(xarray.Dataset.info()) is one existing schema of sorts, but it's hardly machine readable.benbovy commentedon Feb 22, 2018
Somewhat related to this issue, I have implemented in xarray-simlab some logic to validate
xarray.Variableobjects (dimensions, dtype, etc.). See this base class and some sub-classes here. I use that in a way which is quite similar topandas-validator(i.e., using class attributes).I'm currently in the process of refactoring this using attrs, which supports both validator functions and type annotations. Not sure how to use the latter for xarray objects, though (BTW I wasn't aware of python/typing#513, good to know!!).
I agree that it would be nice to have a more generic way to describe xarray objects that can be reused in many contexts.
max-sixty commentedon Feb 22, 2018
@benbovy That looks v interesting.
I think at the moment it would require a bit of work to validate normal xarray objects, is that right? (I'm looking at the
__init__, which doesn't take the traditional args supplied to aVariable- tell me if I'm misreading it)Separately - I didn't know about the project but looks awesome. Do we have a list of projects that integrate xarray? Let's start one somewhere if not @pydata/xarray ?
benbovy commentedon Feb 22, 2018
@maxim-lian you're right. In this case
xsimlab.Variableis a different concept thanxarray.Variable, despite that they both have the same name. The former is tight to the modelling framework while the latter is only used for simulation inputs and outputs in xarray-simlab.There is an ongoing discussion in #1850 about having something like xarray-contrib (likely a github organization).
max-sixty commentedon Mar 28, 2018
The commentary in python/typing#513, and @shoyer 's doc https://docs.google.com/document/d/1vpMse4c6DrWH5rq2tQSx3qwP_m_0lyn-Ij4WHqQqRHY/edit#heading=h.rkj7d39awayl are good & growing
I'll close this as I think riding on those coattails - with the addition of names and Datasets as containers - makes the most sense.
(though reopen if we think there's something we could productively do separately)
JackKelly commentedon Oct 8, 2021
I'm really interested in a machine-readable schema for xarray!
Pandera provides machine-readable schemas for Pandas and, as of version 0.7, panderas has decoupled pandera and pandas types to make pandera more useful for things like xarray. I haven't tried
panderayet but I plan to do some experiments soon.[-]Representing & checking Dataset metadata [/-][+]Representing & checking Dataset schemas [/+]shoyer commentedon Oct 8, 2021
Awesome -- would love to hear how this goes!
JackKelly commentedon Oct 8, 2021
OK, I think
panderaisn't the way forwards because it appears very tighly coupled to Pandas (so, for example, I don't think it's possible to usepanderawith n-dimensional arrays).But Pydantic looks promising. Here's a very quick coding experiment showing one way to use pydantic with xarray... it validates a few things; but it's not super-useful as a human-readable specification for what's going on inside a DataArray or Dataset.
rabernat commentedon Oct 8, 2021
Big 👍 to this.
andersy005 commentedon Jan 9, 2022
pandera.xarraymodule withinpanderajhamman commentedon Jan 10, 2022
Related to the Pandera integration, we are prototyping the xarray schema validation functionality in the xarray-schema project.
kubaraczkowski commentedon Jul 14, 2022
Does this project do (part of?) what's needed?
+1 on making xarrays with explicit 'structure' !