Repository navigation
[Question] Unnecessary copy in rioxarray reproject? #130
Description
I've been trying out rioxarray's reproject method. I've been noticing an unexpected data copy causing memory errors in my workflow. I'm curious if the following copy is necessary?
Code Sample, a copy-pastable example if possible
N/A
Problem description
See above.
Expected Output
Ideally, we can avoid an extra memory copy. Especially important when dealing with very large datasets.
Environment Information
python -c "import rioxarray; rioxarray.show_versions()"
rioxarray (0.0.28) deps:
rasterio: 1.1.5
xarray: 0.15.1
GDAL: 3.0.4
Other python deps:
scipy: 1.4.1
pyproj: 2.6.1.post1
System:
python: 3.8.3 | packaged by conda-forge | (default, Jun 1 2020, 17:43:00) [GCC 7.5.0]
executable: /srv/conda/envs/notebook/bin/python
machine: Linux-4.19.112+-x86_64-with-glibc2.10
Installation method
- conda,
Conda environment information (if you installed with conda):
$ conda list | grep -E "rasterio|xarray|gdal"
gdal 3.0.4 py38h172510d_6 conda-forge
intake-xarray 0.3.1 py_0 conda-forge
libgdal 3.0.4 h3dfc09a_6 conda-forge
rasterio 1.1.5 py38h033e0f6_0 conda-forge
rioxarray 0.0.28 py_0 conda-forge
xarray 0.15.1 py_0 conda-forge
Details about
conda and system ( conda info ):
Details
$ conda info
active environment : notebook
active env location : /srv/conda/envs/notebook
shell level : 1
user config file : /home/jovyan/.condarc
populated config files : /srv/conda/.condarc
conda version : 4.8.2
conda-build version : not installed
python version : 3.7.6.final.0
virtual packages : __glibc=2.27
base environment : /srv/conda (writable)
channel URLs : https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /srv/conda/pkgs
/home/jovyan/.conda/pkgs
envs directories : /srv/conda/envs
/home/jovyan/.conda/envs
platform : linux-64
user-agent : conda/4.8.2 requests/2.22.0 CPython/3.7.6 Linux/4.19.112+ ubuntu/18.04.4 glibc/2.27
UID:GID : 1000:0
netrc file : None
offline mode : False
Activity
snowman2 commentedon Jun 16, 2020
That bit originates from: https://github.com/opendatacube/datacube-core/blob/084c84d78cb6e1326c7fbbe79c5b5d0bef37c078/datacube/api/geo_xarray.py#L96
@andrewdhicks might have some insight there.
Based on pyproj4/pyproj#573, I am thinking it would be better to replace it with__array__as it seems to handle the dask bit nicely: https://github.com/dask/dask/blob/9968a49bb4e9ae03d29110bbc5a41a4179636ab8/dask/array/core.py#L1336. Thoughts?Probably better to just remove the
np.copybit if it makes sense.andrewdhicks commentedon Jun 18, 2020
@snowman, that datacube code is really old, and was written when I didn't really understand how dask worked, and I think before xarray supported ufuncs. The
copy, and.datacan probably go, and the .load()can go if there's a dask solution.I'm not a datacube core dev anymore, but I think @ Kirill888 is working on a dask-backed reprojection for xarray as well, here:
https://github.com/opendatacube/odc-tools/blob/develop/libs/algo/odc/algo/_warp.py
snowman2 commentedon Jun 18, 2020
@andrewdhicks, thanks for the reply, that was some really great information. Looks like there have been quite a few improvements since then.
snowman2 commentedon Jun 18, 2020
@jhamman do you have any cycles to remove
np.copy?snowman2 commentedon Jun 18, 2020
This can be removed as well:
https://github.com/corteva/rioxarray/blob/6ce2f6fc99891d74fb7ab90a2c1af002ec80b2cf/rioxarray/rioxarray.py#L1230