opendap error with version 1.7.0

Hello,

We are suffering this error when we read a netcfd throgh opendap with new version 1.7.0, with a netcdf4 version rollback to version 1.6.5 the error dissapeard, hope it helps!

import xarray as xr

xr.open_dataset('https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061300.nc')

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('[https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061300.nc',)](https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061300.nc%27,)), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '85e04132-c9c8-42a9-a6da-0516ae5e8dc1']
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/api.py", line 571, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 645, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 408, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 355, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 417, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 411, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/usr/local/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2469, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2028, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -90] NetCDF: file not found: '[https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061300.nc'](https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061300.nc%27)

I can reproduce that error on Linux:

Error:curl error: Problem with the SSL CA cert (path? access rights?)
curl error details: 
Warning:oc_open: Could not read url

It is probably a regression b/c we are using the built-in curl library to build it. I'm working on a docker image that will have the same libraries as the previous wheels.

I can reproduce that error on Linux:
Error:curl error: Problem with the SSL CA cert (path? access rights?)
curl error details: 
Warning:oc_open: Could not read url
It is probably a regression b/c we are using the built-in curl library to build it. I'm working on a docker image that will have the same libraries as the previous wheels.

same error when launch this inside a dockered API... thanks!

Looks like we need these unreleased commits:

Unidata/netcdf-c#2690

and the previous wheel was built against latest code instead of a release:

MacPython/netcdf4-python-wheels@b611fcd

I'm modifying the new wheel config to do the same but it would be nice to get a new release of netcdf-c from upstream.

@aragong

@aragong can you confirm that it indeed fixes the problem when you used 1.6.5? Or give more context? I tried:

$ pip freeze
certifi==2024.6.2
cftime==1.6.4
netCDF4==1.6.5
numpy==1.26.4

and then your URL

$ ncinfo "https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061300.nc"
Traceback (most recent call last):
  File "/envs/TEST/bin/ncinfo", line 8, in <module>
    sys.exit(ncinfo())
  File "/envs/TEST/lib/python3.10/site-packages/netCDF4/utils.py", line 551, in ncinfo
    f = Dataset(filename)
  File "src/netCDF4/_netCDF4.pyx", line 2469, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2028, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -90] NetCDF: file not found: 'https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061300.nc'

However, the URL used to test this regression in the wheels do pass:

$ ncinfo "https://icdc.cen.uni-hamburg.de/thredds/dodsC/ftpthredds/hamtide/m2.hamtide11a.nc"                            
<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF3_CLASSIC data model, file format DAP2):
    history: FERRET V6.62 03-Feb-10
    title: HAMTIDE11a: m2 ocean tide
    Conventions: CF-1.0
    dimensions(sizes): LAT(1441), LON(2881)
    variables(dimensions): float64 LON(LON), float64 LAT(LAT), float32 RE(LAT, LON), float32 IM(LAT, LON), float32 AMPL(LAT, LON), float32 PHAS(LAT, LON)
    groups:

Or maybe that URL is changes every hour and I should be using https://ihthredds.ihcantabria.com/thredds/dodsC/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/IBI_2024061400.nc now?

@ocefpaf

We fix the problem only changing the netcdf version to the previous one. 1.6.5

Pip uninstall netcdf4 -y
Pip install netcdf4==1.6.5

No idea about certs and wheels @ocefpaf ... Sorry

@aragong

@aragong, while there is a problem with the latest wheel, I do believe that the URL you provided above it no longer valid. Can you check? I can confirm that the new wheel will work for the previous URLs that had that issue but I cannot test against the URL you reported above.

Looks like we need these unreleased commits:

Unidata/netcdf-c#2690

and the previous wheel was built against latest code instead of a release:

MacPython/netcdf4-python-wheels@b611fcd

I'm modifying the new wheel config to do the same but it would be nice to get a new release of netcdf-c from upstream.

Ah yes - I completely forgot about this, sorry. Since 4.9.3 has not been released yet, we do need to build against github develop.

@WardF

@WardF is working hard to getting 4.9.3 across the finish line right now.

@ocefpaf

Sorry for the delay, I was out of the office!

Still having the same error I test the following code:

pip uninstall netcdf4 -y
pip install netcdf4 --no-cache-dir

import xarray as xr

xr.open_dataset('https://ihthredds.ihcantabria.com/thredds/dodsC/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/global/hourly/global_2024061600.nc')

Error:curl error: Problem with the SSL CA cert (path? access rights?)
curl error details: 
Warning:oc_open: Could not read url
Traceback (most recent call last):
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('https://ihthredds.ihcantabria.com/thredds/dodsC/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/global/hourly/global_2024061600.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '392d7c27-b1ce-44e3-88e0-20555338cc3c']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/api.py", line 573, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 646, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 409, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 356, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 418, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 412, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aragong/miniconda3/envs/TESEO.Apiprocess/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2470, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2107, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -68] NetCDF: I/O failure: 'https://ihthredds.ihcantabria.com/thredds/dodsC/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/global/hourly/global_2024061600.nc'

@ocefpaf @jswhit I do not see any recent patch version...
Do you spect the problem to be solved or I should wait for the next patch version?

Thank you for your time!

@aragong

@aragong, while there is a problem with the latest wheel, I do believe that the URL you provided above it no longer valid. Can you check? I can confirm that the new wheel will work for the previous URLs that had that issue but I cannot test against the URL you reported above.

It is operational data, you can check the catalag here: https://ihthredds.ihcantabria.com/thredds/catalog/IBI_ANALYSISFORECAST_PHY_005_001/IBI/hourly/catalog.html

sorry for the delay in my response!

It will be fixed when a new release, or post release, is issued.Ultimately, the problem is not with netcdf4-python but with the underlying netcdf-c. If you are in a hurry you can re-build the wheel with latest netcdf-c from git source, or wait for a new release that will have the fix in it.

It will be fixed when a new release, or post release, is issued.Ultimately, the problem is not with netcdf4-python but with the underlying netcdf-c. If you are in a hurry you can re-build the wheel with latest netcdf-c from git source, or wait for a new release that will have the fix in it.

Thank you very much for the clarification!

I have been working on getting the new netCDF-C release out; the sticking point has been gathering together some S3/AWS documentation. The first release candidate will be out this week, however, while I finish getting all of that together.

@jswhit

I have been working on getting the new netCDF-C release out; the sticking point has been gathering together some S3/AWS documentation. The first release candidate will be out this week, however, while I finish getting all of that together.

Awesome!

@jswhit we can:

issue a post 1.7.0 release with latest dev code, b/c 1.6.5 was already using it, it is not a big deal.
or yank 1.7.0 and wait for the proper netcdf-c release.

What do you think?

Navigation Menu

Repository navigation

opendap error with version 1.7.0 #1335

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

opendap error with version 1.7.0 #1335

Description

Activity

ocefpaf commented on Jun 14, 2024

aragong commented on Jun 14, 2024

ocefpaf commented on Jun 14, 2024

ocefpaf commented on Jun 14, 2024

aragong commented on Jun 14, 2024

ocefpaf commented on Jun 14, 2024

jswhit commented on Jun 14, 2024

dopplershift commented on Jun 14, 2024

aragong commented on Jun 17, 2024

aragong commented on Jun 17, 2024

ocefpaf commented on Jun 17, 2024

aragong commented on Jun 17, 2024

WardF commented on Jun 17, 2024

ocefpaf commented on Jun 17, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions