Dask dashboards don't work from Dask Gateway Helm release 2022.11.0 possibly due to bokeh version #7512

Note

This has been reported at dask/dask-gateway#676, however it seems from consideRatio's reply this may need reporting here (I haven't yet managed to resolve).

Describe the issue:

Since upgrading our Dask Gateway version (deployed with Helm on K8s) to >2022.10.0, the dashboards don't seem to be working.

The error must be related to the Python version (and libraries) running in the Dask scheduler pod, since I've experimented with different client environments and different Dask Gateway deployments; holding all things constant client-side, and upgrading DG from Helm release 2022.10.0 to 2022.11.0, the issue manifests.

When trying to view the dashboard URL from 2022.11.0 onwards, an error message in the browser states: "Dask needs bokeh >= 2.1.1, < 3 for the dashboard."

If I manually jump into the spawned dask-scheduler pod, I can verify that this is likely the cause; using pip3 freeze | grep bokeh in that pod, the versions corresponding to each DG Helm release are:

i.e. as soon as the scheduler version exceeds v3, the dashboards seemingly break.

I tried to manually reinstall an older version of bokeh in the scheduler pod, to no avail.

Minimal Complete Verifiable Example:

Client side code I've been using to get the dashboard URL:

# List Gateway clusters:
from dask_gateway import GatewayCluster
cluster = GatewayCluster('http://<ip_address>/dask-gateway/', auth="jupyterhub")

cluster.scale(3)

client = cluster.get_client()
client

client widget yields a dashboard URL like: http://<ip_address>/dask-gateway/clusters/my-dask-gateway.ea1...246/status

which is what I've been using to test (i.e. not the Dask Jupyter extension)

Environment:

As per Helm versions discussed above.