You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
setup-gcp: bootstrap deletes and recreates an existing cluster when its subnetwork, network, or Dataplane V2 setting differs #2341
When setup-gcp bootstrap (or create cluster) finds that the cluster already exists, its reconcile path deletes the cluster and creates a new one if any of three settings differ from the tool's config (cluster.go#L161-L190):
the network (projects/<PROJECT_ID>/global/networks/<NETWORK>);
the subnetwork (projects/<PROJECT_ID>/regions/<GCE_REGION>/subnetworks/<SUBNETWORK>);
Dataplane V2 on or off.
There's no prompt and no flag guarding it. The log line before the delete is INFO Mismatch in subnetwork. Everything running on the cluster is lost.
Why it's easy to hit
bootstrap expects Dataplane V2 without exposing the setting. Only create cluster registers --enable-dataplane-v2 (L403). But the two commands share the global cfg, and that registration sets cfg.EnableDataplaneV2 = getEnv("ENABLE_DATAPLANE_V2", true) during init(). So bootstrap silently expects Dataplane V2. I confirmed it with a throwaway test against the tree: bootstrap sees EnableDataplaneV2 == true. GKE Standard clusters use the legacy datapath unless created with --enable-dataplane-v2, so any such cluster passed to bootstrap is deleted.
The subnetwork defaults to default. GKE now commonly creates a per-cluster subnet named gke-<cluster>-subnet-<hash>. In one project I checked (13 clusters, a mix of Standard and Autopilot), 7 clusters are on a subnet other than default, and 2 more are on the legacy datapath. Running bootstrap with default settings against any of those 9 would delete and recreate it.
Shared VPC can never match. The expected network is built from PROJECT_ID. A cluster in a service project whose network lives in the host project reports projects/<HOST>/global/networks/..., so the suffix check fails whatever NETWORK is set to.
Proposed fix
Never delete an existing cluster implicitly. On a mismatch, stop with an error that names the setting and both values, and say how to proceed: recreate it yourself, or use a different cluster name. If automatic recreation is wanted for dev loops, put it behind an explicit flag such as --recreate-on-mismatch.
Register --enable-dataplane-v2 / ENABLE_DATAPLANE_V2 on bootstrap too, so its expectation is visible and can be overridden. Better still, only compare network, subnetwork and datapath when the caller set them explicitly.
The GKE installer runs bootstrap against existing clusters, so I'm adding a guard there now, passing the cluster's actual network, subnetwork and datapath so the reconcile finds nothing to recreate. An upstream fix would protect everyone else who runs bootstrap against an existing cluster. Happy to send the PR.
Reproduced on a throwaway GKE cluster. It was created with a GKE-made subnet (--create-subnetwork "") and left on the legacy datapath, which are the defaults for many Standard clusters. I ran setup-gcp create cluster at 756c2a5 with SUBNETWORK=default; bootstrap runs the same reconcile at step 2/7. It deleted the cluster without a prompt:
INFO Cluster exists. Checking attributes... cluster=net-guard-probe
INFO Mismatch in subnetwork current=projects/<project>/regions/us-central1/subnetworks/gke-net-guard-probe-subnet-b2563343 expected=projects/<project>/regions/us-central1/subnetworks/default
INFO Deleting cluster cluster=net-guard-probe
With NETWORK, SUBNETWORK and ENABLE_DATAPLANE_V2 set to the cluster's own values, the same reconcile passed all three checks and moved on to Workload Identity. That's the workaround the GKE installer is adopting in ai-on-gke/substrate-gke for now.
Fixed in the copy that moved to ai-on-gke/substrate-gke (#55 there): bootstrap now warns and keeps the cluster instead of recreating it. Still applies to the setup-gcp in this repo.
Activity
mastersingh24 commentedon Oct 8, 2026
Reproduced on a throwaway GKE cluster. It was created with a GKE-made subnet (
--create-subnetwork "") and left on the legacy datapath, which are the defaults for many Standard clusters. I ransetup-gcp create clusterat756c2a5withSUBNETWORK=default;bootstrapruns the same reconcile at step 2/7. It deleted the cluster without a prompt:With
NETWORK,SUBNETWORKandENABLE_DATAPLANE_V2set to the cluster's own values, the same reconcile passed all three checks and moved on to Workload Identity. That's the workaround the GKE installer is adopting in ai-on-gke/substrate-gke for now.Hand bootstrap an existing cluster's own network, or refuse the cluster
mastersingh24 commentedon Oct 18, 2026
Fixed in the copy that moved to ai-on-gke/substrate-gke (#55 there): bootstrap now warns and keeps the cluster instead of recreating it. Still applies to the setup-gcp in this repo.