Repository navigation
Catalog maintenance deletes completed Backup objects after a short barman-cloud-backup-list result #1115
Description
Versions: plugin-barman-cloud v0.13.0, CloudNativePG 1.30.0, Kubernetes 1.35, S3-compatible object storage (STACKIT).
What happened
On 2026-09-18 the plugin sidecar on the primary of two unrelated clusters logged Deleting backup not in the catalog for the two newest completed Backup objects (the base backups of the 17th and the 18th) and removed them:
{"level":"info","ts":"2026-09-18T09:50:05Z","msg":"Applying backup retention policy","logging_pod":"portal-postgres-2","retentionPolicy":"30d"}
{"level":"info","ts":"2026-09-18T09:50:07Z","msg":"Deleting backup not in the catalog","logging_pod":"portal-postgres-2","backup":"portal-postgres-daily-backup-20260917011500"}
{"level":"info","ts":"2026-09-18T09:50:07Z","msg":"Deleting backup not in the catalog","logging_pod":"portal-postgres-2","backup":"portal-postgres-daily-backup-20260918011500"}
The operator then logged terminal error: Backup.postgresql.cnpg.io "portal-postgres-daily-backup-20260917011500" not found for both objects. The same happened on a second cluster in another namespace at 13:05 UTC.
The bucket itself was intact: ObjectStore.status.serverRecoveryWindow kept its firstRecoverabilityPoint (2026-08-25) and lastSuccessfulBackupTime, WAL archiving never failed, and the next night's scheduled backup completed normally. The kube-apiserver showed a short 5xx spike in the same minute (about 16 errors/s for two five-minute windows, otherwise ~0.02/s), so the listing most likely came back short or empty.
Cause, as far as I can see
internal/cnpgi/instance/retention.go deletes every completed Backup object of the cluster whose status.backupID is not in the current GetBackupList result. There is no guard for the listing itself: an empty or partial catalog (transient object-store or API error, eventual consistency) deletes valid objects. On this side the effect was a false no backup in 26h alert; on a cluster that relies on Backup objects for restore selection it would hide two valid base backups.
Suggestions
- Skip the deletion pass when the listing is empty, or when it lists fewer backups than the
Backupobjects that are older than the newest catalog entry. - Only delete objects whose backup is older than the retention window, since that is the only case where the catalog is expected to have dropped them.
- Or require an ID to be missing in two consecutive listings before the object is removed.
Happy to provide more logs or test a fix.
Activity
Lumbenlengo commentedon Sep 27, 2026
Hi @ebertlion, I’d like to help fix this issue. Are you already working on it?
bisand commentedon Oct 8, 2026
We see the same thing on a different provider, with a slightly different trigger: a partial listing rather than an empty one.
Versions: plugin-barman-cloud v0.14.0, CloudNativePG 1.29.1, PostgreSQL 18.4, Hetzner Object Storage (hel1),
retentionPolicy: 30d, one scheduled base backup per day.What we observed (34 days of sidecar logs on the primary)
Deleting backup not in the catalog. 17 were expected (backups past the 30-day window, removed right after the nightly backup). 8 were for completed backups 4–26 days old, removed at arbitrary times of day, one object at a time.backup.info+data.tar.gz). The bucket holds 31 daily backups;kubectl get backups.postgresql.cnpg.ioshows 23.GatewayTimeoutonGetObjectfor a singlebackup.info41 times, visible throughbarman-cloud-backup-delete:Likely mechanism
barman-cloud-backup-deletetreats an unreadablebackup.infoas fatal and exits non-zero (we getRetentionPolicyFailed).barman-cloud-backup-listappears to skip the unreadable backup with only a warning and exit 0, soGetBackupListreturns a list that is one entry short anddeleteBackupsNotInCatalogremoves thatBackupobject. This part is inferred: the sidecar logs nothing from the list step on success, so we have no log line tying a timeout to each deletion — only the timing pattern and the intact bucket.If that is right, an "empty listing" guard would not cover this case. Of the suggestions above, deleting only objects older than the retention window, or requiring an ID to be missing in two consecutive listings, would. Surfacing barman's unreadable-backups warning as an error from the list step would also do it.
Happy to share more logs.