need a proper way to change the bucket

Good day!

I tried to make the next experiment.

I had a working ObjectStore with a proper schedule.

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: keycloakdb-daily
  namespace: keycloak
spec:
  backupOwnerReference: self
  cluster:
    name: keycloakdb
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io
  schedule: 0 2 * * *

apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: keycloak
  namespace: keycloak
spec:
  configuration:
    data:
      compression: gzip
      jobs: 2
    destinationPath: s3://[mybucketname]/
    endpointURL: https://storage.yandexcloud.net
    s3Credentials:
      accessKeyId:
        key: AWS_ACCESS_KEY_ID
        name: s3-yandex
      secretAccessKey:
        key: AWS_SECRET_ACCESS_KEY
        name: s3-yandex
    wal:
      compression: zstd
      maxParallel: 8

Now I want to change:

bucketname to another one
and appropriately change the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

So I naively changing the content of the s3-yandex secret and apply the corrected ObjectStore. But.... nothing happens.

I see the next in the logs of barman:

{"level":"info","ts":"2025-06-13T11:30:07.452298887Z","msg":"Skipping retention policy enforcement, no retention policy specified","logging_pod":"keycloakdb-1"}
{"level":"info","ts":"2025-06-13T11:35:11.141188431Z","msg":"Skipping retention policy enforcement, no retention policy specified","logging_pod":"keycloakdb-1"}
{"level":"info","ts":"2025-06-13T11:40:14.628910791Z","msg":"Skipping retention policy enforcement, no retention policy specified","logging_pod":"keycloakdb-1"}
{"level":"info","ts":"2025-06-13T11:45:18.580604564Z","msg":"Skipping retention policy enforcement, no retention policy specified","logging_pod":"keycloakdb-1"}
{"level":"info","ts":"2025-06-13T11:50:21.684530075Z","msg":"Skipping retention policy enforcement, no retention policy specified","logging_pod":"keycloakdb-1"}
{"level":"info","ts":"2025-06-13T11:55:13.513036839Z","msg":"Executing barman-cloud-wal-archive","logging_pod":"keycloakdb-1","walName":"/var/lib/postgresql/data/pgdata/pg_wal/0000000100000003000000AF","options":["--zstd","--endpoint-url","https://storage.yandexcloud.net","--cloud-provider","aws-s3","s3://testcloudnative/","keycloakdb","/var/lib/postgresql/data/pgdata/pg_wal/0000000100000003000000AF"]}
{"level":"info","ts":"2025-06-13T11:55:16.342916796Z","msg":"Archived WAL file","logging_pod":"keycloakdb-1","walName":"/var/lib/postgresql/data/pgdata/pg_wal/0000000100000003000000AF","startTime":"2025-06-13T11:55:13.512903992Z","endTime":"2025-06-13T11:55:16.34287209Z","elapsedWalTime":2.829968224}
{"level":"info","ts":"2025-06-13T11:55:25.139734503Z","msg":"Skipping retention policy enforcement, no retention policy specified","logging_pod":"keycloakdb-1"}
{"level":"info","ts":"2025-06-13T12:00:13.96310529Z","msg":"Executing barman-cloud-wal-archive","logging_pod":"keycloakdb-1","walName":"/var/lib/postgresql/data/pgdata/pg_wal/0000000100000003000000B0","options":["--zstd","--endpoint-url","https://storage.yandexcloud.net","--cloud-provider","aws-s3","s3://testcloudnative/","keycloakdb","/var/lib/postgresql/data/pgdata/pg_wal/0000000100000003000000B0"]}
{"level":"info","ts":"2025-06-13T12:00:16.812934587Z","msg":"Archived WAL file","logging_pod":"keycloakdb-1","walName":"/var/lib/postgresql/data/pgdata/pg_wal/0000000100000003000000B0","startTime":"2025-06-13T12:00:13.963073964Z","endTime":"2025-06-13T12:00:16.812896472Z","elapsedWalTime":2.849822661}
{"level":"info","ts":"2025-06-13T12:00:28.490192702Z","msg":"Skipping retention policy enforcement, no retention policy specified","logging_pod":"keycloakdb-1"}
{"level":"info","ts":"2025-06-13T12:05:13.279134066Z","msg":"Executing barman-cloud-wal-archive","logging_pod":"keycloakdb-1","walName":"/var/lib/postgresql/data/pgdata/pg_wal/0000000100000003000000B1","options":["--zstd","--endpoint-url","https://storage.yandexcloud.net","--cloud-provider","aws-s3","s3://testcloudnative/","keycloakdb","/var/lib/postgresql/data/pgdata/pg_wal/0000000100000003000000B1"]}
{"level":"info","ts":"2025-06-13T12:05:15.905317755Z","msg":"Archived WAL file","logging_pod":"keycloakdb-1","walName":"/var/lib/postgresql/data/pgdata/pg_wal/0000000100000003000000B1","startTime":"2025-06-13T12:05:13.278914875Z","endTime":"2025-06-13T12:05:15.905254787Z","elapsedWalTime":2.62633958}
{"level":"info","ts":"2025-06-13T12:05:32.282742702Z","msg":"Skipping retention policy enforcement, no retention policy specified","logging_pod":"keycloakdb-1"}
{"level":"info","ts":"2025-06-13T12:10:35.701574771Z","msg":"Skipping retention policy enforcement, no retention policy specified","logging_pod":"keycloakdb-1"}
{"level":"info","ts":"2025-06-13T12:15:38.925811524Z","msg":"Skipping retention policy enforcement, no retention policy specified","logging_pod":"keycloakdb-1"}
{"level":"info","ts":"2025-06-13T12:20:42.222088847Z","msg":"Skipping retention policy enforcement, no retention policy specified","logging_pod":"keycloakdb-1"}
{"level":"info","ts":"2025-06-13T12:20:44.612536625Z","msg":"Deleting backup not in the catalog","logging_pod":"keycloakdb-1","backup":"keycloakdb-daily-20250612230200"}
{"level":"info","ts":"2025-06-13T12:20:44.620135745Z","msg":"Deleting backup not in the catalog","logging_pod":"keycloakdb-1","backup":"keycloakdb-daily-20250613000200"}
{"level":"info","ts":"2025-06-13T12:20:44.626424912Z","msg":"Deleting backup not in the catalog","logging_pod":"keycloakdb-1","backup":"keycloakdb-daily-20250613010200"}
{"level":"info","ts":"2025-06-13T12:20:44.633155459Z","msg":"Deleting backup not in the catalog","logging_pod":"keycloakdb-1","backup":"keycloakdb-daily-20250613020200"}
{"level":"info","ts":"2025-06-13T12:20:44.638740297Z","msg":"Deleting backup not in the catalog","logging_pod":"keycloakdb-1","backup":"keycloakdb-daily-20250613030200"}
{"level":"info","ts":"2025-06-13T12:20:44.646144456Z","msg":"Deleting backup not in the catalog","logging_pod":"keycloakdb-1","backup":"keycloakdb-daily-20250613040200"}
{"level":"info","ts":"2025-06-13T12:20:44.652175408Z","msg":"Deleting backup not in the catalog","logging_pod":"keycloakdb-1","backup":"keycloakdb-daily-20250613050200"}
{"level":"info","ts":"2025-06-13T12:20:44.659974535Z","msg":"Deleting backup not in the catalog","logging_pod":"keycloakdb-1","backup":"keycloakdb-daily-20250613060200"}
{"level":"info","ts":"2025-06-13T12:20:44.665997349Z","msg":"Deleting backup not in the catalog","logging_pod":"keycloakdb-1","backup":"keycloakdb-daily-20250613070200"}
{"level":"info","ts":"2025-06-13T12:20:44.671859722Z","msg":"Deleting backup not in the catalog","logging_pod":"keycloakdb-1","backup":"keycloakdb-daily-20250613080200"}
{"level":"info","ts":"2025-06-13T12:20:44.678399927Z","msg":"Deleting backup not in the catalog","logging_pod":"keycloakdb-1","backup":"keycloakdb-daily-20250613090200"}
{"level":"info","ts":"2025-06-13T12:20:44.685202587Z","msg":"Deleting backup not in the catalog","logging_pod":"keycloakdb-1","backup":"keycloakdb-daily-20250613100200"}
{"level":"info","ts":"2025-06-13T12:20:44.692265438Z","msg":"Deleting backup not in the catalog","logging_pod":"keycloakdb-1","backup":"keycloakdb-daily-20250613110200"}
{"level":"info","ts":"2025-06-13T12:20:44.699563098Z","msg":"Deleting backup not in the catalog","logging_pod":"keycloakdb-1","backup":"keycloakdb-daily-20250613120200"}
{"level":"info","ts":"2025-06-13T12:25:44.754412291Z","msg":"Skipping retention policy enforcement, no retention policy specified","logging_pod":"keycloakdb-1"}
{"level":"info","ts":"2025-06-13T12:30:47.517659989Z","msg":"Skipping retention policy enforcement, no retention policy specified","logging_pod":"keycloakdb-1"}
{"level":"info","ts":"2025-06-13T12:35:50.152164082Z","msg":"Skipping retention policy enforcement, no retention policy specified","logging_pod":"keycloakdb-1"}

My expectations: that the backups would be streamed to a new bucket.

when creating a NEW secret + ObjectStore + ScheduledBackup - everything works like a charm

helped:

removal of ObjectStore + ScheduledBackup
removal of plugins section in Cluster
waiting till reconciliation will occur
applying manifests back

I think the Deleting backup not in the catalog lines in the logs above come from a bug in the catalog maintenance rather than from the bucket change itself, and it can be triggered by changing the ObjectStore a cluster points to, or its serverName.

Cause

After applying the retention policy, deleteBackupsNotInCatalog (internal/cnpgi/instance/retention.go) lists the backups at the cluster's current location (barmanObjectName + serverName) and deletes every completed Backup object of the cluster whose backupID is not in that list.

The only filter on which Backup objects are considered is useSameBackupLocation:

meta := newBackupResultMetadataFromMap(backup.PluginMetadata)
return meta.clusterUID == string(cluster.UID) && meta.pluginName == metadata.PluginName

Despite its name, it doesn't compare any location: it only checks that the backup belongs to the same cluster and was taken by this plugin. So when the location changes, every Backup taken against the previous one is missing from the new catalog and gets deleted, even though the data is still in the old bucket. That matches the logs in this issue, where all the daily backups were deleted at the first maintenance run after the change.

The same happens when:

the cluster's barmanObjectName is switched to another ObjectStore;
the serverName plugin parameter is changed (for example to start a new archive folder in the same bucket);
the ObjectStore's destinationPath is edited in place, as above.

Fixing it

Today the plugin can't do better, because a plugin Backup doesn't record where it was written. #1140 proposes recording barmanObjectName, serverName, destinationPath and endpointURL in Backup.status.pluginMetadata. With those keys, useSameBackupLocation could skip Backup objects that were taken against another location. Backups without the keys (taken before the change) would keep today's behaviour.

#1115 reports the same deletion path being triggered by a short barman-cloud-backup-list result. It's a different trigger, but both come from this pass deleting objects based on a catalog that doesn't necessarily cover them.

How I found it

While implementing #1140 on my fork, I ran an AI code review (Claude Code) of the change. It pointed out that useSameBackupLocation ignores the location. I then checked the code path above by hand on main and found this issue matching the behaviour. I haven't reproduced it on a live cluster.

I used an AI assistant (Claude) to help find this and draft this comment, as the AI policy asks for disclosure.

Navigation Menu

Repository navigation

need a proper way to change the bucket #405

Cause

Fixing it

How I found it

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

need a proper way to change the bucket #405

Description

Activity

gecube commented on Jun 13, 2025

gecube commented on Jun 13, 2025

BoxBoxJason commented on Oct 4, 2026

Cause

Fixing it

How I found it

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions