Feature request: remote storage tiers in the Tenant CRD #280

Summary

Let the Tenant CRD declare remote storage tiers (ILM warm/cold backends), so a Tenant and its tiers can be managed fully through GitOps. Together with Transition rules in bucket lifecycle (#) this covers ILM tiering end to end.

Use Case

RustFS is the hot tier; objects older than N days move to a remote S3-compatible tier (Storj in our case; RustFS works with it since rustfs/rustfs#8363). Server-side SSE-S3 is enabled, so the remote holds ciphertext only. One Tenant per environment, deployed with Flux.

Current Situation

• RustFS core manages tiers via the admin API: PUT /rustfs/admin/v3/tier (add), GET /rustfs/admin/v3/tier (list), POST /rustfs/admin/v3/tier/{tiername} (edit credentials), DELETE /rustfs/admin/v3/tier/{tiername} (remove).
• The operator has no way to declare a tier.

Current Workaround

A Job that runs rc ilm tier add s3 <TIER> <alias> --endpoint … --bucket … --prefix … --access-key … --secret-key … after the Tenant becomes ready. Problems:

• Imperative, outside the reconcile loop: nothing notices a deleted or changed tier, and there is no status on the Tenant.
• Rotating the remote credentials means editing and re-running the Job by hand.
• With Flux the Job needs kustomize.toolkit.fluxcd.io/force to be re-runnable and must be kept out of stages with wait: true.

Request

spec:
  tiers:
    - name: COLD                     # used as storageClass in Transition rules
      type: s3                       # s3 | rustfs | minio | azure | gcs | ... (TierType in RustFS)
      s3:
        endpoint: https://gateway.example.com
        bucket: tenant-cold
        prefix: hot-tenant/          # recommended: identifies the source deployment
        region: us-1
      credsSecret:
        name: cold-tier-credentials  # accesskey / secretkey, same contract as credsSecret
      deletionPolicy: Retain         # default; see below

Points we think matter for the design

• Credentials cannot be read back. The tier list API does not return secret keys, so credential drift cannot be detected by comparison. Tracking the Secret's resourceVersion (as observed_secret_resource_version already does for users) and calling POST /v3/tier/{name} on change would cover rotation.
• Removing a tier is destructive. Objects already transitioned exist only on the remote. RustFS rejects removal while the tier is in use (ERR_TIER_BACKEND_IN_USE). A Retain default (stop managing, never call DELETE) seems the safe choice, with an explicit opt-in to delete.
• Add rejects a non-empty remote unless force=true. Re-creating a Tenant from scratch, or re-adding a tier after a restore, hits this. The CRD may need an explicit way to adopt an existing remote prefix.
• Ordering. Tiers should be reconciled before bucket lifecycle documents that reference them.
• Status. Per-tier conditions (registered, credentials in sync, last error), similar to the existing provisioning status for buckets and users.