Resource leak: querier BucketScanBlocksFinder never evicts per-tenant metadata fetchers / Prometheus registries for deleted tenants #7572

AI Tool Usage Notice

Drafted with help from Claude Code. All code references, line numbers, failure excerpts, and reproduction steps were reviewed and validated against master before submitting.

Describe the bug

When the querier is configured with the bucket-index disabled (-blocks-storage.bucket-store.bucket-index.enabled=false), block discovery is performed by BucketScanBlocksFinder (pkg/querier/blocks_finder_bucket_scan.go). It lazily creates one block.MetadataFetcher per tenant and caches it in d.fetchers[userID]. Each fetcher is created with its own prometheus.Registry, registered into the aggregating MetadataFetcherMetrics via AddUserRegistry(userID, userReg).

These per-tenant entries are never removed. When a tenant is deleted from object storage (or otherwise stops being active), its userFetcher and its Prometheus sub-registry remain referenced for the lifetime of the process. In a multi-tenant cluster with tenant churn this is an unbounded leak of:

• heap memory — the fetcher's in-memory meta cache, the per-tenant objstore.Bucket client, and the IgnoreDeletionMarkFilter;
• Prometheus memory / cardinality — the per-tenant registry is retained inside MetadataFetcherMetrics.regs (util.UserRegistries) forever;
• local disk — the fetcher's on-disk meta cache under <sync-dir>/<userID>/meta-syncer/.

The store-gateway already handles the equivalent lifecycle correctly (it removes registries and local dirs for tenants no longer owned — see References). The querier's bucket-scan finder simply never performs the equivalent cleanup.

Affected code

• pkg/querier/blocks_finder_bucket_scan.go:67-68 — the fetchers map[string]userFetcher cache.
• pkg/querier/blocks_finder_bucket_scan.go:347-367 — getOrCreateMetaFetcher only ever inserts into d.fetchers.
• pkg/querier/blocks_finder_bucket_scan.go:369-418 — createMetaFetcher allocates a per-tenant prometheus.NewRegistry() (:372) and registers it via d.fetchersMetrics.AddUserRegistry(userID, userReg) (:416).
• pkg/querier/blocks_finder_bucket_scan.go:181-271 — scanBucket scans active users only (:190-191) and replaces userMetas/userMetasLookup/userDeletionMarks to "discard tenants fully deleted from storage" (:254-258), but never touches d.fetchers or d.fetchersMetrics.
• Constructed at pkg/querier/blocks_store_queryable.go:207-220 (the else branch, i.e. bucket-index disabled); CacheDir is BucketStore.SyncDir.

Root cause

scanBucket reconciles the block-metadata maps against the current active-user set on every scan, but the fetchers map and the registries held by fetchersMetrics are write-only — there is no code path that calls delete(d.fetchers, userID) or d.fetchersMetrics.RemoveUserRegistry(userID). ScanUsers already returns (active, deleting, deleted, err) (pkg/util/users/scanner.go:22-27), but the finder discards everything except the active list (userIDs, _, _, err := ... at :191) and never uses it to evict stale fetchers.

Impact

• Querier (and ruler when it queries blocks via the same BlocksStoreQueryable) memory grows proportionally to the total number of tenants ever seen, not the number of currently active tenants.
• cortex_blocks_meta_* contributions from deleted tenants are retained; the underlying util.UserRegistries slice grows unbounded.
• Long-running clusters with high tenant churn (frequent tenant creation/deletion) are most affected. Only affects deployments with bucket-index disabled.

To Reproduce

1. Start Cortex querier with -blocks-storage.bucket-store.bucket-index.enabled=false (so BucketScanBlocksFinder is used).
2. Create tenant A, write blocks, let a bucket scan run → fetcher + registry for A created.
3. Delete tenant A's data from the bucket.
4. Let subsequent scans run. GetBlocks correctly returns no blocks for A, but d.fetchers["A"] and A's Prometheus registry remain. Repeating across many tenants shows unbounded growth.

Unit-level demonstration: in TestBucketScanBlocksFinder_PeriodicScanFindsDeletedUser, after the deleted user's scan len(s.fetchers) stays 1 today; it should be 0.

Expected behavior

After a scan determines a tenant is no longer active, its metadata fetcher, per-tenant Prometheus registry, and on-disk cache directory should be released — bounding resource usage to the set of currently active tenants.

Proposed fix

After each scan in scanBucket (after wg.Wait()), reconcile d.fetchers against the active-user set, under fetchersMx: for every userID in d.fetchers no longer active, call d.fetchersMetrics.RemoveUserRegistry(userID), delete(d.fetchers, userID), and remove the on-disk cache dir filepath.Join(d.cfg.CacheDir, userID). MetadataFetcherMetrics.RemoveUserRegistry already exists (pkg/storegateway/metadata_fetcher_metrics.go:59-61).

Additional context

• pkg/storegateway/bucket_stores.go:529-530 — RemoveUserRegistry(userID) for meta-fetcher + bucket-store metrics when a user store is closed.
• pkg/storegateway/bucket_stores.go:721-760 — deleteLocalFilesForExcludedTenants removes local sync dirs for tenants no longer owned.
• Same class of resource-leak bug as Resource leak: createTSDB does not close TSDB when db.Compact() fails #7557 and [BUG] Distributor: context.WithTimeout timer leak in doBatch — cancel() not deferred, relies solely on ring.DoBatch cleanup callback #7558.

Environment

• Not environment-specific; code is present on master and all currently supported releases.