[server] KV snapshot scheduling stops permanently after a transient snapshot-init failure #4489

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

main (development)

Please describe the bug

I searched in the issues and found nothing similar. Closest family:

• [Server] PeriodicSnapshotManager.start() silently skips scheduling when initialDelay is 0 #2977
• [Server] Fix PeriodicSnapshotManager.start() silently disables snapshot #2978, A different start-time trigger, already fixed.

Motivation

A transient backend outage during snapshot initialization permanently disables KV snapshots for the affected buckets until the TabletServer restarts. The failure is silent after a single ERROR line: no retry, no metric alarm, no recovery when the backend returns.

Reproduction

• Fluss 1.0.0 (also present on main, file is identical), 1 coordinator + 2 tablet servers, kv.snapshot.interval: 60s, S3-compatible remote storage.
• Stop the object store for ~6 minutes, then restore it.
• During the outage, tablets log Connection refused with AWS SDK retries; writes keep flowing (local hot path, unaffected).
• After the backend returns: no new snapshot for 14+ minutes (~14 missed 60s cycles). Only snap-1/snap-2 remain (retention keeps the last two).
• Restarting one tablet produces a fresh snapshot ~50s later (snap-4), confirming the scheduler — not the backend path — was dead.

Root cause

fluss-server/.../kv/snapshot/PeriodicSnapshotManager.java, triggerSnapshot() (v1.0.0 and main identical):

try {
    snapshotRunnableOptional = target.initSnapshot();
} catch (Exception e) {
    LOG.error("Fail to init snapshot during triggering snapshot.", e);
    return;   // <-- scheduleNextSnapshot() never called
}

Scheduling is a one-shot chain (schedule(triggerSnapshot, delay) + explicit scheduleNextSnapshot()), so returning here without rescheduling breaks the chain permanently for that bucket. The async-phase failure path (whenComplete) does track retries; only the init path drops the chain.

Observed log line proving the trigger:
ERROR org.apache.fluss.server.kv.snapshot.PeriodicSnapshotManager [] - Fail to init snapshot during triggering snapshot.

followed by total silence (no further attempts, successes, or failures).

Solution

Call scheduleNextSnapshot() on the init-failure path too (ideally with backoff after consecutive failures, mirroring the existing numberOfConsecutiveFailures tracking in the async path), so a transient backend outage degrades to missed snapshots instead of a permanently dead scheduler.

Are you willing to submit a PR?

• I'm willing to submit a PR!