Remove fsync of rootfs-upper and durable-dir tars

Today, we always fsync tars e.g. for durable-dirs or microVM's rootfs upper.

Lines 130 to 135 in 5b7417e

    
           // Durable-dir tars are handed to atelet for upload as soon as we return, so 
        
           // flush to disk rather than trusting the page cache to outlive us. 
        
           if err := f.Sync(); err != nil { 
        
           	return fmt.Errorf("syncing tar %q: %w", tarPath, err) 
        
           } 
        
           return nil

Similarly to #1883, I think we can remove the fsync here, unless the local snapshots must survive a node crash.

Michelle Au (@msau42)

cc Michelle Au (@msau42) Benjamin Elder (@BenTheElder) Walter Fender (@cheftako) WDYT?

Dmitry Berkovich (@dberkov)

Local snapshots should survive a node crash? cc Dmitry Berkovich (@dberkov)

We may offer another mode that is not durable to node crash.

Local snapshots should survive a node crash?

Yes, based on the lifecycle V2, local snapshots needs to survive the node restart.

I will come back to it, after we remove SnapshotScope_SNAPSHOT_SCOPE_DATA mode.

Huy Pham (@hdp617)

Hi Huy Pham (@hdp617) ,Benjamin Elder (@BenTheElder), and Dmitry Berkovich (@dberkov), I've been running node-density benchmarks (sweperf, one bare-metal c3-highcpu-192-metal node, one worker) and wanted to share some data on this one in case it's useful.

Setup: main@5035a6fa, microVM, sweperf-astropy-7336, one c3-highcpu-192-metal node, one worker, async suspend, upload budget 64Gi. I compared the stock build with the same build minus the f.Sync() in tarutil.create (one run each).

At the knee (60 users), node sampling showed the worker blocked in this fsync for about 95% of its D-state samples (sync and async both).

users	SuspendActor p50 (stock → no fsync)	ResumeActor p95	resumes/s
20	3.4 s → 0.64 s	0.99 → 0.97 s	1.36 → 1.10
40	10.6 s → 0.81 s	1.31 → 1.07 s	1.46 → 1.79
60	≥32.8 s → 0.90 s	1.69 → 1.15 s	0.96 → 1.68
80	≥32.8 s → 2.4 s	24.9 → ≥32.8 s	1.53 → 1.31

(≥32.8 s is the histogram cap.)

Large suspend-latency improvement below saturation.
No change in where it saturates (60–80 users either way), and node I/O PSI stays around 50–70%.
At 120 users the no-fsync run had 2 restore failures (virtiofsd socket … did not appear within 10s), and the stock runs had none. This is a single run, so it may be noise.

Just wanted to share the measured cost in case it helps with the decision. Happy to rerun anything or share the raw data.

Michelle Au (@msau42)

With lifecycle v2, when a user suspends an actor (ateapi.proto#L1751), they can specify:

DURABILITY_MODE_SYNCHRONOUS: The operation returns only after the snapshot is uploaded to durable storage.
DURABILITY_MODE_ASYNCHRONOUS: The operation returns once the snapshot is stored locally, and it will be uploaded to durable storage in the background.

Users have full control over this trade-off.

When we talk about whether snapshots can "survive a node crash", there are two failure scenarios to distinguish:

Node termination ("node gone"): With or without fsync, it does not matter - the snapshot is stored on the local boot disk and will be lost regardless.
Node restart / reboot: The boot disk persists, meaning resuming on the same node VM would theoretically be possible. fsync is potentially relevant here to protect against inconsistency across an unexpected restart. However, we do not currently guarantee that local-only snapshots survive node restarts, so technically we can remove the fsync here as well.

Michelle Au (@msau42) - WDYT, can we drop the fsync?

Yes I think if we don't guarantee node-restart case, then we can drop fsync (in fact, I think we are missing fsync in local snapshot case today).

#2658 makes it so that we fall back to object storage if the local checkpoint is not found or is missing files.

The other consideration with fsync is that since we currently have all the actors sharing the same filesystem, fsync on one actor's directory can get blocked behind another actor's writes because they get flushed in the order of the journal. (This is another reason why I want to explore per-actor filesystem)

Michelle Au (@msau42)

Thanks! I can put up a PR removing the f.Sync() in tarutil. From what i know the journal-ordering point matches what I saw where the worker was mostly waiting in jbd2_log_wait_commit, behind other actors' writes.

Michelle Au (@msau42) With regards to #2658 if a tar is present but truncated so it crashed before writeback, would restore catch that and fall back, or fail? just want to make sure I'm not missing a case.

It only checks for exisence today:

substrate/cmd/atelet/main.go

Line 1803 in 161aa67

func checkLocalSnapshotFiles(dir string, files []string) error {

, but we can enhance it with more checks.

Thanks! Makes sense. One option could be recording file sizes in the manifest and comparing them in checkLocalSnapshotFiles, which would catch empty or truncated tars after a crash without reading the files. Can take that as a follow-up if useful. I've put up the fsync removal PR separately.

However, we do not currently guarantee that local-only snapshots survive node restarts, so technically we can remove the fsync here as well.

I'm not sure about this one, the performance gain is mighty tempting though. Maybe we should give an option based on sync / async mode even for local?

Re: manifest: I really want us to get rid of that special object and move this data into the API, it's a pointless roundtrip to object storage before we can fetch anything else. Let's not double down on that.

What we decided to do is to invalidate all local snapshots on a reboot (#3050), which we don't support in substrate today (all running actors crash). On drain, actors are sent a SIGTERM and need to figure out how to SYNC suspend.

Scott Funkenhauser (@sfunkenhauser)

FYI Scott Funkenhauser (@sfunkenhauser) (touches on some related areas to crash handling for workers)

tarutil: drop fsync of rootfs-upper and durable-dir tars (agent-subst…

ateworker-microvm: keep an EROFS container's rootfs upper on a virtio…

See #3092 ... FYI ... not fsyncing will mean the pages stay dirty which may be in opposition to problem 2) depending on how tight worker memory is ... we may have to essentially undo that gain to avoid worker crashes.

Navigation Menu

Repository navigation

Remove fsync of rootfs-upper and durable-dir tars #1954

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	// Durable-dir tars are handed to atelet for upload as soon as we return, so
	// flush to disk rather than trusting the page cache to outlive us.
	if err := f.Sync(); err != nil {
	return fmt.Errorf("syncing tar %q: %w", tarPath, err)
	}
	return nil

Remove fsync of rootfs-upper and durable-dir tars #1954

Description

Activity

hdp617 commented on Sep 28, 2026

BenTheElder commented on Sep 28, 2026

dberkov commented on Oct 2, 2026

pmandewalkar commented on Oct 19, 2026

dberkov commented on Oct 20, 2026

msau42 commented on Oct 20, 2026

pmandewalkar commented on Oct 20, 2026

msau42 commented on Oct 20, 2026

pmandewalkar commented on Oct 20, 2026

BenTheElder commented on Oct 21, 2026

msau42 commented on Oct 21, 2026

BenTheElder commented on Oct 21, 2026

BenTheElder commented on Oct 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions