Repository navigation
Best practices for sharing memory with untrusted parties #607
Description
This was the starting point for #152. That got closed saying "volatile is for MMIO", which is further discussed in #321. However, since then we got "also volatile can access memory outside the AM" (rust-lang/rust#141260) which changes the picture again IMO. The question also recently came up in rust-lang/rust#154321.
I would say the safest (least AM/model-breaking) way to share memory with an untrusted party is to never make that memory part of the AM. This avoids any trouble related to memory models or compiler assumptions. If this isn't actual "memory" in the first place, there can't be any assumptions. And now that volatile accesses are allowed to access such "non-memory", this is actually a viable option. I think this also explains why people are naturally drawn towards using volatile for this kind of interaction. If LLVM changes volatile loads to no longer be willreturn (which @nikic says is on the table), this model even allows reads and writes on memory that might get unmapped at any time. (Alternatively, #606 discusses ways for such segfaults to be valid everywhere, that'd also implicitly cover volatile accesses.)
For a Rust program this means only using raw pointers, never creating a reference, and only using read_volatile and write_volatile. Those don't even have to be atomic; all concurrency behavior is hidden behind the opaqueness of volatile accesses. (Atomic volatile accesses are useful if the party those accesses are communicating with also has access to actual Rust memory. That's not the case for this discussion.) It currently also means never using offset or field projections but we could fix that (#350) -- the LLVM IR we generate is already okay here, we "just" have to update our Rust-level model.
The other model we have discussed in the past is to have that shared memory be actual Rust memory and model whatever the untrusted code does as atomic accesses. This means the Rust code must also use atomic accesses. This seems to work fine in practice but the model actually has a gaping hole -- mixed-size accesses are UB, so the untrusted code can still cause UB in the Rust code. The advantage of this model is that the Rust code is allowed to use shared references (as long as all bytes of the pointee type are inside an UnsafeCell).
Activity
comex commentedon Mar 30, 2026
But note that this UB could theoretically go away in the future. As one possibility, if Intel clarifies their documentation (#345), it might be possible to eventually update memory models so that mixed-size atomics just 'do the right thing'. That would be nice. Alternatively, we could continue to assume mixed-size atomics might tear, but make tearing well-defined, a concept which is desired anyway for atomic per-byte memcpy (as discussed in rust-lang/rfcs#3301).
Another note: the "untrusted code performs atomic accesses" model raises the question of whether those accesses are allowed to store uninit bytes, and if so, what programmers are supposed to do to guard against it. This is briefly discussed here.
RalfJung commentedon Mar 30, 2026
Please let's not derail this issue with mixed-size atomics. That's already being discussed in #345, I replied there. The summary is that mixed-size accesses are not happening near-term and atomic bytewise memcpy does not make progress towards mixed-size accesses.
Given that the untrusted code is modeled as asm code, I think it is fair to say that it cannot store uninit bytes, as that concept does not exist in asm.
Darksonn commentedon Mar 30, 2026
In that case, we should reconsider rust-lang/rust#58041 (comment)
jethrogb commentedon Mar 30, 2026
In the “atomic” model, should it be allowed to create
UnsafePinnedmutable references to such memory?comex commentedon Mar 30, 2026
I do not think I was derailing the issue. I was merely summarizing the long-term prospects of mixed-size atomics, which are relevant to the question in this thread of which of the two models (volatile or atomic) should be used when sharing memory with untrusted parties.
On another note,
In my experience, it's pretty common for shared-memory communication protocols to rely on atomic operations like compare-and-swap, swap, etc. For example, WebKit and Chromium both use atomic swaps in their IPC implementations.
Other shared-memory protocols use only loads and stores, but still rely on release/acquire ordering (to establish ordering between loads/stores to different parts of the shared memory buffer). These include Linux's io_uring and xnu's IOSharedDataQueue.
Whatever approach we recommend for sharing memory with untrusted parties, it needs to support these operations somehow.
RalfJung commentedon Mar 30, 2026
RalfJung commentedon Mar 30, 2026
bjorn3 commentedon Mar 31, 2026
The compiler isn't inserting any fence instructions, right? So they would only be totally ordered when operating on strongly ordered memory like MMIO ranges mapped as uncached.
RalfJung commentedon Mar 31, 2026
Oh right I forgot about hardware-level weak memory. 🙈
pitust commentedon May 11, 2026
For shared memory, it would be nice to have a type
Untrusted<T>with special semantics like this:u: &Untrusted<Foo>andFoohas a fieldfieldof typeBar. Then, writing&u.fieldgives a value of type&Untrusted<Bar>(which I think is similar to user-defined places?)unordered? I think? orreadand thenfreeze, more likely).readand.writemethods on values on theUntrusted<T>struct, which may be easier to implement.Untrusted<T>only so that you can know it's safe to return a value of this type from a mmap call in a library.You can also use this to implement seqlocks (I think?), which is nice, and it allows soundly exposing shared memory, which is also a big plus.
ds84182 commentedon Jul 10, 2026
I very much dislike volatile loads.
Assuming there is no explicit synchronization with the untrusted party, I would like a load operation that can only be done once. It can be elided or reordered if needed (unlike volatile), but never duplicated. In a compiler sense it cannot be "rematerialized" from that memory location later.
Now, this may be quite radical.
It is UB to synchronize with the untrusted party because loads before and after synchronization are allowed to be reordered. Changes expected from synchronization may not be reflected by a load after the synchronization point, or may be reflected by a load before the synchronization point. It is completely unordered and no memory barrier can help this. This includes transferring control flow to untrusted party in the same "thread of execution", since fundamentally they are still unsynchronized wrt these "volatile-esque" loads.
This comes with the consequence that, despite possibly running on the same physical CPU, stack, process, thread, etc., the untrusted party is treated as a different thread of execution. Between a context switch or simple function call, it is functionally a different thread of execution.
An additional constraint: within the current thread of execution, a store to a memory location that has been or will be loaded with a special load results in UB. The ordering between the load and store is undefined. You must insert a compiler fence to separate the special loads from subsequent stores. Put another way, these special loads cannot even alias with a store in the same thread of execution until a compiler fence is encountered.
A compiler fence within another function does not count. If you are calling another function that does a store then you must use a compiler fence before that call. Placing the fence within the callee may "work" when inlined but "here be dragons. FAFO".
As a side effect it is not UB to synchronize with the untrusted party if all special loads happen before a compiler fence preceding the synchronization.
This may sound a lot like non-temporal loads and stores. And yeah it is, sorta kinda.
I don't have a name for this. "weak load" may be a good candidate, I'm not sure. I don't think LLVM supports something like this, either. The closest you can possibly get is inline assembly with the nomem option.
Feel free to tear this idea to bits. I just hope it'll be a good starting point for an eventual solution. I just dislike volatile loads, they annoy me.
ds84182 commentedon Jul 10, 2026
I think I can actually simplify this a bit:
On
weak_load, the memory address is tainted such that the only allowed operations areweak_loaduntil the next compiler fence. This tainting happens between compiler fences and is non-temporal. Tainting occurs for the entire section of code if a weak load will be reachable: it is not extralogical. If the weak load is reachable it taints the address once the data dependencies of the weak load's address have been computed. The address is computed at the earliest possible time, which may be before the preceding compiler fence; therefore an address that is computable before the fence is immediately tainted after a fence if a weak load will be reachable within the code before the next fence.A single weak load of an address happens at most once. Multiple weak loads of the same address may be coalesced, since it still happens once, but is not guaranteed. This avoids data races because it is always loaded at most once, so a race cannot be observed because observing a race requires a second access. If you have two equal addresses at runtime, but the equality is not known at compile time, their weak loads are not combinable, therefore they may have separate values. So there is also no guarantee that
weak_load(x) == weak_load(x)because there is no guaranteeweak_load(a) == weak_load(b) even if a == b, but it may just happen to, for a variety of reasons.The data dependency and time travel behavior is extremely horrible, but allows for
weak_load(weak_load(x))without tainting the entire address space(!!),weak_load(x.read())if x points to itself, and also explains how load reordering works without specifying when it may be reordered.Also it might be best to make a separate type of fence for this (a "weak fence") because compiler fence has other effects.
Functionally what this is intended to do is eliminate TOCTOU issues when reading memory controlled by untrusted parties (as the load happens once) while also making no guarantees about, if it happens, when it happens, until the point where it cannot happen (which is after a fence; explicit synchronization with the untrusted party does NOT work).
RalfJung commentedon Jul 10, 2026
I have no idea what you are talking about, but "inline assembly with the nomem option" is definitely UB.
There is no logical connection between the two sides of the "because" here that I can see. I'm afraid you are not making sense. It is frequently necessary to synchronize with the untrusted party to ensure correct behavior for the case that the untrusted party behaves correctly.
robofinch commentedon Sep 23, 2026
How about modeling an untrusted party as being outside the AM and using volatile operations to talk to a background thread, with that background thread executing atomic accesses on shared memory on that untrusted party’s behalf?
As far as I can tell, so long as everything works out at the hardware level, the background thread could even angelically avoid mixed-size atomic accesses (i.e. pretend that any observable stores were due to atomic appropriate accesses). This story requires that the trusted code ensures that the trusted threads and background threads could agree on which locations of memory are accessed with which sizes, so no
load_relaxed_u8(ptr); load_relaxed_u16(ptr);butload_relaxed_u8(ptr); fetch_add_acqrel_u8(fake_place_for_coordinating_accesses); load_relaxed_u16(ptr);should be fine. Same deal as angelic choice elsewhere, there needs to be at least one valid choice that could be angelically made.Are hardware-level concerns about mixed-size atomics the only remaining issue with using atomics on memory shared with an untrusted party (i.e. has anything changed since the PR description was written)? I’ll ask on the mixed-size atomics issue about those concerns.
RalfJung commentedon Sep 23, 2026
We can stop right here I am afraid. Story code can only do what Rust code can do and Rust doesn't have a general angelic-choice primitive. If it did, that would be a problem since angelic choice breaks fundamental reordering assumptions that LLVM relies on.