Performance: Major feature: Adaptive hybrid heap/SHM-backed serializer. #160

Background: Once release 3 goes out -- including major perf-improvements (#143) and a major feature (#121) -- we will have in Flow-IPC 2 directly supported ways of exchanging capnp msgs over IPC:

• IP1 (Integration Point 1): Through our own struc::[sync_io::]Channel, in struc::Msg_*. It just has request/response, msg type mux/demux, and heap-backed + SHM-backed support.
• IP2: Through capnp-RPC. One can use vanilla capnp-RPC, or our version of it but with zero-copy knobs set to fallback mode; in this case the capnp messages -- both user-created ones and capnp-RPC-internal (control) ones -- are heap-backed. Or one can use our zero-copy capnp-RPC integration (with Flow-IPC), centered around ipc::transport::struc::shm::rpc::Session_vat_network; in this case the messages become SHM-backed (zero-copy). In either case, one gets all the capnp-RPC goodies: promise pipelining, -> stream.

Each has its uses -- discussing mutual pros/cons is outside our scope here; but each is useful. And: both have, in basically similar ways, the following perf problem.

Problem: for a given message, consider transmitting it as fully heap-backed versus fully SHM-backed.

• If it's tiny: heap-backed involves copying only a few bytes. SHM-backed avoids copying that, but there is book-keeping/computation on top of it. SHM-backed, ultimately, will be comparable to heap-backed... but it'll be overall slower.
• If it's small (say, 1Ki, maybe 8Ki even): less so but still so.
• If it's large (say, megabytes): SHM-backed wins. Not having to come megabytes into IPC then out of IPC -- not to mention whatever reassembly of large-leaf-containing capnp segment might be required (more copying, or more complexity to avoid it) -- is a big savings.
• In-between: who knows? It depends.

So, basically, heap-backed is better up to a point; then SHM-backed is better. Currently though, for both IP1 and IP2, one has to decide to use either heap-backed or SHM-backed regardless.

• Chose heap-backed... but some messages are big: Processor use spikes? Perf corner cases muddying the picture?
• Chose SHM-backed... but most messages are small: Paying an (acceptable but real) perf tax for the mainstream so as to not get messed-over by the big outliers.

It is workable, but it's not ideal. E.g., for the #121 feature, we can advertise capnp-RPC integration (IP2 here) as a zero-copy "backstop": for small messages perf is comparable to vanilla capnp-RPC; but you need not worry about big ones messing you up. Valuable... but not as valuable as being able to say, "it's faster; uses zero-copy when it helps. Do not worry."

Solution:

In broad strokes: like it says, make a [de]serializer -- perhaps one replacing (~backwards-compatibly extending) the SHM-serializer (core in transport::struc::shm) -- that uses heap-backing where it's best; then SHM-backing where it's best.

The general algorithm -- whether heap- or SHM-backed -- today, as seen in capnp's own MallocMessageBuilder (and our derivatives), is basically: seg 0: reserve SEG0 (user can estimate SEG0 so as to include, they think, the entire eventual message... but in general it's a guess like 8Ki). If it runs out: basically reserve 2xSEG0. If that runs out: reserve 4xSEG0, 8x, 16x, etc. (This is not accurate-accurate. It's "basically" about right, but it is modified by the case of large-leaf-bigger-than-preferred-next-seg-size and such. Not important for us here. For illustration purposes a 2x-exponential growth = OK representation.)

(Side note: If SHM-backed, 2x might be insufficient ambitious. Maybe try another factor, trading more aggressive RAM use for less allocating. Allocating is expensive. <-- Could apply this without yet doing the hybrid serializer.)

So then the proposal here is: Divide seg-sizes into:

• Cat1: Sized <K, where K is some constant or knob; like, say, 32Ki or 8Ki. Cat 1 => heap-backing is best.
• Cat2: Sized >=K. Cat 2 => SHM-backing is best.

Then for each message: we start in cat 1 (or maybe even immediately cat 2 but read on); and since seg sizes do not go down, first 0+ segs will be cat1, last 0+ segs will be cat2. It is quite clean. Side note: undecided but probably this is fine: if a segment (while still in cat1) has to be sized > IPC capacity (which can only occur if in the first place there's a too-big capnp leaf -- List, Data, Text), then staying in cat1 would mean the split/reassemble (with all the extra copying) is required. So perhaps this situation (too-big capnp leaf) should automatically go to cat2 from that segment on. (Main thing is, to keep algo + data simple, it should be cat1-if-any=>cat2-if-any, no interleaving. If possible. If interleaving ends upbeing desirable, then we can modify the mdt below accordingly.)

A possible representation in the outer serialization is as follows. (Today: heap-backed: there is only outer serialization; SHM-backed: outer ser = 1 tiny segment with single SHM-handle to essentially list<vector> in SHM; inner ser = actual user structure in SHM, in segs listed in that list.)

• Blob1: capnp struct with tiny SHM-handle again; same meaning. Describes in-SHM (cat2) segments if any. (Null or whatever if no cat2 segs needed.)
• Blobs 2+ (if any): cat1 segs in order.

Inner serialization: Exactly like today for SHM-backed. If no cat2 segs then none.

Impl: IP1: Code details TBD, but the basic idea follows established patterns. Mdt header remains. For this serializer, what is to struc::Msg_* machinery seen as "outer serialization seg1 fragment1" continues to be seen that way; no changes needed. But inside the serializer layer -- so struc::shm::Builder and/or struc::shm::Capnp_msg_builder (the MessageBuilder impl) (and the rcv-side equivalents) -- this blob -- blob1-past-mdt-header -- is actually:

• self-describing (includes size) tiny capnp header (yes, another one; yes, including its own size per same technique as ipc_transport_structured: Perf: Include self-recorded length for the mdt header in struc::Msg_* encoding (no more reserved-128 bytes over IPC). ipc_transport_structured#36) which contains also (like today) the SHM-handle to the list describing cat 2, or null if none. Then after this guaranteed-tiny capnp msg, no gap:
• seg1 = rest of blob1. (It was stipulated above that with this hybrid serializer there will be no split/reassembly of heap-backed segments necessary. Hence it's no longer "seg1 fragment1" but just "seg1." (And the remaining blobs = heap-backed segs, no splits.)

After that, 0+ more blobs = remaining cat1 segs. shm::Reader side just has to understand all this; and viola [sic].

Impl: IP2: The capnp-RPC integration is the most open-ended part, but even there we've carried quite a decent look at TwoPartyVatNetwork and co. + our own version thereof (the zero-copy-enabled Session_vat_network): enough to pretty much know what to do. (TODO for self: I have some more deep-dive notes, especially about IP2-impl, which should be linked or attached here.) The main thing:

A vanilla TwoPartyVatNetwork (TPVN: the engine that implements non-zero-copy transmission of capnp-messages for RPC) has a layer wherein sender sends a capnp-msg -- its 1+ segs -- and receiver receives them. They're encoded in the classic capnp way, a-la writeMessageToFd(): seg count N, N individual seg sizes, N segs in order. While creating payloads they're treated as capnp, and after receipt again. But in-between these MessageBuilders and MessageReaders, when mentioned, are treated only as their representations as 1+ blobs. I say blobs because while "in reality" they're capnp segments (in vanilla TPVN), this is irrelevant between capnp-mutation and capnp-reading.

Our data structure is the same! At least conceptually it is the same. At its simplest it would just be:

• Blob1: capnp struct with tiny SHM-handle.
  • Add the Bool for short-lived message (same as today).
• Blobs 2+ (if any): cat1 segs in order.

Blob1 is identical to what we have today in our Session_vat_network... but there are no blobs 2+ today.

Impl-wise (in capnp-RPC land now, so Session_vat_network): There's no way of leaving S_v_n unchanged; the current code cleverly leaves as much as possible to existing capnp-RPC TPVN parts; all that basically changes is:

• The as-capnp getBody() accessor works off the in-SHM representation a/k/a inner serialization, not the outer serialization.
• The capnp-payload of the actual stuff going through the IPC-transport = always 1 segment, always the outer serialization (the SHM-handle + short-lived-msg Boolean).

Now, though, there are N segments (no problem), but in total they don't encode any capnp necessarily: blob1 by itself does and blobs2+ together do, but in total: nope. So, we'll just have to reimplement something like 100-200 lines of core code from TPVN this time. Hand-wavily:

• Rcv-side: Should remain unchanged except Rpc_msg_in_impl's ctor (to account for the TBD shape of the hybrid-capable struc::shm::Reader and/or Capnp_msg_reader, w/r/t how to load the incoming blobs into it... some modification of ::borrow() API?). So almost no change actually.
• Snd-side: Because OutgoingRpcMessageImpl::send() works with a MallocMessageBuilder -- fundamentally not what we have -- without splitting the transmission from the mutation/serialization, we can no longer use it; so we'll write our own. It's 100-200 lines, and we'll need to know what it does and why and proceed accordingly. But it's still a matter of sending 1+ blobs, with message boundaries respects and describes as "segments."

Priority/Order: Do it after Flow-IPC/ipc_transport_structured#36 (which increases its priority). Importance: High. I explained why near the top; but informally I'd add that -- if I had time at least -- I would want to put out a release 3.1 centered on this. Its existence makes the perf story, just, nice. No -- or at least no major -- caveats would remain in this area. It would raise the value of capnp-RPC and of struc::Channel too.

Wildcard: What is K? Where should the cut-off happen? Answer: Don't know. Good question. If we can get some decent benchmarking about it, we can provide a default with some basis. Regardless it probably needs to be configurable (at least through Builder::Config -- readily available -- but perhaps at higher layers through Channel, Session_vat_network, maybe ipc::session too... TBD).