id: research/design: fork a Rust KDL stack for ID — universal + multiple annotations, upstream archaeology, fork-base selection (extends #5688) #5753

Activity

1. dezren39

   added

   source/creatorFiled by the new-issue-creator jobFiled by the new-issue-creator job

   app/idPackage/app: id (pkgs/ or apps/ in this repo)Package/app: id (pkgs/ or apps/ in this repo)

   on Nov 5, 2026
2. developing-today commented on Nov 5, 2026

   

   developing-todayBot

   on Nov 5, 2026 – with developing.today

   Contributor

   More actions

   CODE-1441
3. github-actions

   assigned

   dezren39

   on Nov 5, 2026
4. dezren39 commented on Nov 5, 2026

   

   dezren39

   on Nov 5, 2026

   MemberAuthor

   More actions

   WS-I follow-on investigation (2026-11-04): "KDL 2 compact", hygge + parser landscape re-survey, fast serde alternates, and the fast-query form

   Four threads from Drew's 2026-11-04 20:06 CST additions. Frame remains §4's commitments: KDL 1 compat absolute, universal + multiple annotations, performance and search as first-class goals. Cross-refs: #5688 (parent design — triples + SQL/PGQ), #5674 (postcard wire), #5675 (querying ID servers), #5681 (storage tiers).

   ---

   0. "KDL 2 compact" — pinned down as a named requirement

   The reading, stated as a requirement so it can be confirmed or corrected: the fork's baseline is KDL 2.0, and it defines a compact profile: a canonical minimal-whitespace emission of a document, used for stored / interchange copies (the search corpus, the provenance payload, content hashing), while the human form stays the pretty autoformat. If Drew meant something narrower (e.g. only "emit v2, never v1"), the profile below still stands on its own evidence; the open question is recorded at the end of this section.

   What the spec allows. Nothing in KDL 2.0 defines a canonical or minimal form — formatting is implementation territory. But the 2.0 grammar is unusually minification-friendly, more so than 1.0:

   • Nodes may be separated by ; instead of newlines, and the last node in a children block needs no terminator even with } on the same line (node{foo;bar;baz} is legal 2.0).
   • Node-space around = in props is permitted but never required; zero is the minimum.
   • Whitespace inside (annotations) is legal in 2.0 — minification must therefore choose (foo), never be forced into it.
   • Bare identifiers cover more ground in 2.0, so a compact emitter can drop quotes more often than a v1 emitter could (subject to the identifier rules).

   What upstream's formatter can do — checked against the kdl 6.7.1 source (src/fmt.rs). FormatConfig has exactly four knobs: indent_level, indent (default four spaces), no_comments, entry_autoformate_keep. Every one serves pretty-printing. There is no compact/minified mode, and none can be configured into existence (autoformat always emits line-broken, indented documents). Conclusion: compact emission is a small new emitter mode in the fork — a serializer walk that joins nodes with ;, emits one space only where the grammar requires separation, strips comments, and picks bare-vs-quoted identifiers minimally. It is not a parser change and does not touch §4a compatibility.

   Measured size deltas on our real documents (script minification of the actual files; rules: comments stripped, node-internal whitespace collapsed to single spaces, ; separators, no spaces around = or braces, strings preserved verbatim — reproducible in the follow-up pass):

   Document	As authored	Compact	Δ
   pkgs/mappy/migrations/limit.kdl (3,491 B on disk)	3,450 B	2,296 B	−33.4%
   pkgs/identity/config.kdl + configuration/embed/config.kdl	633 B	497 B	−21.5%

   Caveats: most of the win is comment + indentation removal, i.e. compact is a storage form, not an authoring form — the comments in limit.kdl are design content and must survive in the authored copy. Multi-line string payloads cannot shrink below their content. (limit.kdl's torture-test tail also contains constructs outside the v2 grammar; the figure measures byte volume, not a round-trip promise for that file.)

   Why compact earns its place beyond bytes: canonical byte-stable text makes a document content-hashable and dedupable (composes with the treed-hash identity idea in pkgs/id/README.original.md), and a search ingest that stores compact text scans ~⅓ fewer bytes before the parse even starts (thread 3).

   Open question for Drew (the uncertainty, stated plainly): is "compact" (a) this storage/interchange profile, (b) a v2-only emission policy ("KDL 2, compactly" as in the short version of the plan), or (c) both? Filed as (a)+(v2 baseline) pending correction.

   ---

   1. hygge, read in full — and the parser landscape since

   hygge (developing-today/hygge, created 2022-02, last push 2023-12; v0.4.0; depends on kdl = "3.0.0"). Read end-to-end this pass: src/lib.rs is 780 lines.

   • What it actually is: fluent builder wrappers over the kdl v3 AST — KdlNodeBuilder (name/values/props/children), KdlValueBuilder (terse constructors: str/int/flt/y/n/nil), KdlValuesBuilder, KdlPropertiesBuilder — with build() producing real KdlNodes. It contains no parser (by design; the README defers input to knuffel/kdl-rs) and no serializer of its own — output is upstream's Display.
   • The one live idea: HashSetMap — a hash-consing register that deduplicates values by content hash into Rc<T>. As code it's a sketch; as a concept it's the direct ancestor of two things the fork wants anyway: string/value interning in the AST (§4c memory posture) and ID's content-hash identity thinking.
   • What's obsolete: the &mut String plumbing (the source's own trailing comments list the ergonomics as unresolved — "unify str vs String", "&mut self vs Self"); the v1-shaped KdlValue model it wraps (the kdl crate is three major versions past it, with a different numeric model); and it never touches annotations at all.
   • Verdict: mine hygge for two patterns — the builder-API shape for §4d's renderer/output side, and hash-consing as prior art for interning — and port no code. Nothing in it competes with, or could accelerate, the parser fork.

   The landscape since #5753's survey (verified this pass):

   Candidate	State now	Annotation posture	Verdict vs the fork's needs
   kdl (kdl-rs) 6.7.1	Actively developed. v6 (Dec 2024, alongside KDL 2.0.0) is a rewritten v2 parser (multi-error collection, span-carrying AST); v1 is delegated to the sibling kdlv1 crate (the v4.7 lineage); v1⇄v2 conversion both ways; serde (src/de.rs/se.rs), KQL (src/query.rs), and the FormatConfig pretty formatter all in-tree. Current docs put MSRV at Rust 1.95 — noted, not blocking for us.	Unchanged from #5753 §3: ty: Option<…> on nodes/entries, bare prop keys, no children slot. The surgery is still exactly where §3 located it.	Base — confirmed (below).
   knuffel	Repo declares itself unmaintained.	—	Dead; superseded by knus.
   knus (knuffel's maintained fork)	3.4.x, releases into 2026 — but the implementations table marks it KDL v1 only, and its maintainer describes a review-and-merge posture and names facet-kdl as the intended successor. Best-in-class derive UX + miette errors for typed decode.	Consumed as types during decode; no document fidelity.	Pattern donor (derive ergonomics, error quality for #5688's non-developer authors), not a base: v1-only grammar fails §4a's dual-version bar on its own.
   kaydle	Unchanged: 0.2.0, self-described early alpha (2022), a serde skin over the old kdl crate.	Inherits its era's AST.	Dead end, re-confirmed. The niche it filled is now kdl-rs's own built-in serde.
   facet-kdl	New since #5753 was framed. The facet project's KDL format crate: parsing by kdl 6.x, typed decode by facet reflection. In real use (e.g. bearcove/tracey at facet-kdl 0.35 over kdl 6.5). The knus maintainer independently points at it as knus's successor.	Whatever the kdl AST exposes — i.e. it inherits the fork's annotation work for free once landed there.	Not a base — a consumer of the base, and the strongest external validation of the fork's shape (see below).
   kdly	Go; format/comment-preserving; v2-only.	—	No Rust relevance; noted as cross-language evidence that document-API designs converged on the kdl-rs shape.

   Fork-base verdict: CONFIRMED, and strengthened. Every living Rust KDL effort now either is kdl-rs or sits on top of it (facet-kdl, kdlfmt, historically kaydle). The alternatives are v1-only (knus), maintenance-mode (knus), or abandoned (knuffel, kaydle). The annotation generalization remains a contained AST change in exactly one place — and it is the place the rest of the ecosystem already depends on, so the fork's improvements propagate to facet-kdl/knus-style consumers rather than stranding us on a private island.

   ---

   2. Serde, but faster — the alternate-path shortlist

   Baseline stands: the kdl crate's built-in serde (default feature) is the fork's serde story. What follows is what can sit beside it, feature-flagged, per Drew's "support serde but alternate performance versions maybe."

   Candidate	Maturity / evidence	Perf & RAM posture	Integration cost with a kdl-rs fork	Slot
   Borrowed / zero-copy decode (serde 'de lifetimes, Cow, borrowed scalar views)	Standard serde practice; no new dependency.	Eliminates per-string allocation on string-heavy documents — the common KDL case. The bigger half lives one layer down: a no-format-preservation parse profile (already §4c) so the AST itself stops paying for repr/span fidelity on the ingest hot path.	Low: expose borrowed views alongside owned AST accessors.	Shortlist — do first.
   Arena / bump AST allocation (bumpalo-style)	Mature pattern in parser engineering.	One arena per document: fewer allocator round-trips, better cache locality for parse-once/query-many ingest. Fights the mutable, format-preserving AST (in-place format edits + arena lifetimes don't mix).	Medium: a second AST storage mode behind the ingest profile.	Shortlist — scoped to the no-preserve ingest profile only.
   facet (reflection framework; SHAPE const per type; format crates per format)	Young (started Mar 2025), moving fast, pre-1.0 churn; facet-kdl exists and is used in production by its author's tools.	Not a speed play today, by its author's own measurements: facet-json started 5–7× slower than serde_json; later dashboard results are publicly flagged by the author himself as not yet trustworthy ("zero-copy apples-to-oranges" concern); compile times went from worse to roughly even after sustained work; facet-postcard is reported faster than the reference postcard implementation (author-reported).	Unusually low for us: facet-kdl consumes the kdl-rs parser layer, so a fork that keeps that layer's shape gets facet support nearly free.	Shortlist — as an optional decode layer (one derive, many formats + schema generation), explicitly NOT as the performance path.
   postcard (binary form)	Mature; already ID's wire choice (#5674, #5688 §1.3).	Not a KDL path at all — it is the stored/binary form candidate. Third-party micro-benchmark (single small struct, indicative only): postcard ≈ 41 B on the wire vs ≈ 99–100 B for JSON text, ~1.5× bincode's runtime at ~70% of its size.	Zero conflict: postcard is serde-based, so it composes with the baseline rather than competing.	Adopt in its lane: serialized AST / triple-row payloads where a binary stored form is wanted.
   miniserde	Self-described prototype — its own docs say it is "not a production quality engineering artifact"; JSON-centric.	By its own account: on par with serde_json in some cases, 1.5–2× slower in most. Its real win is compile time / code size (no monomorphization), not runtime.	Would be a second, weaker data-model to maintain.	Reject as a performance alternate; the honest summary is that it optimizes a different axis (build weight) than Drew's ask (parse/serialize speed + RAM).
   SIMD-accelerated scanning (simdjson-style structural indexing)	Proven technique for JSON; no KDL implementation exists anywhere (this pass found none).	KDL scanning is delimiter/whitespace-bound; the cheap 80% is memchr-class delimiter search plus thread 0's compact profile (fewer bytes to scan). Full structural-index SIMD is a research project, not a feature flag.	High, speculative.	Defer, benchmark-gated: revisit only if the §5 benchmark pass shows scanning dominating profiled parse time.

   Net: the credible "alternate performance version" is a parse-profile feature set (no-preserve + borrowed views + arena) inside the fork, with facet as an optional ergonomics layer and postcard as the binary form. There is no drop-in faster serde for KDL, and miniserde is not one either.

   ---

   3. The fast-query form — what querying should run against

   Framing note: #5688 §1.3 already rules that KDL text is not the query substrate. This thread answers the remaining question — what the substrate is, per tier — against §4c's "designed against parsed-structure indexes rather than re-parse-per-query."

   (a) Rows at ingest (the triples / node-edge normalization). The limit.kdl comment sketch from 2023 (nodes: id, type, key, value / links / meta_links) is the same shape as #5688 §3.2's triple table (subject, predicate, object + statement linkage) with property-graph views over it. The decisive external evidence: every shipping SQL/PGQ implementation is exactly this architecture — property graphs defined as views over ordinary relational tables: Oracle 23ai (first commercial SQL/PGQ, in production), DuckPGQ (the CWI research extension bringing SQL/PGQ to DuckDB), and PostgreSQL's in-core work (below). Indexing is the known triple-store playbook: B-tree permutations (subject-predicate-object and its rotations) on the embedded engines; path queries via recursive CTE where PGQ isn't available. Ingest pays parse + decompose once; no query ever re-parses text.

   (b) A Postgres extension — split the idea in two, because the halves have opposite fates.

   • Grammar (real SQL/PGQ) via extension: impossible. PostgreSQL has no parser hook, so no extension can add GRAPH_TABLE syntax — a structural fact, re-confirmed by the PG19 episode: SQL/PGQ was committed to core for PostgreSQL 19 (Mar 2026) and reverted (Sep 2026) over catalog/dependency design, expected back no earlier than PG 20. If the platform itself had to go in-core and still slipped, an extension-side grammar is not a plan.
   • A bespoke kdl document type + operators + GIN opclass: feasible but wrong target. The jsonb playbook (GIN's key-extraction contract is public and type-agnostic; jsonb_path_ops shows the compact-path-index shape) proves a document type can be indexed well in Postgres. But it buys document-local search in exactly one engine — and per design(id): serverless edition storage — D1 vs Turso vs both for id on Cloudflare Workers #5681/id: design: Querying local and remote ID servers v1 #5675, ID's server data plane is consolidating on Turso/SQLite, where a Postgres extension does nothing. If a Postgres-resident tier ever needs document-local KDL search, the cheaper move already exists: store id: design: KDL as human presentation format + triples + ANSI graph SQL (SQL/PGQ) for ID data #5688 stage-2's canonical JSON in jsonb and use the existing GIN stack. Do not build a kdl type. The Apache AGE experience is the standing warning for extension-side graph layers generally: queries as opaque strings the planner can't see into, sequential scans, no automatic label indexes.
   • (No KDL-specific Postgres extension surfaced anywhere in this pass — consistent with id: design: KDL as human presentation format + triples + ANSI graph SQL (SQL/PGQ) for ID data #5688 §4's "no adapter is known to exist.")

   (c) SQLite / Turso tier. Same rows as (a) — that sameness is the point. Permutation B-tree indexes and recursive CTEs are native-shaped here; SQLite's JSONB storage (3.45+) covers the stage-2 JSON anchor if it needs local search; TursoDB is evolving quickly but offers no PGQ and no KDL surface, and the design should not wait on either. The answer is explicitly per-tier: one row model everywhere; the query surface differs — SQL/PGQ where an engine grows it (Oracle-class servers today; Postgres from PG20-era; DuckPGQ for analytics), a translated PGQ subset over CTEs on SQLite/Turso (the capability matrix is #5688 §3.3's deliverable), and KQL inside documents only.

   (d) Native KQL over purpose-built indexes. Two independent problems. First, status: the QUERY-SPEC's current text describes "KQL next" as unreleased, and the 2.0.0 release notes already flagged KQL as not finalized — the fork would be building store infrastructure on a draft. Second, shape: KQL is a CSS-selector-style document extraction language — node selection within a document, with no joins, aggregation, or graph patterns. It is the right tool where upstream already puts it (in-document selection, CLI/human surface, the fork inherits it free) and the wrong substrate for cross-document graph search. Building dedicated indexes for it duplicates (a) with a weaker model.

   Recommendation.

   1. Rows are the query substrate on every tier (triple table + node/link tables per the limit.kdl sketch, permutation indexes); KDL text — authored pretty, stored compact (thread 0) — is the provenance/interchange payload attached to the statement, never the thing queried.
   2. Query surface per tier: SQL/PGQ as the target model (id: design: KDL as human presentation format + triples + ANSI graph SQL (SQL/PGQ) for ID data #5688 §3 stands); translated subset + recursive CTEs on SQLite/Turso until/unless those engines grow PGQ; DuckPGQ as the analytics-side proof the model works embedded; KQL confined to in-document selection.
   3. No Postgres extension work. Revisit only if a Postgres-resident tier becomes real, and then via jsonb, not a bespoke type.

   Experiments that settle the remainder (proposed for the follow-on pass):

   • Rows-vs-document bake-off on a limit.kdl-shaped generated corpus (10k / 100k / 1M statements in the character/predicate/object shape): (i) triple rows + permutation indexes, CTE and translated-PGQ arms; (ii) canonical-JSON-in-jsonb + GIN on a Postgres side instance (the (b) control arm); (iii) KQL over parsed documents with an in-memory index (the (d) control arm). Query set drawn from id: design: KDL as human presentation format + triples + ANSI graph SQL (SQL/PGQ) for ID data #5688's shapes: path patterns, annotation filters including parameterized payloads ((object#book#author:…)), qualifier joins. Measure ingest s/1k docs, index bytes, p50/p95 per query class.
   • Compact-arm test: the same bake-off with the stage-2 anchor stored as compact KDL vs canonical JSON — bytes, and reparse cost when a statement must be re-expanded to its authored form.
   • Parse-profile benchmark (feeds thread 2): upstream kdl vs fork profiles (preserve / no-preserve / no-preserve+arena) on the same corpus — this is also §5 question 5's acceptance bar, now with the corpus defined.

   ---

   Amendments this pass makes to §5's follow-on list

   • §5.5 (benchmarks) gains a defined corpus (the limit.kdl shape, generated at three scales) and the three-arm query bake-off above.
   • New follow-on: the compact emitter (thread 0) — small, serializer-side, with the canonicalization rules written down (identifier quoting choice, number rendering, annotation spacing) before implementation, since canonical bytes are what get hashed.
   • §5.3 (serde shape for annotation lists) should now also answer for facet: facet-kdl consumes the same AST, so the annotation-list shape must be exposable through both derive ecosystems.

   — WS-I investigation pass, 2026-11-04. Sources: hygge repo read in full; kdl 6.7.1 source (src/fmt.rs, tree listing) and docs; KDL 2.0.0 release notes + QUERY-SPEC current text; kdl.dev implementations table; knus/knuffel/kaydle docs; facet author's own performance posts (figures attributed and caveated in-thread); miniserde/postcard docs and a third-party serialization micro-benchmark (flagged as such); PostgreSQL PGQ commit/revert coverage, DuckPGQ and Oracle 23ai documentation, PostgreSQL GIN documentation; size measurements computed from the laptop's real files.
5. dezren39

   added

   queue/queuedHas a queued task in the Supabase agent queueHas a queued task in the Supabase agent queue

   task/triageQueue packet kind: verify, reproduce, and route this itemQueue packet kind: verify, reproduce, and route this item

   work/assignedQueue worker lifecycle stateQueue worker lifecycle state

   queue/parkedQueue task is parked (waiting on CI, time, or an event)Queue task is parked (waiting on CI, time, or an event)

   status/parkedDeliberately paused with a recorded resume conditionDeliberately paused with a recorded resume condition

   and removed

   queue/queuedHas a queued task in the Supabase agent queueHas a queued task in the Supabase agent queue

   on Nov 5, 2026
6. dezren39 commented on Nov 5, 2026

   

   dezren39

   on Nov 5, 2026

   MemberAuthor

   More actions

   🤖 Agent note · handed off

   • Agent: queue worker (repo-worker-code), lane code-g
   • Session / ID: task 10163
   • Found via: queue claim
   • Task: Triage issue id: research/design: fork a Rust KDL stack for ID — universal + multiple annotations, upstream archaeology, fork-base selection (extends #5688) #5753 — assigned to the opencode execution lane; handing off to the opencode claimant, which launches the bound session. This lane did not work the task directly.