research: normalize the .lnk format — two incompatible variants exist today #1357

Activity

1. fastled-project-sync

   added this to FastLED Trackeron Aug 22, 2026
2. zackees commented on Aug 22, 2026

   

   zackees

   on Aug 22, 2026

   MemberAuthor

   More actions

   Tracked in the meta issue: FastLED/FastLED#3986 — remote assets: hosting, build-time resolution, and on-device filesystem deployment.
3. zackees commented on Aug 22, 2026

   

   zackees

   on Aug 22, 2026

   MemberAuthor

   More actions

   Research: the split is wider than this issue describes — and JSON was already deliberated

   Investigated against current origin/master of both repos (FastLED 3de8d4f8d, fbuild 49ec69ce). Three corrections to the issue body.

   1. The text format has THREE implementations, not one committed file

   I originally found only examples/AudioUrl/data/track.mp3.lnk and assumed it was a stray. It is not — it is one artifact of a complete, shipped v1 asset pipeline (FastLED#2284):

   Layer	Location	Role
   C++ runtime	src/fl/stl/url.h:274 fl::parse_lnk(), :325 parse_lnk_with_metadata()	parses .lnk at runtime
   C++ asset API	src/fl/asset/asset.h, asset.cpp.hpp:109	FL_ASSET("data/track.mp3") handles
   Python build	ci/compiler/asset_scanner.py:85 _parse_lnk_content()	emits the WASM manifest; docstring says it "Mirrors the C++ fl::parse_lnk_with_metadata() logic"

   ci/wasm_build.py:1712-1720 calls scan_sketch_assets() / write_manifest_json() on every WASM build. So the text format is live in the shipping product, not legacy cruft.

   Its grammar (url.h:312-325, mirrored in asset_scanner.py:85-126): first non-comment line is the primary URL; subsequent key=value lines carry metadata; recognized keys are sha256 and fallback; unknown keys ignored for forward-compat. LnkMetadata (url.h:252-259) is explicit that v1 "parsed but not enforced... The runtime ignores everything except primary."

   2. JSON-vs-text was already deliberated in fbuild, and git-LFS was explicitly rejected

   crates/fbuild-toolchain/src/lnk/format.rs:27-31 documents the rationale:

   ## Why JSON
   - human-readable + git-diff-able
   - no external tooling needed (vs Git LFS pointer format)
   - sha256 is mandatory -> reproducible builds + content-addressable cache
   - extensible via `v` field if a v2 ever happens
   

   So my framing of "git-LFS precedent argues for text" is already-considered ground, not a new input. That weakens research question 1 considerably — the fbuild side has a written position.

   3. Two schema facts I got wrong

   • sha256 is REQUIRED in fbuild's schema, not optional (format.rs:22 — "Required fields: v, url, sha256"). The text format treats it as optional and unenforced. This is the sharpest incompatibility: examples/AudioUrl/data/track.mp3.lnk has no digest at all, so it cannot become a valid fbuild .lnk without someone fetching and hashing a third-party MP3 whose bytes may change.
   • There is an extract field (file | zip | tar.gz, format.rs:41-50) with no text-format equivalent. Any normalization has to account for it.

   What this means for the decision

   The two formats are not competing drafts — they serve different consumers with different guarantees:

   • Text: runtime-parseable on an MCU (no JSON parser in the hot path), digest optional and advisory, mirrors declared via fallback=.
   • JSON: build-time only, digest mandatory, content-addressable caching, archive extraction.

   That reframes the question from "which format wins" to "should these stay two formats with a documented boundary, or does one absorb the other — and at what cost?"

   Three real options:

   1. Keep both, document the boundary. Text = runtime asset links; JSON = build-time blob pointers. Cheapest, but the shared .lnk extension guarantees continued confusion, and asset.h:13 already misdescribes reality ("At build time, fbuild scans <sketch>/data/ for *.lnk files" — fbuild's parser would reject the text files it is describing).
   2. Tolerant reader on both sides. Sniff { -> JSON, else text. ~30 lines each in Rust and Python; C++ runtime would need a JSON path it currently does not have and arguably should not carry.
   3. Converge on JSON, migrate the text pipeline. Cleanest end state, but requires a JSON parser in the C++ runtime (fl::parse_lnk is deliberately allocation-light), a digest for the AudioUrl asset, and touching a shipped v1 API.

   My recommendation is option 1 plus a rename — distinct extensions for the two roles (e.g. keep .lnk for the runtime asset link, use something distinct for build-time blob pointers), which removes the ambiguity without forcing a JSON parser into the MCU runtime. Recording it as a recommendation, not a decision: this touches a shipped API in both repos and deserves a maintainer call.

   Related discovery

   src/fl/asset/asset.h:21-22 states: "v1 scope: WASM and host/stub only. ESP32 LittleFS / SD-card resolution is future work and intentionally omitted." That is independent confirmation of the gap tracked in FastLED/FastLED#3984 and #1358 — the asset pipeline was deliberately built without a device-filesystem path.
4. zackees

   added 2 commits that reference this issue on Aug 23, 2026

   feat(ci): fold all asset declaration parsing into one scanner (#3990)

   ...

   b208ac7

   feat(lnk): accept the text-format .lnk alongside JSON (#1362)

   ...

   21e7295
5. zackees commented on Aug 23, 2026

   

   zackees

   on Aug 23, 2026

   MemberAuthor

   More actions

   Parser side landed — #1362 merged

   fbuild now reads both formats. from_str_any() sniffs the first non-blank, non-comment character ({ -> JSON, else text) and from_path() routes through it, so lnk pull, lnk check, and the ESP embed stage all gained the text format with no call-site changes. JSON stays canonical — fbuild lnk add is unchanged.

   Paired with FastLED/FastLED#3990, which folded FastLED's three Python/C++ parsers into one scanner that accepts both .lnk forms plus assets.json. Both sides of the split now read both formats.

   One deliberate asymmetry

   sha256 remains mandatory in fbuild even for the text form, where the C++ runtime treats it as optional. fbuild's resolver caches by content digest, so a .lnk without one cannot be content-addressed or verified — relaxing that would give up the guarantee the cache is built on. The error is actionable instead:

   text-format .lnk for <url> has no `sha256=` line; fbuild caches by content
   digest and cannot verify or content-address without one. Add a
   `sha256=<64 hex chars>` line, or regenerate the file with `fbuild lnk add <url>`.
   

   Verified

   • text-format .lnk fetched and sha256-verified into the blob cache
   • a deliberately wrong digest rejected, .partial never promoted — the integrity path, not just the happy path
   • 43 lnk tests pass (7 new), clippy clean

   Caveat on how it landed: merged with --admin at the user's explicit instruction. The 98-job CI matrix was still running and got cancelled by the merge (92 cancelled, 2 passed), so CI did not validate this — the evidence above is all local.

   Still open on this issue

   The parser accepts both, but the underlying question this issue asks — should there be two formats — is unanswered:

   1. examples/AudioUrl/data/track.mp3.lnk still has no sha256, so it now gets a clear error instead of a confusing one. Adding a digest means hashing a third-party MP3 whose bytes can change.
   2. Long-term direction. My recommendation stands: keep both roles but give them distinct extensions — .lnk for the runtime asset link, something else for build-time blob pointers — rather than forcing a JSON parser into the MCU runtime. That is a maintainer call, not a parser change.
   3. asset.h:13 still misdescribes reality ("At build time, fbuild scans <sketch>/data/ for *.lnk files") — truer now than before, but worth a pass.

   Leaving open for those.
6. fastled-project-sync

   moved this to Triage in FastLED Trackeron Aug 23, 2026
7. zackees

   added 3 commits that reference this issue on Aug 23, 2026

   feat(examples): mirror the AudioUrl track

   ...

   8795dcc

   fix(examples): give AudioUrl's .lnk a verifiable digest (#4012)

   ...

   1162b26

   docs(asset): correct what actually emits the asset manifest (#4013)

   ...

   d85572b
8. zackees commented on Aug 23, 2026

   

   zackees

   on Aug 23, 2026

   MemberAuthor

   More actions

   Resolved — both parsers read both formats, and the one file in the wild verifies

   The research this issue asked for is done, and every actionable item from it has landed.

   What was actually wrong

   Wider than the issue first described. Three implementations of the text format, all live in the shipping product:

   Layer	Location
   C++ runtime	src/fl/stl/url.h — fl::parse_lnk(), parse_lnk_with_metadata()
   C++ asset API	src/fl/asset/asset.h — FL_ASSET("data/track.mp3")
   Python build	ci/compiler/asset_scanner.py — emits the WASM manifest

   …against one implementation of JSON in fbuild. The single .lnk committed to FastLED was text, so the tool named lnk could not read it.

   Also corrected along the way: sha256 is required in fbuild's schema, not optional; there is an extract field (file/zip/tar.gz) with no text equivalent; and format.rs already documented a deliberate choice of JSON over the git-LFS pointer style, so "git-LFS argues for text" was settled ground rather than new input.

   Landed

   Change	Where
   fbuild reads text .lnk (sniffs { -> JSON, else text)	#1362
   FastLED's three parsers folded into one that reads both, plus assets.json	FastLED/FastLED#3990
   examples/AudioUrl gains a verifiable digest, mirrored off a third-party host	FastLED/FastLED#4012
   asset.h stops attributing the manifest to fbuild	FastLED/FastLED#4013

   sha256 stays mandatory in fbuild even for the text form. The resolver caches by content digest, so a .lnk without one cannot be content-addressed or verified — relaxing that to accommodate the text format would surrender the guarantee the cache is built on. The error names the file and the fix instead.

   A concrete demonstration of why this mattered

   While adding the digest to examples/AudioUrl, I converted its .lnk to fbuild's JSON schema — the obvious "tidy it up" move. fl_asset_asset failed:

   tests/fl/asset/asset.cpp:108: { == www.soundhelix.com
   

   fl::parse_lnk reads only the text form, so it took { as the URL. Someone was always going to do that; now the assertion carries a comment explaining why the format cannot change there.

   Not resolved here — spun out

   Whether the two .lnk roles should keep sharing an extension long-term is a design decision rather than a defect, and it needs a maintainer. My recommendation, recorded above: give them distinct extensions — .lnk for the runtime asset link, something else for build-time blob pointers — rather than forcing a JSON parser into the MCU runtime, where fl::parse_lnk is deliberately allocation-light.

   Filed as #1369 so it is not blocked behind an issue whose work is finished.

   Closing.
9. zackees

   closed this as completedon Aug 23, 2026