Skip to content

fix(ai): media-type sniffer labels every ISO-BMFF file video/mp4 — the ftyp major brand is never read, so M4A/M4B audio and AVIF/HEIC images are published as video #2127

New issue
New issue
Closed
#2144
#2142
Closed
fix(ai): media-type sniffer labels every ISO-BMFF file video/mp4 — the ftyp major brand is never read, so M4A/M4B audio and AVIF/HEIC images are published as video#2127
#2144
#2142
Labels
app/aiPackage: ai (pkgs/ or apps/ in this repo)kind/bugSomething isn't working (prefixed form of the classic bug label)priority/normalOrdinary priority (the default)queue/parkedQueue task is parked (waiting on CI, time, or an event)source/creatorFiled by the new-issue-creator jobstatus/parkedDeliberately paused with a recorded resume conditiontask/triageQueue packet kind: verify, reproduce, and route this itemwork/assignedAn agent holds this item via the queue (claimed task)

Description

@developing-today
developing-today
bot
opened on Nov 8, 2026

Problem

detectMediaType in packages/ai/src/utils/media-type.ts:20 treats any ISO-BMFF file as video:

if (bytes.length >= 12 && ascii(bytes, 4, 8) === "ftyp") return "video/mp4"

It never reads the major brand at bytes 8–12. That brand is what distinguishes the containers that share the ftyp box:

  • M4A / M4B — audio-only MP4 → should be audio/mp4
  • avif / avis — AVIF image → should be image/avif
  • heic / heix / mif1 (HEIF) — HEIC/HEIF image → should be image/heic / image/heif
  • isom / mp41 / mp42 / avc1 / dash etc. — genuine video MP4 → video/mp4
  • 3gp* — 3GPP video → video/3gpp

Because sniffing wins over the file extension (fileMediaType, media-type.ts:57: detectMediaType(bytes) ?? extensionMediaType(path)), even a correctly named song.m4a or photo.avif file is published with the wrong media type.

Evidence (verified this run, main 87c05e11db)

Probe probe-media-ftyp.ts (laptop ~/git/fleet/probe-2026-11-08-creator-media-ftyp/, run with bun against the real packages/ai/src/utils/media-type.ts on the laptop primary at 87c05e11db), constructing 12-byte ftyp headers per brand:

M4A audio brand       detect= video/mp4  file= video/mp4   (path song.m4a)
M4B audiobook brand   detect= video/mp4  file= video/mp4   (path book.m4b)
avif image brand      detect= video/mp4  file= video/mp4   (path photo.avif)
heic image brand      detect= video/mp4  file= video/mp4   (path photo.heic)
isom video brand      detect= video/mp4  file= video/mp4   (path clip.mp4)  ← correct
mp42 video brand      detect= video/mp4  file= video/mp4   (path clip.mp4)  ← correct

Note the extension table cannot rescue the audio/image cases even where it knows them: EXTENSIONS in the same file has no avif/heic entry, and the sniff result pre-empts the m4a → audio/mp4 entry that does exist.

Why it matters

The sniffed type is authoritative at every call site:

  • packages/ai/src/media.ts:140 — bytes-source assets take detectMediaType(...) first
  • packages/ai/src/media.ts:323 — Media.bytes defaults to the sniffed type
  • packages/ai/src/media.ts:361 and packages/ai/src/promise.ts:116 — file loads via fileMediaType
  • packages/ai/src/protocols/meta-responses.ts:130 — the sniffed type (or octet-stream) is what is sent on the wire

So an M4A voice note / audiobook or an iPhone HEIC / AVIF photo handed to a provider is labelled video/mp4: audio routes can reject or mishandle it, image routes never see it as an image, and downstream mediaTypeExtension / kind logic inherits the wrong kind. This is the media façade's own sniffing contract — the doc comment at media-type.ts:8-9 promises the type "of a file's contents", and for these brands it returns a type the contents are not.

Acceptance criteria

  • detectMediaType dispatches on the ftyp major brand (bytes 8–12): at minimum M4A /M4B → audio/mp4, avif/avis → image/avif, heic/heix (and HEIF mif1 if claimed) → the matching image type; genuine MP4 brands (isom, mp41, mp42, avc1, …) still → video/mp4.
  • An unrecognized ftyp brand does not silently become video/mp4 — it falls through to undefined (so fileMediaType can use the extension) or the chosen fallback is stated in the code comment and pinned by a test.
  • fileMediaType(m4aBytes, "song.m4a") returns audio/mp4 and fileMediaType(avifBytes, "photo.avif") returns image/avif (add avif/heic to EXTENSIONS if the brand dispatch does not already cover extension-only cases).
  • Tests in packages/ai/test/media.test.ts pin each brand family above plus the unknown-brand case, alongside the existing magic-byte cases; existing media tests still pass.
  • No other sniffer branch (PNG/JPEG/GIF/RIFF/EBML/frame-sync/…) changes behaviour.

Dedup: searched open + closed issues for ftyp / detectMediaType / media-type sniffing on 2026-11-08 — the only hit is closed #906 (prototype-chain lookups in the same file, fixed with Object.hasOwn); no open or closed issue covers the ftyp brand dispatch.

Activity

  1. developing-today
    added
    kind/bugSomething isn't working (prefixed form of the classic bug label)
    task/triageQueue packet kind: verify, reproduce, and route this item
    status/needs-triageNot yet triaged; needs a first look
    source/creatorFiled by the new-issue-creator job
    priority/normalOrdinary priority (the default)
    queue/queuedHas a queued task in the Supabase agent queue
    app/aiPackage: ai (pkgs/ or apps/ in this repo)
    on Nov 8, 2026

  2. developing-today commented on Nov 8, 2026

    @developing-today
    developing-todayBot
    on Nov 8, 2026 – with developing.today
    Author

    🤖 Agent note · filed

  3. dezren39
    removed
    status/needs-triageNot yet triaged; needs a first look
    on Nov 8, 2026

  4. developing-today commented on Nov 8, 2026

    @developing-today
    developing-todayBot
    on Nov 8, 2026 – with developing.today
    Author

    Triage verdict (task 13770, lane opencode-a): REAL on current main de7b444. Probe in packages/ai: a 12-byte ftyp header with the M4A major brand returns detectMediaType = video/mp4 (expected audio/mp4), and fileMediaType lets the sniff pre-empt the m4a extension entry; media-type.ts has a single ftyp branch and no avif/heic EXTENSIONS entries, exactly as filed. Fix enqueued as task 13884 (small_fix, acceptance per this issue).

    agent-note:v1 lane=opencode-a task=13770 state=triaged follow_on=13884

  5. dezren39
    added
    work/assignedAn agent holds this item via the queue (claimed task)
    queue/claimedA queue worker has claimed the task for this item
    status/in-progressActively being worked
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 8, 2026

  6. dezren39 commented on Nov 8, 2026

    @dezren39
    dezren39
    on Nov 8, 2026

    🤖 Agent note · claimed

    • Agent: repo-worker-opencode (lane 3fe37e8a-c61b-419c-9e97-769ecd264d5d)
    • Task ID: 13884
    • State: claimed
    • Summary: Claimed task 13884 to implement ftyp major-brand dispatch in media-type.ts per acceptance criteria.
  7. dezren39
    added
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    status/parkedDeliberately paused with a recorded resume condition
    and removed
    status/in-progressActively being worked
    queue/claimedA queue worker has claimed the task for this item
    on Nov 8, 2026

  8. dezren39 commented on Nov 8, 2026

    @dezren39
    dezren39
    on Nov 8, 2026

    🤖 Agent note · parked

  9. dezren39
    added a commit that references this issue on Nov 8, 2026

    Merge pull request #2144 from developing-today-forks/queue/13884-2127…

    391e43e
  10. dezren39
    closed this as completedin #2144on Nov 8, 2026

  11. dezren39 commented on Nov 8, 2026

    @dezren39
    dezren39
    on Nov 8, 2026

    🤖 Agent note · done — ftyp major-brand dispatch merged

    • Agent: repo-queue-worker-shift-opencode lane A (repo-worker-opencode, lane opencode-a); queue task 13884 (fix follow-on from triage task 13770)
    • Outcome: PR fix(ai): dispatch media-type sniffer on ftyp major brand (#2127) #2144 MERGED 2026-11-08T17:43:07Z, merge commit 391e43eb3675fb967b0ee89a86a161f222287553; this issue auto-closed COMPLETED by the PR.
    • Change: detectMediaType now dispatches on the ftyp major brand (bytes 8–12): M4A /M4B /M4P → audio/mp4, avif/avis → image/avif, heic/heix → image/heic, mif1/msf1 → image/heif, genuine MP4 brands (isom/iso2/iso5/iso6/mp41/mp42/avc1/dash/M4V ) → video/mp4; an unrecognized brand falls through to undefined instead of silently claiming video/mp4. EXTENSIONS gained avif/heic/heif. No other sniffer branch changed.
    • Verification: bun test test/media.test.ts (packages/ai) — 12 pass, new brand test 14 expects (each family + unknown brand + fileMediaType for song.m4a/photo.avif/photo.heic); prettier clean; no Codex findings on PR fix(ai): dispatch media-type sniffer on ftyp major brand (#2127) #2144 (no reviews posted).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    app/aiPackage: ai (pkgs/ or apps/ in this repo)kind/bugSomething isn't working (prefixed form of the classic bug label)priority/normalOrdinary priority (the default)queue/parkedQueue task is parked (waiting on CI, time, or an event)source/creatorFiled by the new-issue-creator jobstatus/parkedDeliberately paused with a recorded resume conditiontask/triageQueue packet kind: verify, reproduce, and route this itemwork/assignedAn agent holds this item via the queue (claimed task)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions