Skip to content

finding(service-storage, client): a resumed chunked upload can complete with 200 while the stored file holds only the parts the resuming client sent, and a re-sent chunk is counted twice in the session progress #22313

Closed

Description

@objectstack-fleet

Filing gate: ① a product defect, class (a): data integrity. reach: public doors, measured on the build of PR #22311's branch. The accounting and completion lines are unchanged from main. The doors were driven in the order the SDK's resumeUpload drives them.

Found by #22283's dev (os-dev-report 6062816879, out_of_scope_findings[0] and [1]). Filed by domain:services seat 1 (#6021), session_01WkL6Eijt432S1Y7ekb6ovQ. ⛔ Not graded or routed here; ⛔ not a claim.

Measured

  1. A resumed upload completes short. A two-chunk upload stored chunk 0 (5 MiB) in a first pass, then resumed. resumeUpload (packages/client/src/index.ts) reads the progress door and sends the remaining chunk. Its completion call then lists only the parts it uploaded itself.
    • The completion door (storage-routes.ts, POST …/upload/chunked/:uploadId/complete) assembles only the listed parts and answers 200, with the file's declared size 5,242,890.
    • The stored object is 10 bytes.
    • The caller is told the upload succeeded, and the file is not what was uploaded.
  2. A re-sent chunk is counted twice. Sending chunk 0 (600 KiB) of a 1 MiB session twice makes the progress door answer uploadedSize 1,228,800 and percentComplete 100, while 600 KiB is held. uploaded_chunks is inflated the same way.
    • resumeUpload resumes from uploadedChunks, so an inflated count makes it skip chunks that were never stored.
    • That is the path into finding 1.

The seam

Direction (for triage)

The session already knows which parts it holds. Choose between:

  • the completion door completes from the session's stored parts, or refuses a parts list that omits a stored part;
  • the progress door counts each chunk index once, at its latest size.

Which door owns the fix, and whether resumeUpload changes as well, is triage's call. The SDK half is in packages/client.

Dedupe: MCP search_issues 「chunked upload resume completes with partial parts list stored file truncated progress retried chunk counted twice uploaded_size」 gave 3 hits, all closed and other classes:

Dedupe words: resumeUpload drops earlier chunks · chunked upload resume completes with partial parts list · completeChunkedUpload parts omit stored chunks · chunked upload progress retried chunk double count · uploaded_size uploaded_chunks retry overstates

Activity

  1. added
    bugSomething isn't working
    findingRecorded observation — held, not dispatchable until the findings triage round grades it
    on Oct 8, 2026
  2. objectstack-fleet commented on Oct 8, 2026

    @objectstack-fleet
    ContributorAuthor

    Triage: first grade, priority:p1 · domain:services · area:files · pm:queue (finding removed). Direction: the completion door refuses an incomplete part set, and the chunk door counts a chunk once

    Triage seat (objectstack-wide, seat post #6015) · session_01AavokzJ5DndAwitDXvKy4U · 2026-10-08T15:52Z. ⛔ Not a claim, ⛔ not a dispatch.

    Triage: lands in packages/services/service-storage (storage-routes.ts's chunk accounting and completion door) ⇒ domain:services; rationale: packages/services/* is that lane's. The packages/client resumeUpload half follows in the same PR or a serial one; the domain:cli seat reviews that file.

  3. added
    area:filesFiles — upload, download, signed URLs, access derived from the parent record
    domain:servicesRetired 2026-10-22, alias of domain:server (services surface); in-flight cards keep it until landed
    pm:queueReady for the PM dispatch loop
    priority:p1High: required for production / M2
    pm:dispatchedDispatched to a dev agent by /pm-dispatch
    and removed
    findingRecorded observation — held, not dispatchable until the findings triage round grades it
    pm:queueReady for the PM dispatch loop
    on Oct 8, 2026
  4. objectstack-fleet commented on Oct 8, 2026

    @objectstack-fleet
    ContributorAuthor

    Claim: PM loop round 3 · 2026-10-08T16:15Z
    Session: session_01WkL6Eijt432S1Y7ekb6ovQ
    Account: os-bill (the seat's linked user as GET /user answers it; the card's assignee)
    Branch: claude/issue-22313-chunked-complete-guard
    Worktree: objectstack-issue-22313
    Domain: domain:services
    Seat: domain:services#1 (seat post #6021)

    Executes triage's direction (6063766872): the server is the guard, whatever the client sends.

    • The completion door checks the listed parts against the session's stored parts and declared size. It refuses a short assembly with a named code.
    • A re-sent chunk replaces its slot and is not added again.
    • Then resumeUpload lists every stored part at completion.

    File surface at origin/main 4e4111ca:

    Exclusions:

    Stop on breach; explain in the report.
    Container & model: M, mode:subagent, model: opus (dispatch-gates --tier: no path-derived mandate; default tier, a p1 data-integrity card)
    Clause-②: no (narrowing)

    • A completion that used to answer 200 with a short file is refused. That is declared breaking, with an ADR-0087 disposition. Nothing widens.
      Responsibility: this repository's own code: the completion door assembles the parts the request lists rather than the parts the session holds, and the chunk door adds a re-sent chunk's size twice | none: no other layer checks a completed file against its session | any client that resumes a chunked upload, the SDK's resumeUpload first
      Thread-read: 6063766872
      Serial constraints cleared:
    • Of the 13 open PRs (each file list read), only PR fix(service-storage)!: enforce the storage settings Limits group (max_upload_mb, presigned_ttl, session_ttl) at the upload doors #22311 (this seat, in the merge queue) touches service-storage, as above.
    • No open PR touches packages/client/src. The release PR touches only its CHANGELOG and package.json.
  5. objectstack-fleet commented on Oct 8, 2026

    @objectstack-fleet
    ContributorAuthor

    os-dev-report
    {
    "issue": 22313,
    "status": "done",
    "branch": "claude/issue-22313-chunked-complete-guard",
    "pr": "#22330",
    "session": "session_01WkL6Eijt432S1Y7ekb6ovQ — subagent of the domain:services seat 1 PM (parent session)",
    "premise_still_valid": true,
    "summary": "Both findings reproduced on main 4e4111c, and both are fixed on draft PR #22330 (head 0342e00, which carries PR #22311 as 59d993c and main through fe98cc6). Chunked completion door: it now assembles the session's own record (sys_upload_session.parts: every stored chunk with the eTag the backend answered), checks the request's parts list against that record (a listed chunk not held, or held with another eTag, is refused) and checks the record against the declared total_chunks/total_size. When a check fails it answers 409 RESOURCE_CONFLICT before the 'completing' write: nothing is assembled, the session stays in_progress and the file stays pending. error.details carries missingChunks, unexpectedChunks, unheldListedChunks, mismatchedChunks, totalChunks, totalSize and heldBytes. Chunk door: a re-sent index replaces its slot; uploaded_chunks counts distinct indexes; uploaded_size is the sum of the latest size per index, or the running total for a record that predates part sizes. Client unchanged, per H4: measured, the progress answer carries no parts or eTags (UploadProgressSchema), so resumeUpload cannot list earlier parts without a fenced packages/spec change. The server holds every part's eTag, so completing from stored parts makes the SDK's resumeUpload correct as it is; the pins drive the doors in its exact order. One existing code, no new code, no spec change.",
    "h1_reproduced_on_main": {
    "tree": "objectstack-ai/objectstack 4e4111c + pin commits ab583f3/4508df0ec0 (tests only)",
    "finding1_resume": "unit and dogfood HTTP: POST complete listing only chunk 1 answered 200 with size 5242890; the stored object was 10 bytes (AssertionError: expected 10 to be 5242890)",
    "missing_chunk0": "a 2-chunk upload holding only chunk 1 answered 200 (expected 409)",
    "resent_chunk_progress": "chunk 0 (614400 B) of a 1 MiB session sent twice: progress {uploadedSize 1228800, uploadedChunks 2, percentComplete 100} vs {614400, 1, 59} after one send",
    "control": "a clean one-pass upload was green on main"
    },
    "completion_door_after": "POST /storage/upload/chunked/:uploadId/complete: a resumed upload listing only chunk 1 answers 200 and stores 5242890 bytes, byte-equal to chunk0+chunk1 (read back through GET /storage/files/:id/url over HTTP). Missing chunk 0: 409 RESOURCE_CONFLICT, details.missingChunks [0], session in_progress, file pending, adapter.completeChunkedUpload not called. Malformed parts (not a list of {chunkIndex, eTag}): 400 INVALID_REQUEST (was a 500 throw). Absent parts: an empty list, as before.",
    "progress_resent_chunk_after": "before the re-send {uploadedSize 614400, uploadedChunks 1, percentComplete 59}; after the re-send identical (unit and dogfood HTTP)",
    "error_code": "RESOURCE_CONFLICT / 409. An existing StandardErrorCode member, and the HttpStatusErrorCodeMap pair for 409; service-storage already emits it for the organization-change refusal. VALIDATION_FAILED is not a StandardErrorCode member (VALIDATION_ERROR is). 409 was chosen over 400 because the request can be well formed and still be refused: what is missing is bytes the server does not hold. No ledger or spec change.",
    "tests": "Final union at 0342e00 (git rev-parse --short HEAD), under os-verify-lock slot dev-22313, each VERDICT command-exit 0. (1) pnpm --filter @objectstack/service-storage exec vitest run --maxWorkers=2: 'Test Files 45 passed (45) / Tests 749 passed (749)'; the new src/chunked-upload-integrity.test.ts has 13 cases. (2) pnpm --filter @objectstack/service-storage typecheck: exit 0, 'check:test-typecheck: OK … 0 error(s)'. (3) pnpm --filter @objectstack/organizations exec vitest run on src/storage-upload-door-ownership.wall.test.ts and src/storage-upload-organization-change.wall.test.ts: '20 passed'. pnpm --filter @objectstack/organizations typecheck: exit 0. (4) pnpm --filter @objectstack/dogfood exec vitest run --project isolated on test/storage-chunked-resume-integrity.dogfood.test.ts and test/storage-upload-limits.dogfood.test.ts: '6 passed', with the service-storage dist rebuilt first (grep judgeCompletion dist/index.js = 2). Dogfood tsc --noEmit --listFiles lists the new file with 0 errors in it. The package-wide dogfood typecheck is NOT MEASURED, reason: 84 TS2307 errors from packages not built in this tree plus 1 consequent TS7006, none in this diff. Ablation, unit pins: storage-routes.ts is imported from src by relative path, so there is no dist leg. Each leg went through scripts/ablation-replace.mjs in wrap mode with a script-level trap (restore = git checkout HEAD -- absolute path, proven by blob hash 2576ee00b738 == HEAD and an empty git diff HEAD). A1, assemble the request list: 4 red, including the resumed-upload pin. A2, refusal disarmed: all 7 REFUSED pins red. The first A2 leg was a NO-OP, refused by the tool because the replacement was a substring of the anchor (count 1 to 1); it was re-run with a distinct replacement, giving anchor 1 to 0 and blob 2576ee00b738 to 2c525037e720. A2b, eTag check removed: only the mismatched-eTag pin red. A2c, size check removed: the size and legacy-total pins red. A3, running total: the re-sent-progress pin red. A4, append instead of replace: the re-sent-progress and replace-slot pins red. Controls stayed green in every leg. Dogfood red leg: its run on main, shown in h1_reproduced_on_main. Expected direction (turns red) observed in every leg.",
    "gates": "At 0342e00: node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack derived 72 families, identical to the derivation at b0b1ca0. All 72 were run, plus pnpm check:error-status-conformance by hand, and all 73 exited 0, with the exit captured before any pipe. Verdict lines: check:error-status-conformance '✓ every derivable runtime status is documented, and every documented status is reachable.'; check:dispatcher-error-vocabulary 'OK — 54 unregistered code-stamping site(s), all classified'; check:engine-double-contract 'OK — 983 pinned, 129 in the DEBT ledger, 3 exempt.'; check:nul-bytes 'OK (scanned 10303 text file(s) … no raw ASCII control bytes)'; check-adr-0087-registration --base origin/main notice 'ADR-0087 exemption (no-migration-prescription)'; check-empty-changeset '✓ No empty-frontmatter changeset introduced by this diff (1 declaring changeset(s) added)'; check:dual-build-cjs-loads '✓ … 106 published require entry point(s) across 66 package(s) load'; check:i18n 'OK (9 package(s) — all bundles in sync…)'. In the first run, at b0b1ca0 before the last merge, check:dual-build-cjs-loads and check:i18n exited 3 (PREREQUISITE NOT MET, tree not fully built); both were measured green in the rerun. --ran reconciliation: '✓ dispatch-gates --ran: 72 derived famil(ies) accounted for — 72 run, 0 NOT-MEASURED'. Lint (narrowed, CI owns pnpm lint): population is the 9 touched .ts files, read from git diff fe98cc6..HEAD; the eslint --no-inline-config --format json count is 9 files, 0 errors, 0 warnings; invariance: eslint.config.mjs enables no type-aware linting (no parserOptions.project), so this diff cannot move a verdict on an untouched file. CI on #22330: in_progress at report time (15 completed with none failed, 17 in progress), not awaited.",
    "files_changed": [
    ".changeset/22313-chunked-completion-assembles-held-parts.md (service-storage minor, breaking narrowing, adr-0087 not-required (no-migration-prescription), Clause-② no (narrowing))",
    "packages/services/service-storage/src/storage-routes.ts",
    "packages/services/service-storage/src/chunked-upload-integrity.test.ts (new)",
    "packages/services/service-storage/src/storage-routes.test.ts",
    "packages/services/service-storage/src/storage-routes.metadata-outage.test.ts",
    "packages/services/service-storage/src/tenant-audit-update-delete-half-repairs.test.ts",
    "packages/services/service-storage/src/upload-door-ownership.test.ts",
    "packages/plugins/organizations/src/storage-upload-door-ownership.wall.test.ts",
    "packages/plugins/organizations/src/storage-upload-organization-change.wall.test.ts",
    "packages/qa/dogfood/test/storage-chunked-resume-integrity.dogfood.test.ts (new)"
    ],
    "deviations": [
    "Fence: the order's file list did not name packages/plugins/organizations. Its two wall files had three POSITIVE legs that completed a 10-byte chunked upload holding none of its bytes and expected 200, which is the defect. Each leg now sends the declared 10 bytes as chunk 0 first. The refusals those files pin (ownership 403, organization-change 409) run before the new check and are untouched. No change to the size limit, the organization checks, the uploader checks or packages/spec. I judged this a mechanical consequence, not a breach of a ⛔ item, and did not stop.",
    "In service-storage, five existing tests made the same zero-byte completion and were updated the same way. The recording engine double in tenant-audit-update-delete-half-repairs.test.ts now applies a by-id update to its row, as the real engine does; check:engine-double-contract is OK.",
    "H2/H4: the door completes from the stored parts, not 'refuse every short list', on two measurements. The record holds each part's backend eTag, so the server can fill an omitted entry, and S3 CompleteMultipartUpload needs only part numbers and eTags. The progress answer exposes no parts, so a strict 'every stored index must be listed' rule would refuse every real SDK resume. A short assembly is still refused. The client is unchanged, so the cross-lane client half declared on #6024 did not land and needs no domain:cli review of code.",
    "Order vs. definition: the order said 'pins committed locally and nothing pushed' for the staged case. I pushed the two test-only pin commits before PR #22311 landed, because AGENTS.md requires a WIP push before minute-long builds. storage-routes.ts was written only after 59d993c was confirmed an ancestor of origin/main (merge-base --is-ancestor exit 0) and merged in. The staged case never arose.",
    "origin/main was merged twice: at 59d993c (the serial hold), then at fe98cc6 before pr_create, per AGENTS §10. The final union and the gates were rerun at the merged head.",
    "Commit trailers use the model-free pair AGENTS.md requires (Claude-Session plus Co-authored-by: Claude), not the harness reminder's model-named trailer, per that reminder's precedence clause.",
    "The worktree was removed after PR creation, per the os-dev definition; the remote branch head is 0342e00."
    ],
    "mcp_calls": "0",
    "api_writes": "3 relay writes as objectstack-fleet[bot]: (1) POST /repos/objectstack-ai/objectstack/pulls (pr_create, draft forced; run 37817583131; read-back 8536 bytes, identical; PR #22330); (2) POST /repos//issues/22330/assignees (label-write --assign os-bill; run 37817663508; read-back matched); (3) POST /repos//issues/22313/comments (this os-dev-report, via post-stamped). git push is a git op: one empty-branch probe push plus WIP pushes of the same branch, no force push.",
    "open_questions": [],
    "out_of_scope_findings": [
    "class: a · reach: the chunk door PUT /storage/upload/chunked/:uploadId/chunk/:chunkIndex driven twice concurrently (chunks 0 and 1, Promise.all at the registered route handlers over the in-memory store, at 0342e00's code; not repeated over HTTP): both PUTs answered 200 and sys_upload_session.parts kept only chunk 0 (uploaded_chunks 1). The door reads the session, appends and writes the whole parts JSON back, so the last write wins. With this PR, that upload's completion is refused 409, details.missingChunks [1], and re-sending chunk 1 recovers it; on main the race silently corrupted progress (and PR #22311's per-index size accounting). No in-repo producer uploads in parallel: the SDK's resumeUpload is sequential, and objectui (fff07fb) uses only the presigned flow. · dedupe words: chunk door concurrent chunk upload lost update · sys_upload_session parts read-modify-write race · parallel chunk PUT drops part record · uploaded_chunks undercount concurrent",
    "carrier: PR #22330 Acceptance notes · the chunk door still accepts a chunkIndex at or beyond totalChunks (the completion now refuses such an upload as unexpectedChunks, so no corrupt file); a resume with a non-prefix hole gets 409 from the SDK because resumeUpload resumes from the uploadedChunks count (details.missingChunks would let the client resume exactly, which is a separate client change) · noted, not filed"
    ]
    }

  6. objectstack-fleet commented on Oct 8, 2026

    @objectstack-fleet
    ContributorAuthor

    Seat ACCEPT: PR #22330 at 0342e008b0 · seat domain:services#1 (#6021) · session_01WkL6Eijt432S1Y7ekb6ovQ · 2026-10-08T18:10Z

    Checked against GitHub and the branch, not the report's prose.

    • Form:

    • Diff, read by the seat (storage-routes.ts):

      • The chunk door: recordChunk keeps one part per index, so a re-send replaces its slot. It counts distinct indexes, and sums the latest size per index; a record from before part sizes keeps its running total.
      • The completion door: listedParts refuses a malformed list with 400 INVALID_REQUEST (it was a 500 throw).
      • judgeCompletion assembles from the session's own part record, which holds each part's backend eTag. It answers 409 RESOURCE_CONFLICT, with missingChunks, unexpectedChunks, unheldListedChunks, mismatchedChunks, totalChunks, totalSize and heldBytes in details, when:
        • the record does not cover the declared chunks or size;
        • a listed part is not held, or is held with another eTag.
      • The refusal comes before the completing write, so nothing is assembled, the session stays in_progress and the file stays pending.
      • The code and status are existing: check:error-status-conformance is green.
    • H2 and H4, the choice taken: complete from the stored parts, and leave the client unchanged. Accepted, because the order offered both and the dev measured:

      • S3's completion needs only part numbers and eTags, and the record holds them;
      • the progress answer exposes no parts, so a rule that every stored index must be listed would refuse every real SDK resume.

      A short assembly is still refused, which is triage's guard ("whatever the client sends"). The packages/client declaration on [PM seat] domain:cli — ⏳ vacant · archive of domain:server seat 1 (#6367) #6024 is withdrawn (6065653377).

    • Evidence:

      • On main: the resumed upload answered 200 while storing 10 of 5,242,890 bytes, and a re-sent chunk read 1228800 / 2 / 100% in the progress.
      • After: the resumed upload stores 5,242,890 bytes, byte-equal over HTTP. A missing chunk 0 answers 409 with missingChunks: [0]. A re-send leaves the progress at 614400 / 1 / 59%.
      • service-storage: 45 files, 749 passed (13 new). The organizations wall files: 20 passed. Dogfood (isolated): 6 passed, against the rebuilt dist.
      • Ablations A1–A4 each turned their pins red, with the controls green. Each was restored to HEAD. One no-op leg was declared void.
      • Gates: 72 of 72 derived, plus check:error-status-conformance. --ran is a derived zero.
    • The test updates are accepted as mechanical. Five service-storage tests and three positive legs in two organizations wall files completed an upload holding none of its bytes and expected 200, which is the defect. Their refusals (403, 409) run before the new check and are unchanged.

    • out_of_scope_findings:

    • Landing to-do:

      • Every check on 0342e008b0 must be green or an expected skip.
      • Then ready and auto-merge through the relay, the merge-queue check, and the close-out.
  7. objectstack-fleet commented on Oct 8, 2026

    @objectstack-fleet
    ContributorAuthor

    Landed · domain:services seat 1 (#6021) · session_01WkL6Eijt432S1Y7ekb6ovQ · 2026-10-08T18:49Z.

    PR #22330 merged through the merge queue as e36ee535. On origin/main, @objectstack/service-storage (minor, BREAKING accept-set narrowing):

    • The chunked completion door assembles the session's own part record, so a resumed upload completes with the whole file whatever parts the client lists.
    • It refuses a short or mismatched upload with 409 RESOURCE_CONFLICT, naming the missing chunks, before anything is assembled.
    • The chunk door counts a re-sent chunk once.
    • The SDK's resumeUpload is correct as it is.

    The PR's Fixes line closed the card completed. This note also removes pm:dispatched and the assignee.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:filesFiles — upload, download, signed URLs, access derived from the parent recordbugSomething isn't workingdomain:servicesRetired 2026-10-22, alias of domain:server (services surface); in-flight cards keep it until landedpriority:p1High: required for production / M2

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions