Repository navigation
audit — hotcrm ships 69,419 lines of hand-written test against a 39,734-line metadata app: split them three ways and measure how many lines os verify should have derived #15418
Description
Maintainer-requested, 2026-09-04. The deliverable is a number and its derivation, not a code change.
Why this card exists
This project's central claim is on its own repository description: "a complete CRM in under 150k tokens, one context window. Agents read it whole, reason it whole, refactor it whole."
Measured on objectstack-ai/hotcrm at 71a3452 (the 17.3.0 upgrade merge):
| lines | files | |
|---|---|---|
src/ — the metadata app itself |
39,734 | 172 |
test/ — hand-written tests |
69,419 | 169 |
content/ |
26,996 | 243 |
docs/ |
13,476 | 30 |
scripts/ |
3,765 | 13 |
160 .test.ts files against 44 object files.
And the app's own shrink-only ratchet, whose scope is src/**/*.ts minus src/translations/ and src/data/, comment- and blank-stripped:
| layer | scope | reading | ceiling | used |
|---|---|---|---|---|
| business semantics | objects flows actions hooks |
84,060 | 85,000 | 98.9% |
| interaction layer | views pages dashboards apps |
37,429 | 40,000 | 93.6% |
| authored total | the above plus the rest of src/ |
135,830 | 140,000 | 97.0% |
test/ is not in that ratchet at all. Nothing watches it.
So the honest reading of "怎么越改越复杂" is: the metadata app is disciplined and nearly at its ceiling; the growth is in the half nobody measures — and that half is where the app's actual behaviour is pinned. An agent that "holds the app whole" holds 135,830 tokens of metadata and none of the 69,419 lines that say what it must do.
What to produce
A three-way split of all 69,419 lines, by file, with the line count of each bucket.
① Should be derived by the platform, is hand-written today. A test that asserts only that the metadata is self-consistent — it needs no knowledge of the business. Candidates spotted by name and size (⛔ starting set, not the scope, and not verified by reading):
metadata-references.test.ts 1,288 reference integrity
sharing-coverage.test.ts 1,309 sharing-rule coverage
docs-view-rosters.test.ts 1,095 view rosters
i18n-references.test.ts 866 translation coverage
─────
4,558 at a glance
② Genuinely the app's own — business judgement. The platform cannot know these and should not try:
quote-discount-ceiling.test.ts 898 a discount ceiling is company policy
case-assignment.test.ts 897 who a case goes to
win-loss-capture.test.ts 857 how won/lost is booked
③ Grey zone — testing PLATFORM behaviour, but the app carries the cost.
hooks-runtime-service.test.ts 1,503
action-sandbox.test.ts 976
These assert what a hook does inside the platform's own sandbox. Related: #15325.
How to do it, and the traps
⚠️ Read the tests. ⛔ Do not classify by filename. The three names above are a starting hypothesis I formed from names and sizes and explicitly did not verify — treat them as unclassified like everything else, and say so if any of them lands in a different bucket than guessed.- Classify by the question each test asks, not by what it touches: "could this assertion be written by something that has only the metadata, and no knowledge of this company?" Yes ⇒ ①. No ⇒ ②. "is the subject the platform's behaviour rather than the app's?" ⇒ ③.
- A file may split. Report per-file line counts and, where a file straddles buckets, split it at
describegranularity rather than forcing the whole file into one. - Measure what
os verifyderives TODAY —packages/verify/src/derive.tsandrls.tsare its whole surface — and subtract it from ①. The remainder is the actual roadmap: what bucket ① asks for that the platform does not yet produce. That subtraction is the point of the card; ① alone is not the answer. ⚠️ os verify's multi-package path has been deriving zero cases and reporting success (record — the option-B reader by-shape sweep: method, derived 37-key collection set, triaged false positives, and the two packages it found #15210, fixed by reader program 5/4 —@objectstack/verify: an option-B artifact makesos verifyreport a green run that measured nothing #15229 / PR fix(verify): a multi-package app no longer gets anos verifythat measures nothing (#15229) #15281, landing now). So part of why the app hand-writes is that the derived half could not be trusted. Note where a bucket-① test predates or postdates that, if the history shows it.
⛔ What this card must NOT do
- ⛔ Delete or rewrite a single test. They are green today and pinning real behaviour; removing one before the platform derives its replacement trades a loud protection for a silent gap. This card produces a measurement, full stop.
- ⛔ Do not build the derivation. The roadmap this produces gets costed and dispatched separately, ordered by the measured demand.
- ⛔ Do not change hotcrm at all. This is read-only against that repository.
Deliverable
A record comment on this card (and, if it is long, a file under docs/ in a docs-only PR) carrying:
- the per-file three-way table with line counts and a one-line reason per file;
- the three bucket totals, and the ① minus already-derived remainder;
- a ranked list of the derivation capabilities bucket ① implies, heaviest first — this is
os verify's roadmap, ordered by real measured demand rather than by guess; - the honest reading of the size claim:
src+ the tests a metadata app cannot avoid writing (② and any of ③ the platform will never absorb) is what a truthful "how small can a metadata app be" number looks like. State it.
Activity
hotlong commentedon Sep 4, 2026
Claiming this card.
01UHvF5hyiZjnCyExFnfQB8m(https://claude.ai/code/session_01UHvF5hyiZjnCyExFnfQB8m)claude/issue-15418-test-split-audit(pushed, empty, as the write-route probe)Scope as read: a measurement only — three-way (plus undecided) split of hotcrm's
test/at71a3452, per file, with the ① minus already-derived remainder measured againstpackages/verify/src/derive.ts+rls.ts. No test deleted or rewritten, no derivation built, hotcrm read-only.Generated by Claude Code
Generated by Claude Code
docs(audits): split hotcrm's 69,419 hand-written test lines three ways (
hotlong commentedon Sep 4, 2026
Record — the three-way split, measured
Full audit:
docs/audits/2026-09-hotcrm-handwritten-test-split.mdin #15431 (draft,docs-only). It carries the per-file table (169 rows), the describe-level detail for
the 22 files that straddle buckets, and the method. Summary here.
Read-only against
objectstack-ai/hotcrmat71a3452. Every one of the 169 fileswas read; nothing was classified by filename. hotcrm was not modified and no test
was deleted, rewritten or reclassified in place.
1 + 2. Bucket totals, and the subtraction
Bucket (1) by derivation family — this split is the roadmap, so it is not folded away:
(1) minus already-derived = 20,505 lines. The subtraction is zero.
derive.ts+rls.tsderive exactly two app-agnostic families: per-object CRUDround-trip type fidelity, and the RLS "you cannot write what you cannot read"
invariant fanned out per declared position. Neither corresponds to any bucket-(1)
test in hotcrm — there is no per-object CRUD round-trip test anywhere in its 160
files, and the RLS prover asks a runtime question where (1c) asks a static one.
(The package's other three exports —
checkLedger,checkReadCoercion,checkDateBucketParity— derive nothing from an app's metadata; they are platformand driver conformance helpers, so they subtract nothing either.)
And, more bluntly: hotcrm does not depend on
@objectstack/verifyat all. It isabsent from
package.json;deriveCrudCases/runCrudVerification/runRlsProofsreturn zero hits across the repository. hotcrm's own
pnpm verifyis a locallycomposed script that never invokes the platform's derived proofs.
On the #15210 hypothesis — the history does not support it. Every bucket-(1)
file was added between 2026-07-27 and 2026-08-17; #15210 was filed 2026-09-04, three
to six weeks later, and describes a multi-package composed config that a
single-package app like hotcrm cannot reach. The honest reading is simpler than
"the derived half could not be trusted": the derived half was never offered to
this app.
3. The derivation roadmap, heaviest first
half of bucket (1), and the family furthest outside
os verify's current remit:it needs the docs corpus as a second input, which the platform does not read today.
view, action, nav entry, dashboard, dataset, import mapping, skill or AI binding
mentions.
os validatechecks shape; nothing checks the names inside it. Closestto what the platform already does — the cheapest first win.
returns a verdict for every record shape its trigger can hand it. Four files, all
written as explicit "HOUSE RULE" suites because the failure is silent.
ungranted object is denied for everyone, admins included.
concept for its own spec (ADR-0049, the liveness ledger); this is the same
discipline applied to an app's metadata, hand-rolled.
4. The honest reading of the size claim
Token figures use hotcrm's own ratchet rule (
check-source-token-ratchet.mjsstripComments, blank-stripped,chars / 4). Running that script unmodifiedreproduces its published
authored total ~135,830, which is the check that the samerule is being applied here.
srcas the ratchet scopes itsrc(adding translations + seed data)src+ the tests the app cannot avoid — floorsrc+ everything still hand-written after (1) is derived — ceilingsrcandtesttodayThe number to quote is ~344,000 tokens —
srcat 135,830 plus bucket (2) at208,252 — and that floor already assumes the platform derives all of (1) and
absorbs every last line of (3), the most generous assumption available. It is 2.3x
the claim and does not fit one context window. An agent holding the app whole at
135,830 tokens holds the metadata and none of the 208,252 tokens that say what the
metadata must do.
test/at 466,587 tokens is 3.4x the authoredsrcsurface the claim ismeasured on. Of its 69,419 raw lines, 22,953 are comment and 5,947 blank — 40,519
execute.
I did not split bucket (3) into absorbable and permanent, because that line is
not measurable from the test text — it depends on roadmap decisions not yet made. So
the app's unavoidable test surface is reported as a range, 208,252 to 306,224 tokens.
What can be said precisely: 4,069 lines are unambiguously absorbable — the
harness family (
helpers/flow-harness,helpers/hook-harness,helpers/action-sandbox,helpers/tenancy-probe, plus the four suites that existonly to prove those stand-ins have not drifted from the engine). Every line of it
exists because the platform ships no first-party test harness.
Corrections to the starting hypothesis
hooks-runtime-service.test.ts(1,503) is (2), not (3) — the one real miss,and the largest file in the guess set. The stated reason ("inside the platform's
own sandbox") describes
action-sandbox.test.ts. This file imports thehand-written
helpers/hook-harness, callshook.handler(ctx)with the closureintact, and boots no sandbox and no engine. It asserts the 8-hour SLA clock, what
a guest submission may state, contract and campaign validation — app policy.
sharing-coverage.test.ts(1,309) is 90% a docs test — bucket right (1),family surprising: 1,178 lines are (1f), only 131 are (1c). Five of its eight
describes check the admin docs' OWD, sharing-rules and related-list tables against
the app in every locale. The name reads as authorization.
action-sandbox.test.ts(976) — (3) confirmed.metadata-references.test.ts(1,288) — (1) confirmed (1,188 (1a) + 60 (1b) +a 40-line retired-persona tail that is (2)).
docs-view-rosters.test.ts(1,095) andi18n-references.test.ts(866) — (1)confirmed. Note 432 of docs-view-rosters' 1,095 lines are its file-level comment.
quote-discount-ceiling.test.ts(898) — (2) confirmed.case-assignment.test.ts(897) — mostly (2), but 214 lines measure a platformpremise (what the transfer gate can see of a
beforeInsertowner_idstamp).win-loss-capture.test.ts(857) — mostly (2); 62 lines are the same assertionre-run on real SQLite only because driver storage shapes differ.
Bucket (1) is 4.5x the at-a-glance guess (20,505 vs 4,558), and its two heaviest
families appear nowhere in the starting set.
The undecided bucket, and why
gates and CI config. Not business judgement, not metadata self-consistency, not
platform behaviour: a fifth family the three-way taxonomy has no slot for. The
platform will never derive these.
forecast-current-quarter-view/ "no view label promises a time scope its filterdoes not express". A platform could mechanically check that a view whose label
carries a period token has a filter expressing it — but deciding that "This
Quarter" promises a quarter means reading an app-authored string as language.
Left undecided rather than forced.
The softest calls are named in the doc's Method section so they can be checked rather
than trusted — chiefly the (1f) family as a whole, which passes the card's test (no
business knowledge needed) but needs an input surface
os verifydoes not read.Generated by Claude Code
Generated by Claude Code
hotlong commentedon Sep 4, 2026
os-dev-report
{ "issue": 15418, "status": "done", "branch": "claude/issue-15418-test-split-audit", "pr": "https://github.com/objectstack-ai/objectstack/pull/15431", "premise_still_valid": true, "summary": "Measurement card, delivered as docs/audits/2026-09-hotcrm-handwritten-test-split.md (draft docs-only PR #15431) plus a record comment. All 169 files of hotcrm test/ at 71a3452 were read and classified — none by filename; hotcrm was not modified. Split: (1) platform-derivable 20,505 lines / ~129,752 tokens (29.5%); (2) app business judgement 28,242 / ~208,252 (40.7%); (3) platform behaviour the app pays for 15,265 / ~97,972 (22.0%); (4) undecided 5,407 / ~30,610 (7.8%), of which 5,172 is a fifth family the taxonomy has no slot for (tests of hotcrm's own scripts and CI) and 235 is genuinely ambiguous. The ①-minus-already-derived subtraction is ZERO: os verify derives only per-object CRUD type-fidelity round-trips and the RLS cannot-write-what-you-cannot-read invariant, neither of which corresponds to any bucket-(1) test — and hotcrm does not depend on @objectstack/verify at all (absent from package.json; deriveCrudCases/runCrudVerification/runRlsProofs return zero hits). The remainder is the whole of bucket (1). The #15210 hypothesis is not supported: every bucket-(1) file was added 2026-07-27..2026-08-17, #15210 was filed 2026-09-04 and describes a multi-package path a single-package app cannot reach — the derived half was never offered, not distrusted. Honest size reading, in the claim's own units (hotcrm's ratchet rule, which reproduces its published 135,830 for src): src + the tests the app cannot avoid = ~344,082 tokens, 2.3x the 'under 150k tokens, one context window' claim, and that floor already assumes the platform absorbs ALL of bucket (3). test/ alone is 466,587 tokens, 3.4x the authored src the claim is measured on.", "tests": "No product code changed — one new file under docs/audits/. Gate union derived from the real diff, not guessed: `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack` = 16 runnable families. All 16 PASS on the final commit 74c774f06, re-run after commit through `bash scripts/pm/os-verify-lock.sh -c ...` (VERDICT command-exit 0, held 106s). Exit codes captured by redirect-then-capture, never through a pipe. One gate first returned exit 3 — check:doc-formula-expressions, its OWN 'PREREQUISITE NOT MET / nothing was measured' because @objectstack/formula and @objectstack/lint were unbuilt in the fresh worktree; read as NOT MEASURED, not as a red. Both were built (turbo build, joined with && so the lock verdict covers both legs) and it then printed its own verdict lines: 'self-test: 58 cases passed', '22 record-scoped formula example(s) across 427 files / 1371 TS blocks judged clean', '9 @example(s) judged clean across 1159 packages/spec/src files', '14 predicate(s) judged clean; 6 skipped as undeterminable'. Byte hygiene: `pnpm check:nul-bytes` OK (7482 files, no raw control bytes) and a direct grep -naP control-byte scan of the new file returned exit 1 (no matches). No ablation applies — this card produces a measurement, not a behaviour change. MEASUREMENT PROVENANCE: token figures are produced by importing hotcrm's OWN scripts/check-source-token-ratchet.mjs (stripComments + tokensOf, chars/4), and running that script unmodified reproduces its published authored total of 135,830 for src — that reproduction is the control proving the same rule was applied to test/. Line totals reconcile exactly: the per-file classification sums to 69,419 across 169 files with zero unassigned, matching wc -l.", "mcp_calls": "12 — 1 claim comment, 2 record/report comments, 4 issue/PR reads, 1 PR create, 1 label write, 1 label read-back, 1 failed get_labels (GraphQL could not resolve a just-created PR as an issue), 1 issue_read of #15210 for the history question", "open_questions": [], "out_of_scope_findings": [] }Generated by Claude Code
Generated by Claude Code
yinlianghui commentedon Sep 9, 2026
Close-out (consolidated seat,
session_018rzQyhLGC5iVs11V3TzRs5): the deliverable landed as PR #15431 (merged 2026-09-05T02:41:32Z, docs-only,skip-changeset), the dev report of 2026-09-04T15:33:04Z names it, and the card stayedpm:dispatched+ assigned for four days with the branch already reaped. Closedcompleted;pm:dispatchedand the assignee cleared in the same stroke. Nothing else on this card is owed.Generated by Claude Code
docs(audits): split hotcrm's 69,419 hand-written test lines three ways (