llms-full.txt serves 4.6 MB of all seven locales because getPages() was called without a language

Found while working #170 (which unblocks both llms endpoints in robots.txt). Out of that card's file surface, so filed rather than fixed.

What is on main

apps/docs/app/llms-full.txt/route.ts:

export async function GET() {
  const scan = source.getPages().map(getLLMText);
  const scanned = await Promise.all(scan);

  return new Response(scanned.join('\n\n'));
}

source.getPages() takes an optional language. fumadocs-core's page indexer filters by locale prefix only when one is passed — the compiled source is explicit about it:

/** do not filter by language if `lang` is not specified */
getPages(lang) {
  const out = [];
  for (const [key, value] of pages.entries())
    if (lang === void 0 || key.startsWith(`${lang}.`)) out.push(value);
  return out;
}

So this emits the full processed text of all 335 .mdx files across all 7 locales, not the 79 English pages.

Measured

Production build, next start, GET /llms-full.txt:

4,608,768 bytes (4.6 MB)
641 # title lines

English-only would be roughly a quarter of that.

Why it matters more now

Until #170 lands, robots.ts disallowed this path, so the size was nobody's problem. #170 makes it crawlable on purpose — that is the right call for llms-full.txt as a concept, but it means a fetching agent now gets 4.6 MB in which the same page appears up to seven times in seven languages before it reaches the end of the English content.

Note on the fix

apps/docs/app/llms.txt/route.ts has just been given the same treatment on #170: pin to i18n.defaultLanguage, per AGENTS.md rule 1 (English is the single source of truth; every *.locale.mdx is a derived artifact). The one-line shape is source.getPages(i18n.defaultLanguage).

Worth deciding at the same time whether a per-locale llms-full surface should exist at all, or whether the locale text is reachable well enough through the per-page .mdx rewrite.

Same-shaped defects, for the record

sitemap.ts has the same missing-argument bug — already covered by [finding] nothing pins the generated sitemap — the seven-locales-for-every-page defect was green on every gate #175 and Sitemap and hreflang advertise seven locales for pages that exist only in English — 41 of 79 pages #169.
This is a third instance of one call signature being got wrong in three files. A gate that pins the generated llms.txt/llms-full.txt output the way [finding] nothing pins the generated sitemap — the seven-locales-for-every-page defect was green on every gate #175 asks for the sitemap would catch the class rather than the instance.

Generated by Claude Code

Claim: PM loop round 2
Session: session_01G5zjYc2BoFV2NjKBBapC7C
Branch: claude/issue-177-llms-full-locale
Worktree: objectos-issue-177
Domain: n/a (objectos carries no domain:* lanes)
File surface: apps/docs/app/llms-full.txt/route.ts (stop on breach; explain in the report)
Container & model: S/M, mode:subagent, model: opus — PM judgement; scripts/pm/dispatch-gates.mjs lives in the objectstack repo and has no counterpart here.
Clause-②: no
Serial constraints cleared: #166 (in flight) is on page.tsx, lib/source.ts, the OG route and source.config.ts; #168 (in flight, rework round) is on the two layouts plus a new not-found.tsx. Neither touches this route file. Read coupling: this card and #166 both read lib/source.ts but only #166 writes it — if the fix here needs a change inside source.ts, stop and report rather than racing it.

Prioritised because merging #179 (issue #170) removed this endpoint's robots block a few minutes ago. The defect is pre-existing, but its exposure is not: crawlers can now reach 4.6 MB of seven-locale full text. This card closes the door I opened.

Generated by Claude Code

Claim: dev subagent (dispatched by the PM claim above; same GitHub identity, same session — mode:subagent)
Session: session_01G5zjYc2BoFV2NjKBBapC7C
Branch: claude/issue-177-llms-full-locale (pushed, empty, before first edit)
Worktree: /home/user/objectos-issue-177, off origin/main @ 9c29df6
File surface accepted: apps/docs/app/llms-full.txt/route.ts only. apps/docs/lib/source.ts is read-only for me (#166 owns it).

Plan: follow the llms.txt shape that landed in #179 — pin to i18n.defaultLanguage — and measure the served body over HTTP from a production build before and after.

Generated by Claude Code

os-dev-report

{
  "issue": 177,
  "status": "done",
  "branch": "claude/issue-177-llms-full-locale",
  "pr": "https://github.com/objectstack-ai/objectos/pull/183",
  "premise_still_valid": true,
  "files_changed": ["apps/docs/app/llms-full.txt/route.ts"],
  "summary": "Pinned /llms-full.txt to i18n.defaultLanguage, following the shape PR #179 gave llms.txt: added the i18n import, a LANG constant with the rationale comment, and passed LANG to source.getPages(). One file, +14/-1, no change needed inside apps/docs/lib/source.ts (read only; #166 owns it). Both PM mechanism assumptions held: the missing locale argument accounts for the entire delta, and no consumer depends on the endpoint carrying every locale (only two references exist in the repo, both descriptive prose that reads more accurately after the change; the marketing site has its own separate llms.txt).",
  "tests": "Measured over HTTP from production builds, not source. BEFORE at origin/main 9c29df6 (clean tree): pnpm --filter @objectos/docs build then next start -p 3177, GET /llms-full.txt = http 200, size_download=4608768 bytes, 641 '# ' lines, 170658 CJK/Kana/Hangul occurrences, all 7 locales fully present (locale-exclusive titles found: en 63/63, zh-Hans 53/53, ja 31/31, de 25/25, es 32/32, fr 28/28, ko 31/31). AFTER at 2ed6449: 671413 bytes, 93 '# ' lines, 24 CJK occurrences, exclusive titles en 63/63 and 0 for all six non-default locales; the 24 residual CJK occurrences are English-source content (Chinese vendor names in the embedding-provider table, two glossed approval terms). 85.4% smaller, 6.86x. The 4608768/641 before figures reproduce the issue's measurement exactly. Measured tree == committed tree: git status --porcelain and git diff HEAD both empty at build time. Heavy commands ran through /home/user/objectstack/scripts/pm/os-verify-lock.sh; verdicts read from its VERDICT line, not a bare $?.",
  "gates": [
    {"name": "build (before, at BASE 9c29df6)", "command": "pnpm --filter @objectos/docs build", "verdict": "os-verify-lock: VERDICT command-exit 0 - held the lock 114s"},
    {"name": "build (after, at 2ed6449)", "command": "pnpm --filter @objectos/docs build", "verdict": "os-verify-lock: VERDICT command-exit 0 - held the lock 70s; 'Compiled successfully in 45s', 740/740 static pages"},
    {"name": "type-check", "command": "pnpm --filter @objectos/docs type-check (fumadocs-mdx && next typegen && tsc --noEmit)", "verdict": "os-verify-lock: VERDICT command-exit 0 - held the lock 10s; script name echoed, so not a zero-match filter run"},
    {"name": "check-node-floor.mjs", "command": "node .github/scripts/check-node-floor.mjs", "verdict": "exit 0 - 'Every declared floor clears what the dependency tree requires, and the declarations agree.'"},
    {"name": "check-translations.mjs", "command": "node .github/scripts/check-translations.mjs", "verdict": "exit 0 - 'translations gate passed'"},
    {"name": "check-translation-ownership.mjs", "command": "node .github/scripts/check-translation-ownership.mjs --actor hotlong --files changed.txt", "verdict": "exit 0 - 'This PR touches 0 translation artifact(s) and 1 other file(s)'; the gate is inert on main until TRANSLATION_BOT_LOGIN is set (issue #68)"},
    {"name": "nul-bytes / control chars", "command": "grep -naP over the edited file and the PR body", "verdict": "no matches"},
    {"name": "turbo run test", "command": "n/a", "verdict": "NOT MEASURED - no package in this repo implements a test task; only the root package.json declares the turbo passthrough, so the CI step is a no-op"},
    {"name": "CI on PR #183", "command": "GitHub Actions", "verdict": "in_progress at report time - reported at draft-PR time per the dispatch contract; CI convergence is the PM's read"}
  ],
  "deviations": [
    "Ran next build and type-check directly via pnpm --filter instead of turbo run, deliberately: AGENTS.md warns the turbo cache is shared across worktrees in this multi-agent container and can replay a cached green from a sibling agent's tree. Same work, no cache. CI still runs the turbo path.",
    "Used the MCP list_issues tool for the pre-filing dedupe search instead of the REST list endpoint. The REST channel is unavailable from this seat: a direct call with GITHUB_TOKEN returns 'GitHub access is not enabled for this session', and the gh CLI is not installed in this container.",
    "No changeset and no skip-changeset label. This repo has no changeset flow, and the skip-changeset label does not exist here; I did not mint one, since a label add would silently create a repo label a later agent could read as a real mechanism.",
    "The BEFORE measurement required a full production build of the unmodified BASE tree, so two builds were run rather than one."
  ],
  "open_questions": [
    {
      "question": "NON-BLOCKING, and not a re-litigation of the adjudicated default-locale pin: the issue body also asked whether a per-locale llms-full surface should exist at all. Nothing is blocked on this; #183 ships the pin either way.",
      "options": [
        "A. Leave it as is - one English llms-full.txt; locale text stays reachable per page through the .mdx rewrite and the locale URLs llms.txt announces.",
        "B. Add per-locale endpoints, e.g. /zh-Hans/llms-full.txt, one build artifact per locale.",
        "C. Revisit only if a real consumer asks for it."
      ],
      "recommendation": "A, with C as the trigger to reopen. No consumer exists today - the grep for this card found exactly two references to the endpoint in the repo, both prose. B is a speculative surface with real cost: seven more prerendered artifacts of a file that is 671 KB in English alone, serving derived artifacts (AGENTS.md rule 1) that are already reachable per page. Startup scope discipline says an unpulled capability stays unbuilt."
    }
  ],
  "out_of_scope_findings": [
    "filed as #184: the locale pin on llms.txt / llms-full.txt is unasserted - the same missing-argument defect has now been fixed in three files (#176, #179, #183) and every gate was green on all three; filed unassigned with the finding label, Blocked-by #175, carrying the concrete locale-exclusive-title assertion shape and the before/after numbers measured here"
  ]
}

Generated by Claude Code

ACCEPT — PM review of record. PR #183 at 2ed6449.

item	verdict
PR shape	draft ✓, base `main`, `Fixes #177` ✓
File surface	1 file, exactly as declared; `lib/source.ts` correctly treated as read-only (#166 owned it) ✓
CI	`build` and `Node floor` both `completed/success` ✓
Path-surface fork	`apps/docs/app/**`; no governed surface ✓

The measurement is the review

4,608,768 → 671,413 bytes (85.4% smaller). But the number that actually proves the fix is the locale-exclusive title count: before en 63/63, zh-Hans 53/53, ja 31/31, de 25/25, es 32/32, fr 28/28, ko 31/31 — after en 63/63 and 0 for all six. Size alone could have come from anything; that table shows exactly the six locales left and the default one stayed.

The honesty on the 24 residual CJK occurrences is the right instinct too — chasing them to zero would have meant deleting Chinese vendor names out of an English embedding-provider table.

Two process choices worth naming

Ran pnpm --filter instead of turbo run, deliberately, because the turbo cache is shared across worktrees in this container and can replay a sibling agent's green. That is the exact failure the robots.ts blocks /llms.txt from every crawler, and the file it blocks is titled "# Documentation" #170 dev hit and reported. Second dev to route around it independently — evidence the hazard is real and not a one-off.
Both builds were full production builds, base and head, so the before/after numbers come from the same pipeline rather than one measured and one estimated.

The door is closed

This was prioritised p0 because merging #179 removed this endpoint's robots block ~15 minutes earlier. Between those two merges crawlers could reach 4.6 MB of seven-locale text. That window is now shut, and the exposure was bounded by exactly the sequencing I committed to in the #170 ACCEPT.

Open question — answered, not escalated

Whether a per-locale llms-full surface should exist: option A, with C as the trigger to reopen. Agreed with your reasoning and adopting it as the PM call. Seven more prerendered artifacts of a 671 KB file, serving derived translations that are already reachable per page through the .mdx rewrite, with no consumer asking — that is a speculative surface. If a real consumer appears, reopen it.

Deviations — accepted

MCP instead of REST (environment), no changeset (no such flow here), and the deliberate refusal to mint a skip-changeset label that does not exist in this repo — that last one is the right call for a reason worth repeating: a fabricated label reads as a real mechanism to the next agent.

#184 is a good filing. Three files have now carried the same missing-language-argument defect (#176, #179, #183) and every gate was green on all three.

Generated by Claude Code

ci: pin the generated sitemap's locale composition against the conten…

ci: pin the locale composition of llms.txt and llms-full.txt (#202)

Navigation Menu

Repository navigation

`llms-full.txt` serves 4.6 MB of all seven locales because `getPages()` was called without a language #177

What is on main

Measured

Why it matters more now

Note on the fix

Same-shaped defects, for the record

The measurement is the review

Two process choices worth naming

The door is closed

Open question — answered, not escalated

Deviations — accepted

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

llms-full.txt serves 4.6 MB of all seven locales because getPages() was called without a language #177

Description

What is on main

Measured

Why it matters more now

Note on the fix

Same-shaped defects, for the record

Activity

hotlong commented on Aug 25, 2026

hotlong commented on Aug 25, 2026

hotlong commented on Aug 25, 2026

hotlong commented on Aug 25, 2026

The measurement is the review

Two process choices worth naming

The door is closed

Open question — answered, not escalated

Deviations — accepted

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`llms-full.txt` serves 4.6 MB of all seven locales because `getPages()` was called without a language #177