Measure definitions are not fully compiled: aggregate: 'count' ignores field, and /api/v1/analytics/query drops every per-measure filter — the SQL is COUNT(*) for all of them #10298

Activity

1. os-zhuang

   added

   bugSomething isn't workingSomething isn't working

   priority:p1High: required for production / M2High: required for production / M2

   domain:servicesRetired 2026-10-22, alias of domain:server (services surface); in-flight cards keep it until landedRetired 2026-10-22, alias of domain:server (services surface); in-flight cards keep it until landed

   on Aug 20, 2026
2. os-warren

   added

   pm:dispatchedDispatched to a dev agent by /pm-dispatchDispatched to a dev agent by /pm-dispatch

   and removed

   pm:queueReady for the PM dispatch loopReady for the PM dispatch loop

   on Aug 20, 2026
3. os-warren

   self-assigned this

   on Aug 20, 2026
4. os-warren commented on Aug 20, 2026

   

   os-warren

   on Aug 20, 2026 – with Claude

   Collaborator

   More actions

   Claim: PM seat #6021, session session_01PnJHU45vPJj5UQrxe946Bx.

   • Branch: claude/issue-10298-measure-compiler-field-and-filter
   • Worktree: /home/user/objectstack-issue-10298（⛔ 禁 git stash）
   • File surface: 度量编译器（analytics/dataset 面，以实测落点为准）+ 其测试 + changeset
   • Container & model: claude-opus-5
   • Clause-②: no（主路径）—— 让编译出的 SQL 与已声明的 aggregate+field+filter 一致，是修正错误的算术，不改端点的接受/拒绝集
   • Serial constraints: 与在飞的 finding: the direct /sso/register endpoint's ADR-0024 before-hook admits org owners/admins — wider than the platform-admin posture #9653 landed on the /admin/sso/* bridges #10009（plugin-auth）、同批的 app-showcase's email template declares locale: 'en', which no en-US send can resolve — and it bypasses defineEmailTemplateDefinition #10267（examples/app-showcase）、[finding] A clean first install shows the newcomer two red ✕ unmet-peer warnings (better-sqlite3, better-call) #10326（依赖清单）面不相交

   ⚠️ 分诊已命名的分叉，我把它写成硬约束：若实测发现 per-measure filter 在严格包装器上确实不受支持，卡面自己的回退是 400 点名该 measure，绝不 200 配不同的算术——而那个变体是契约变更 ⇒ ⛔ 停下回报，不要实现，由 PM 重新定级。⛔ 不得先实现再声明。

   ⚠️ 本卡是 p1，且两扇门（dashboard 与 /api/v1/analytics/query）对同一个 cube 给出不同数字。响应里的 sql 字段让回归可钉——要求把 COUNT("col") 与每个带过滤的 measure 的 FILTER/CASE WHEN 钉成断言，而不是只断言结果数值。

   ---

   Generated by Claude Code
5. os-warren commented on Aug 20, 2026

   

   os-warren

   on Aug 20, 2026 – with Claude

   Collaborator

   More actions

   {
     "issue": 10298,
     "status": "done",
     "branch": "claude/issue-10298-measure-compiler-field-and-filter",
     "pr": "https://github.com/objectstack-ai/objectstack/pull/10411",
     "premise_still_valid": true,
     "summary": "Both defects reproduced on the branch before any edit, then fixed in NativeSQLStrategy. (1) AGGREGATE_SQL['count'] discarded the resolved column, so a measure declaring `field` emitted COUNT(*); it now wraps the column. (2) A compiled dataset's per-measure filters AND its definition-level filter live in the dataset registry beside the Cube, and only DatasetExecutor read them, so /api/v1/analytics/query answered unfiltered aggregates under the author's measure names; they now reach the strategy through a `getDatasetScope(cubeName)` hook on the context service-analytics builds for its own strategies (same shape and registry as the neighbouring getAllowedRelationships), and lower to portable CASE WHEN conditional aggregates plus a plain WHERE conjunct. The 400 fork was NOT reached and that determination was made before writing the fix: per-measure filter is supportable with no change to what the endpoint accepts or rejects, so Clause-② is no. No packages/spec edit; no check-type-check-coverage ledger entry raised (436 frozen raw errors unchanged, service-analytics stays at 10). The dataset-level filter was in scope beyond the card's two named defects and is argued in the PR body: it is the same defect class on the same door, mechanically pinned by DatasetExecutor.runMeasurePass, and the card's own 'both doors agree for the same cube' criterion is unreachable without it.",
     "tests": "Gate union derived with `node scripts/pm/dispatch-gates.mjs` (no paths passed) and run AFTER the final commit on a clean worktree at 69143f891 — 15 families, every exit code captured by redirecting before any pipe, all EXIT=0: changeset-gate-self-tests, objectui-changeset, slot-lookup, test-source-alias, type-source-resolution, check-adr-0087-registration, check-changeset-no-major, check-empty-changeset, check-affected-docs, query-options-erasure, type-check-coverage, type-check-debt, engine-double-contract, where-matcher, nul-bytes. Quoted verdict lines (each printed by the gate itself, never a bare $?): `check-type-check-coverage: OK — 64/77 workspace packages type-checked (plus the root), 13 in the DEBT ledger (436 frozen raw errors), 1 exempt` · `check-engine-double-contract: OK — 338 pinned, 133 in the DEBT ledger, 2 exempt.` · `✓ where-matcher conformance holds: 266 matcher(s) discovered … none new` · `check-nul-bytes: OK (scanned 6116 text file(s) … no raw ASCII control bytes)` · `✓ slot-lookup ratchet holds: 107 unswept site(s) in 25 file(s), none new`. The debt gate needs the built closure — `turbo run build --filter=./packages/* --filter=./packages/*/*` ran first, `Tasks: 70 successful, 70 total`; its refusal on an unbuilt tree would have been a precondition, not a pass. PACKAGE SUITE: `pnpm --filter @objectstack/service-analytics exec vitest run --maxWorkers=2` → `Test Files 78 passed (78)`, `Tests 1734 passed (1734)`. PREMISE (measured on the branch at origin/main 801296050, before any edit): `SELECT COUNT(*) AS \"closed_count\", COUNT(*) AS \"kb_resolved_count\" FROM \"crm_case\"` — the declared field:'resolved_by_article' absent; and `SELECT COUNT(*) AS \"opp_count\", COUNT(*) AS \"won_count\", COUNT(*) AS \"lost_count\", SUM(amount) AS \"won_amount\" FROM \"crm_opportunity\"` with `params: []` — no FILTER, no CASE WHEN, no bound comparand. Both match the card's reported SQL. A third, same-class measurement: a dataset declaring `filter: { is_deleted: false }` compiled with no WHERE clause at all. COMPILED-SQL PINS after the fix: `COUNT(resolved_by_article) AS \"article_count\"` · `COUNT(\"crm_case\".\"resolved_by_article\")` when the cube joins · `COUNT(CASE WHEN stage = $1 THEN 1 END) AS \"won_count\"` · `COUNT(CASE WHEN is_closed = $2 THEN resolved_by_article END) AS \"kb_resolved_count\"` · `SUM(CASE WHEN stage = $3 THEN amount END) AS \"won_amount\"` · `WHERE is_deleted = $2`, with `params` asserted as an ordered array so a filter compiled outside the SELECT loop would misalign the binds. CASE WHEN rather than FILTER (WHERE …) because FILTER is Postgres/SQLite-only and this strategy compiles one statement for whichever SQL driver owns the object. BOTH DOORS: run against a real SQLite (sql.js) over the card's ground truth — 24 opportunities, 8 won, 5 lost, won revenue 1,290,000, grand total 5,632,500. The API door now answers 24/8/5/1,290,000, queryDataset answers identically measure-for-measure, ungrouped and grouped; the grouped leg groups by OWNER, not by stage, because grouping on the filter's own column makes filtered and unfiltered aggregates coincide inside the matching group. ABLATION — predicted signature stated FIRST: neutering both halves turns 8 of the 11 new tests red, survivors being the fieldless COUNT(*), the manifest-cube no-op and the fixture self-check, with the lockstep pin staying green (it reads the two tables, not the call site). Observed exactly that: `Tests 8 failed | 12 passed (20)`, including `expected { opp_count: 24, won_count: 24, …(2) } to match object { opp_count: 24, won_count: 8, …(2) }` and `expected 8 to be 24` — the two doors disagreeing again. NOTE: a FIRST ablation pass produced 7, not 8; the survivor was my own grouped test, which grouped by `stage` and so passed with the filter dropped. The test was strengthened (group by owner) and the ablation re-run to meet the prediction — reported because the discrepancy was a real finding about the test, not about the fix. REBUILD, argued from the files: none sits between the edit and the run. Both the ablated module and the test resolve through RELATIVE specifiers inside one package (`../analytics-service.js`, `../strategies/native-sql-strategy.js`), so vitest transforms src/ directly; service-analytics ships no vitest config and the repo has no alias table redirecting them; the only exports-map resolution is @objectstack/spec, untouched here and built before the first measurement. RESTORE proven byte-identical, not assumed: pre b2ac0c23212f1e766c29524bcafb499633f16e0b → ablated a51e178f328ef6626054da443042e025fb2e0f31 → restored b2ac0c23212f1e766c29524bcafb499633f16e0b, and the restore leg re-run green (`Tests 20 passed (20)`). ZERO-GREP DISCIPLINE: the claim that Metric.filters has no consumer rests on a search proven live by a positive control — `git grep -c \"\\.format\"` over the same paths returns 5 files while `\\.filters` returns none.",
     "open_questions": [],
     "out_of_scope_findings": [
       "filed as #10413: /api/v1/analytics/query still drops per-measure AND dataset-level filter on the ObjectQL path — engine.aggregate receives no filter at all (measured: aggregations [{field:'*',method:'count',alias:'won_count'}] with no filter key), so the two-door disagreement survives on objectqlAggregate drivers with this PR merged; the remedy is either widening the aggregate contract with a per-aggregation filter or a second fan-out inside the strategy, which is a decision this card does not own",
       "filed as #10414 (label `finding`): MetricSchema.filters is an authorable per-metric filter, pinned by the spec's own parse test, and consumed nowhere — a hand-authored cube's filters:[{sql}] parses and is dropped, the same declared-not-enforced shape as this card one level up; enforcing vs retiring it is a packages/spec / ADR-0049 call and this lane holds packages/spec as a zero-ownership boundary"
     ]
   }

   ---

   Generated by Claude Code
6. os-warren commented on Aug 20, 2026

   

   os-warren

   on Aug 20, 2026 – with Claude

   Collaborator

   More actions

   os-dev-report

   Supersedes the comment above, whose HTML-comment marker the GitHub body sanitizer stripped on write — verified by reading the comment back, and reposted with the marker as literal text so a PM scan can find it. Same payload, nothing else changed.

   {
     "issue": 10298,
     "status": "done",
     "branch": "claude/issue-10298-measure-compiler-field-and-filter",
     "pr": "https://github.com/objectstack-ai/objectstack/pull/10411",
     "premise_still_valid": true,
     "summary": "Both defects reproduced on the branch before any edit, then fixed in NativeSQLStrategy. (1) AGGREGATE_SQL['count'] discarded the resolved column, so a measure declaring `field` emitted COUNT(*); it now wraps the column. (2) A compiled dataset's per-measure filters AND its definition-level filter live in the dataset registry beside the Cube, and only DatasetExecutor read them, so /api/v1/analytics/query answered unfiltered aggregates under the author's measure names; they now reach the strategy through a `getDatasetScope(cubeName)` hook on the context service-analytics builds for its own strategies (same shape and registry as the neighbouring getAllowedRelationships), and lower to portable CASE WHEN conditional aggregates plus a plain WHERE conjunct. The 400 fork was NOT reached and that determination was made before writing the fix: per-measure filter is supportable with no change to what the endpoint accepts or rejects, so Clause-② is no. No packages/spec edit; no check-type-check-coverage ledger entry raised (436 frozen raw errors unchanged, service-analytics stays at 10). The dataset-level filter was in scope beyond the card's two named defects and is argued in the PR body: it is the same defect class on the same door, mechanically pinned by DatasetExecutor.runMeasurePass, and the card's own 'both doors agree for the same cube' criterion is unreachable without it.",
     "tests": "Gate union derived with `node scripts/pm/dispatch-gates.mjs` (no paths passed) and run AFTER the final commit on a clean worktree at 69143f891 — 15 families, every exit code captured by redirecting before any pipe, all EXIT=0: changeset-gate-self-tests, objectui-changeset, slot-lookup, test-source-alias, type-source-resolution, check-adr-0087-registration, check-changeset-no-major, check-empty-changeset, check-affected-docs, query-options-erasure, type-check-coverage, type-check-debt, engine-double-contract, where-matcher, nul-bytes. Quoted verdict lines (each printed by the gate itself, never a bare $?): `check-type-check-coverage: OK — 64/77 workspace packages type-checked (plus the root), 13 in the DEBT ledger (436 frozen raw errors), 1 exempt` · `check-engine-double-contract: OK — 338 pinned, 133 in the DEBT ledger, 2 exempt.` · `✓ where-matcher conformance holds: 266 matcher(s) discovered … none new` · `check-nul-bytes: OK (scanned 6116 text file(s) … no raw ASCII control bytes)` · `✓ slot-lookup ratchet holds: 107 unswept site(s) in 25 file(s), none new`. The debt gate needs the built closure — `turbo run build --filter=./packages/* --filter=./packages/*/*` ran first, `Tasks: 70 successful, 70 total`; its refusal on an unbuilt tree would have been a precondition, not a pass. PACKAGE SUITE: `pnpm --filter @objectstack/service-analytics exec vitest run --maxWorkers=2` → `Test Files 78 passed (78)`, `Tests 1734 passed (1734)`. PREMISE (measured on the branch at origin/main 801296050, before any edit): SELECT COUNT(*) AS closed_count, COUNT(*) AS kb_resolved_count FROM crm_case — the declared field:'resolved_by_article' absent; and SELECT COUNT(*) AS opp_count, COUNT(*) AS won_count, COUNT(*) AS lost_count, SUM(amount) AS won_amount FROM crm_opportunity with `params: []` — no FILTER, no CASE WHEN, no bound comparand. Both match the card's reported SQL. A third, same-class measurement: a dataset declaring `filter: { is_deleted: false }` compiled with no WHERE clause at all. COMPILED-SQL PINS after the fix: `COUNT(resolved_by_article) AS article_count` · `COUNT(\"crm_case\".\"resolved_by_article\")` when the cube joins · `COUNT(CASE WHEN stage = $1 THEN 1 END) AS won_count` · `COUNT(CASE WHEN is_closed = $2 THEN resolved_by_article END) AS kb_resolved_count` · `SUM(CASE WHEN stage = $3 THEN amount END) AS won_amount` · `WHERE is_deleted = $2`, with `params` asserted as an ordered array so a filter compiled outside the SELECT loop would misalign the binds. CASE WHEN rather than FILTER (WHERE …) because FILTER is Postgres/SQLite-only and this strategy compiles one statement for whichever SQL driver owns the object. BOTH DOORS: run against a real SQLite (sql.js) over the card's ground truth — 24 opportunities, 8 won, 5 lost, won revenue 1,290,000, grand total 5,632,500. The API door now answers 24/8/5/1,290,000, queryDataset answers identically measure-for-measure, ungrouped and grouped; the grouped leg groups by OWNER, not by stage, because grouping on the filter's own column makes filtered and unfiltered aggregates coincide inside the matching group. ABLATION — predicted signature stated FIRST: neutering both halves turns 8 of the 11 new tests red, survivors being the fieldless COUNT(*), the manifest-cube no-op and the fixture self-check, with the lockstep pin staying green (it reads the two tables, not the call site). Observed exactly that: `Tests 8 failed | 12 passed (20)`, including `expected { opp_count: 24, won_count: 24, …(2) } to match object { opp_count: 24, won_count: 8, …(2) }` and `expected 8 to be 24` — the two doors disagreeing again. NOTE: a FIRST ablation pass produced 7, not 8; the survivor was my own grouped test, which grouped by `stage` and so passed with the filter dropped. The test was strengthened (group by owner) and the ablation re-run to meet the prediction — reported because the discrepancy was a real finding about the test, not about the fix. REBUILD, argued from the files: none sits between the edit and the run. Both the ablated module and the test resolve through RELATIVE specifiers inside one package (`../analytics-service.js`, `../strategies/native-sql-strategy.js`), so vitest transforms src/ directly; service-analytics ships no vitest config and the repo has no alias table redirecting them; the only exports-map resolution is @objectstack/spec, untouched here and built before the first measurement. RESTORE proven byte-identical, not assumed: pre b2ac0c23212f1e766c29524bcafb499633f16e0b → ablated a51e178f328ef6626054da443042e025fb2e0f31 → restored b2ac0c23212f1e766c29524bcafb499633f16e0b, and the restore leg re-run green (`Tests 20 passed (20)`). ZERO-GREP DISCIPLINE: the claim that Metric.filters has no consumer rests on a search proven live by a positive control — grepping the same paths for the `format` key returns 5 files while the `filters` key returns none.",
     "open_questions": [],
     "out_of_scope_findings": [
       "filed as #10413: /api/v1/analytics/query still drops per-measure AND dataset-level filter on the ObjectQL path — engine.aggregate receives no filter at all (measured: aggregations [{field:'*',method:'count',alias:'won_count'}] with no filter key), so the two-door disagreement survives on objectqlAggregate drivers with this PR merged; the remedy is either widening the aggregate contract with a per-aggregation filter or a second fan-out inside the strategy, which is a decision this card does not own",
       "filed as #10414 (label `finding`): MetricSchema.filters is an authorable per-metric filter, pinned by the spec's own parse test, and consumed nowhere — a hand-authored cube's filters:[{sql}] parses and is dropped, the same declared-not-enforced shape as this card one level up; enforcing vs retiring it is a packages/spec / ADR-0049 call and this lane holds packages/spec as a zero-ownership boundary"
     ]
   }

   ---

   Generated by Claude Code
7. os-warren commented on Aug 20, 2026

   

   os-warren

   on Aug 20, 2026 – with Claude

   Collaborator

   More actions

   PM 验收：ACCEPT —— PR #10411 翻 ready、入合并队列

   按内容核对 diff，不采信报告自述。

   ① 车道红线守住了，而且是靠论证而非规避。 6 个文件全在 packages/services/service-analytics/ 内（含 changeset），无 packages/spec/**，无 content/docs/releases/**。值得点名的是 DatasetScopedStrategyContext 声明在分析包自己这边，而不是去加宽 spec 已发布的 StrategyContext —— 理由写在类型旁边：这条通道不是可编写面（没有元数据键、没有线上形状、没有错误码），加宽已发布契约买不到任何东西却要付一次 spec 编辑。这是正确的判断，不是绕道。

   ② 扩范围（数据集级 filter）我接受。 它超出卡面点名的两个缺陷，但卡面自己的验收标准是「同一个 cube 上两扇门answer一致」—— 只修 per-measure filter 而留下 definition-level filter，那个标准根本达不到。同一缺陷类、同一扇门、同一个注册表读取。这是补全，不是夹带。

   ③ Clause-②: no 经得起推敲。 端点的接受/拒绝集没变，变的是数值 —— 因为原来的数值是错的。唯一新增的 throw 我单独查过：开发论证它不可达（DatasetMeasure.aggregate 的类型是 AggregationFunction，其每个成员都是条件表的键），并且用 aggregation-lockstep.test.ts 的集合相等断言把这个不可达性钉住了 —— 往一张表加聚合而不加另一张，这里就会红。所以它不构成新的拒绝类别。
   选择裸抛而不是退化成无过滤聚合，我认可开发给的理由：后者「返回 200 但携带与作者声明不同的算术」正是本卡要关掉的缺陷本身。响亮地失败优于安静地答错。

   ④ CASE WHEN 而非 FILTER (WHERE …) 是对的：FILTER 只有 Postgres 和 SQLite ≥3.30 有，MySQL 从来没有，而这个 strategy 为拥有该对象的任意 SQL 驱动编译同一条语句。这是唯一不会「在一个受支持驱动上报语法错、在另一个上返回数字」的形式。

   ⑤ changeset 的姿态是对的。 它以 "Reported figures change, and the new ones are the declared ones" 开头，并且点名了会缩小的那一类（COUNT(col) 数非空值而非行数，因此建立在其上的比率会下降 —— "一个 100% 的磁贴其实是在读自己的分母"）。这是 p1 改动既有读数时应有的写法。

   特别记一笔：消融抓到的是测试弱，不是修复弱

   第一遍消融预测 8 红、实得 7。存活的那条是开发自己写的分组测试 —— 它按 stage 分组，而 stage 正是那些 filter 点名的列，于是「过滤后」与「未过滤」的聚合在匹配组内重合，丢掉 filter 它照样绿。开发把它改成按 owner 分组后重跑，达到预测，并且把这段差异如实报出来而不是悄悄修好。

   这正是消融该起的作用，也是本车道反复强调的那件事：预测在先，然后让不符合预测的地方说话。记进车道条款。

   ---

   一条必须明说的边界：本卡只关了 native-SQL 那扇门

   开发另填了 #10413：ObjectQLStrategy（服务于 objectqlAggregate 而非 nativeSql 的驱动 —— MongoDB、内存驱动等）仍然丢弃 per-measure filter 和 dataset-level filter，engine.aggregate 收到的 options 里根本没有 filter 键。也就是说本卡描述的「两扇门不一致」在那条路径上随 #10411 合并后依然存在，而且那条路径不回显 sql，所以它不可见。

   我判定关闭本卡是正确的，理由是卡面自身的前提就是 SQL 门（标题即 "the SQL is COUNT(*) for all of them"，而 ObjectQL 路径不产生 SQL）。但这个边界必须写明，否则会有人读到「analytics filter 修好了」并以为它在所有驱动上成立。

   ⚠️ 有界声明：#10411 修复的是 NativeSQLStrategy。ObjectQLStrategy 上的同类缺陷未修，见 #10413。

   开发正确地没有顺手扩进去：那边的补救要么加宽 engine.aggregate 的聚合契约（落在 @objectstack/objectql + packages/spec 的 executeAggregate 桥 —— 本车道零所有权），要么在 strategy 内部再做一次扇出（DatasetExecutor.runMeasurePass 上一层已经在做的重复机制）。两条都是决策而非机械修复，本卡不拥有它。

   另填 #10414：MetricSchema.filters 是可编写的 per-metric filter，被 spec 自己的 parse 测试钉住，却零消费者 —— 与本卡同形状的 declared ≠ enforced，高一层。实施还是裁撤是 packages/spec / ADR-0049 的决定，同样不在本车道。

   两张都填得对：报告而非就地修，并且都明确指向红线之外。

   ---

   Generated by Claude Code
8. os-warren

   closed this as completedin #10411on Aug 20, 2026
9. akarma-synetal

   added a commit that references this issue on Sep 1, 2026

   fix(service-analytics): resolve {current_user_id} on the direct analy…

   ...

   74cee59
10. akarma-synetal

    added a commit that references this issue on Sep 17, 2026

    fix(service-analytics): draft preview counts a declared field's non-n…

    ...

    fd5cff2