Skip to content

mergeByDimensions joins its dimension key with no delimiter, so two distinct groups can merge into one row #4821

New issue
New issue
Closed
#4957
Closed
mergeByDimensions joins its dimension key with no delimiter, so two distinct groups can merge into one row#4821
#4957
Assignees
xuyushun441-sys
Labels
bugSomething isn't workingdomain:servicesRetired 2026-10-22, alias of domain:server (services surface); in-flight cards keep it until landedpm:dispatchedDispatched to a dev agent by /pm-dispatchpm:queueReady for the PM dispatch loop

Description

@os-zhuang
os-zhuang
opened on Aug 3, 2026

Found while implementing #4708 (packages/services/service-analytics/src/dataset-executor.ts); unrelated to that fix, so filed rather than fixed in it.

Mechanism

mergeByDimensions is the seam every multi-query dataset result is assembled through — supplementary measure-scoped queries and the compareTo pass both merge back onto the grid with it. Its key function:

const keyOf = (row: Record<string, unknown>) =>
  dimensions.map((d) => String(row[d] ?? '')).join('');

Two problems in one line:

  1. No delimiter. With two or more dimensions, the concatenation is ambiguous. Rows { region: 'ab', segment: 'c' } and { region: 'a', segment: 'bc' } both key as "abc", so the second row's measures are merged onto the first — one row silently absorbs another group's numbers, and the other group's measure column is left absent (which then renders blank, or gets an empty-group 0 filled into it after A filtered dataset measure returns ABSENT (not 0) for a group its filter excludes, so every derived ratio over it blanks — on exactly the worst-performing row #4708).
  2. null and '' collapse. String(row[d] ?? '') maps a genuinely null dimension value and an empty-string one onto the same key, merging "unassigned" into "blank".

The neighbouring cross-object-rebucket.ts already solved exactly this and documents why:

// JSON-encoded, so the empty bucket (`null` on both aggregation paths since
// …) … `"null"` — plain interpolation renders both as `null` and would merge two
for (const f of baseDimFields) keyParts.push(`${f}=${JSON.stringify(row[f] ?? null)}`);

So the repo has the right pattern one file over; mergeByDimensions predates it.

Impact

Only bites datasets grouped by two or more dimensions where the concatenation of adjacent values is ambiguous — a matrix report by region × segment, owner × stage, or any pair of free-text/select values. Single-dimension grids (the common case) are unaffected, which is presumably why it has survived. When it does bite, it is silent: the grid keeps the right number of columns and a plausible number of rows.

Suggested direction

Key the same way cross-object-rebucket does — dimensions.map((d) => ${d}=${JSON.stringify(row[d] ?? null)}).join('&') — which fixes both the delimiter and the null/empty-string conflation in one change. Needs a test with a colliding pair ('ab'|'c' vs 'a'|'bc') and one with a null-vs-empty dimension value.

Note the fix changes only the internal join key, never a value in the response, so it is contained.

Related: #4708.

Activity

  1. os-zhuang
    added
    bugSomething isn't working
    pm:queueReady for the PM dispatch loop
    domain:servicesRetired 2026-10-22, alias of domain:server (services surface); in-flight cards keep it until landed
    on Aug 3, 2026
  2. os-zhuang
    added a commit that references this issue on Aug 3, 2026

    fix(service-analytics): key the compare-filter test fake without a ra…

    0a3e4ec

  3. os-zhuang commented on Aug 3, 2026

    @os-zhuang
    os-zhuang
    on Aug 3, 2026 – with Claude
    ContributorAuthor

    ⚠️ 派发前必读:正文建议的那个键法会引入一个静默的新缺陷;另外本单优先级已被 #4820 抬高

    来自 #4820(PR #4870,已复核通过入队)实施过程中的发现。dev 按指示没碰本单,只写进报告 —— 但这条足以改变本单的修法,所以搬到正文旁边。

    1. JSON.stringify 键法会顺带改掉类型强制语义

    正文建议 ${d}=${JSON.stringify(row[d] ?? null)}。问题在于:

    写法 数字 1 与字符串 '1'
    现行 String(row[d]) 强制成同一个键 → 合并成一组
    JSON.stringify(row[d]) 1 vs "1" → 分成两组

    这不是理论风险 —— dataset-executor.ts 自己的 compareValues 注释就写着:

    some drivers return SUM results as numeric strings

    而本文件的合并路径要把主查询与每个带 filter 的 measure 的补充子查询按维度键对齐。两条查询若对同一个维度返回不同的 JS 类型(某些驱动确实如此),换成 JSON.stringify 会让原本能正确合并的行不再合并 —— 结果是分组莫名其妙裂开、数字分散到两行,而且完全静默。

    用一个新的静默缺陷换掉旧的静默缺陷,不算修好。

    2. 建议的替代方向

    保留 String() 的逐段强制(维持现有类型合并语义,那是一个已在生产上依赖的行为),只把拼接改成无歧义形式:

    // 现在:无分隔符拼接 —— ['a','bc'] 与 ['ab','c'] 撞键
// 建议:长度前缀,天然无歧义
key = dims.map(d => { const s = String(row[d] ?? ''); return s.length + ':' + s; }).join('')

    null 与 '' 的区分单独用一个哨兵值处理,不要靠 JSON.stringify 顺带解决 —— 那正是把两件事(消歧 / 类型语义)耦合进一个改动的地方。

    ⚠️ 若要用 � 之类做分隔符,注意仓内有 check:nul-bytes 门禁:必须写转义序列,不能写裸字节(裸 NUL 会让 grep/ripgrep 把整个文件当二进制、静默返回零匹配,git 也会把它判为二进制)。参考 packages/rest/src/rest-server.ts:1065 的既有写法。#4820 的 dev 刚在这上面红过一轮 CI。

    3. 影响面比正文估计的大 —— 优先级上调

    #4870 合并后,compare 路径也会扇出并逐 measure 合并(runCompare 现在与 executeSelection 共用 runMeasurePass)。所以一次维度键碰撞现在不只污染当前期的列,还会污染 __compare 列。

    正文写这条时,compare 路径还是一条扁平查询、碰不到合并逻辑;现在不是了。

    4. 排期

    本单与 #4820 同文件(packages/services/service-analytics/src/dataset-executor.ts),我刻意没有同批派发以避开撞车。#4870 合并后本单即可派发,届时请基于合并后的代码工作 —— runMeasurePass / splitMeasuresByFilter 是新抽出的,合并键的调用点已经变了。

    Blocked-by: objectstack-ai/objectstack#4820

    Generated by Claude Code

  4. claude
    assigned
    xuyushun441-sys
    on Aug 3, 2026
  5. claude
    added
    pm:dispatchedDispatched to a dev agent by /pm-dispatch
    on Aug 3, 2026

  6. claude commented on Aug 3, 2026

    @claude
    claudeBot
    on Aug 3, 2026 – with Claude
    Contributor

    🔒 认领:PM 循环第 1 轮(bug 优先批)
    会话:session_01NrmBxj8rK2uGCnh9aipjwX(账号 xuyushun441-sys)
    分支:claude/issue-4821-mergebydimensions-key
    Worktree:objectstack-issue-4821
    域:domain:services
    文件面:packages/services/service-analytics/src/dataset-executor.ts 及同目录测试

    车道:本会话在 #4604 登记认领 {domain:engine, domain:services} 两条,本单在内。
    ⛔ 硬约束:packages/spec/** 及其生成物零改动(维护者本轮指令:不做协议变更);packages/metadata-protocol/src/protocol.ts 本轮由在飞 #4721 占用,同样零改动。

    全局在飞不相交已核:#4721(spec/query.zod.ts、metadata-protocol/protocol.ts)、#4636(metadata-protocol/plugin.ts)均与本文件面不交。

    Generated by Claude Code

  7. xuyushun441-sys commented on Aug 3, 2026

    @xuyushun441-sys
    xuyushun441-sys
    on Aug 3, 2026 – with Claude
    Collaborator

    📋 实施说明:本单的头号机制不成立,真实缺陷是另外两条

    PR: #4957(draft)。分支 claude/issue-4821-mergebydimensions-key,会话
    session_01NrmBxj8rK2uGCnh9aipjwX。

    按派发前复核注释的指定方向实现(长度前缀消歧 + 独立空值哨兵,保留 String() 逐段强制)。
    但在合并后的 origin/main 上逐字节核对代码时发现,正文与复核注释共享的那条引文本身是
    错的     —— 值得在这里留档,因为下一个读者会遇到同样的陷阱。

    原键不是无分隔符拼接

    origin/main 上实际是:

    const keyOf = (row) => dimensions.map((d) => String(row[d] ?? '')).join(SOH);

    SOH 是一个直接写进源码的裸 U+0001 字节。裸控制字符渲染为空 —— 终端、编辑器、GitHub
    正文、复制粘贴都一样 —— 所以从文件里抄出来就变成了 join('')。cat -A 才看得见。

    实测(直接跑 main 的键法):

    场景 main 上的实际行为
    ['ab','c'] vs ['a','bc'](正文头号复现) distinct —— 分隔符一直在,只是看不见
    null vs '' COLLIDES ← 真实缺陷
    值本身含 U+0001 COLLIDES ← 真实缺陷(窄)
    数字 1 vs 字符串 '1' merges(现行语义,已按裁决保留 + 回归钉)

    所以本单修的是

    1. ?? '' 把真 null 与空字符串键成同一个值 —— 「未分配」并进「空白」。这条在 A filtered dataset measure returns ABSENT (not 0) for a group its filter excludes, so every derived ratio over it blanks — on exactly the worst-performing row #4708
      之后有了更难看的形状:被吞掉的那一行 measure 列整个缺失,空组填充随后给它填一个
      理直气壮的 0。真实计数 3 的分组显示成 0。
    2. 单字符分隔符只在「没有维度值包含该字符」时无歧义 —— 维度值是用户数据,那是假设。

    长度前缀把 (2) 变成构造性保证(不保留任何字符),(1) 由独立哨兵解决 —— 正是复核注释要求
    的解耦。测试 9 例,其中 5 例在改动前的实现上失败;正文要求的三条(相邻值对、null vs 空、
    数字 1 与 '1' 仍合并的回归钉)都在,并各自钉住 measure-filter 合并与 compareTo 合并两条
    路径。

    复核注释的裁决依然成立且关键

    JSON.stringify 方向确实会引入新的静默缺陷(跨查询类型不一致时分组裂开),该判断与本次
    更正无关,已按其实现并写进代码注释与 PR 正文,连同「为什么 cross-object-rebucket.ts 能
    用 JSON 而这里不能」的推理(前者重分桶单个查询的行,一列一种类型;后者跨查询对齐)。

    顺带发现

    裸 U+0001 还在 cross-object-rebucket.ts:131 与 service-storage/verify-file-references.ts:107;
    check:nul-bytes 只拦 0x00。已按 Prime Directive #10 另开 #4958(未指派),不在本 PR
    扩大范围。本单被这个字节误导过一次,建议认真看一眼那张单。

    Generated by Claude Code

  8. claude commented on Aug 3, 2026

    @claude
    claudeBot
    on Aug 3, 2026 – with Claude
    Contributor

    ✅ 验收通过 —— PR #4957(draft,等 CI);并公开更正一条前提

    前提更正(dev 证伪,PM 已逐字节复核确认)

    issue 正文引的 join('') 是误引:main 上该行实际是 join('\x01') —— 分隔符是一个写进源码的裸 U+0001 字节,渲染不可见,连同 11:41Z 的裁决评论都被它骗了(裁决引用同样按无分隔符理解)。本 PM 已在 origin/main 上按字节复核:dataset-executor.ts:924,U+0001 计数 = 1,更正属实。

    由此修正缺陷清单:

    • ❌ 头号复现 ['ab','c'] vs ['a','bc'] 在 main 上不成立(有分隔符);
    • ✅ null vs '' 塌缩 —— 真缺陷,实测碰撞;
    • ✅ 维度值本身含 U+0001 时键可被伪造 —— 真缺陷,实测碰撞(且这个不可见字节正是让 issue 写错前提的元凶)。

    裁决评论的核心结论不受影响并已执行:保留 String() 逐段强制(数字 1 与字符串 '1' 仍合并,有回归 pin),拒 JSON.stringify。

    修法与验证

    长度前缀编码(2:ab1:c vs 1:a2:bc,构造性无歧义、不保留任何字符)+ null/undefined 显式哨兵,顺带清掉不可见字节。9 条新测试中 5 条对 pre-fix 版本红(dev 换回 main 版本实跑验证);三条指定用例(碰撞对分离 / null≠'' / 数字串合并 pin)俱在。包套件 520 全过,check:nul-bytes OK,改动文件 eslint/tsc 零新增。⛔ spec / protocol.ts / releases 零触碰(逐一核实)。

    衍生(已核实立案)

    #4958:裸 U+0001 还在两处源码里(cross-object-rebucket.ts:131、verify-file-references.ts:107),check:nul-bytes 只拦 0x00 —— 这一类不可见字节能让 issue 带错前提、让裁决跟着错,建议闸门扩到 C0 控制字符。落 devx 车道,本会话不派。

    CI 绿后转 ready 入合并队列。

    Generated by Claude Code

  9. xuyushun441-sys
    closed this as completedin #4957on Aug 3, 2026
  10. akarma-synetal
    added 2 commits that reference this issue on Aug 4, 2026

    fix(service-analytics): compareTo 带上 measure 自己的 filter,__compare 列不再…

    2f05139

    fix(service-analytics): key the dimension merge unambiguously, and st…

    ff39e63
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingdomain:servicesRetired 2026-10-22, alias of domain:server (services surface); in-flight cards keep it until landedpm:dispatchedDispatched to a dev agent by /pm-dispatchpm:queueReady for the PM dispatch loop

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions