Repository navigation
service-messaging: NotificationDispatcher issues 32 statements per tick on an EMPTY outbox (reap runs per partition, twice) — idle cost scales with partitions × table size and never backs off #17610
Description
现象
outbox 完全为空（零 pending）时，NotificationDispatcher 每一跳仍然：
runTick遍历partitionCount个分区（默认 8，dispatcher.ts构造函数partitionCount ?? 8）- 每个分区
runPartition依次调用outbox.claim()与outbox.claimDigest() sql-outbox.ts里这两个方法各自先跑一次 reap（UPDATE … WHERE status='in_flight' AND claimed_at < now - claimTtlMs，代码注释自称 environment-wide），再跑一次候选find，没有候选即返回
⇒ 空闲时每跳 8 × (2 + 2) = 32 条语句。其中 reap 逻辑上每跳至多需要 1 次，现在跑 16 次。
intervalMs 默认 500。inflightTick 守卫会跳过重叠的跳，所以实际速率被单跳耗时封顶——远程 Turso 下每条语句一次 HTTP 往返，生产实测稳态约每秒 28 条，与 32 条 ÷ 约 1.1 秒/跳吻合。
后果（生产实测；租户信息已匿名）
在热表缺索引的租户库上（驱动侧另卡），这 32 条每条都是全表扫描。一个生产租户库在 sys_notification_delivery 约 5,700 行时，稳态约 15 万行读/秒，9 天 1,011.6 亿次读，占其 Turso 组织 99.9%。
即使驱动侧索引修好，空闲成本仍是每跳 32 次索引查找、永不停歇、每个温热 kernel 一份。
验收
- reap 与领取解耦：reap 每跳至多一次（或更慢的独立节奏，例如
claimTtlMs量级），不再随 分区数 × 2 放大 - 空闲退避：连续若干跳零领取时逐步拉长间隔（上限可配）；有投递入队时立即唤醒。若退避会让新通知的首次投递延迟变长，写明上界
- 计数测试防回归：对空 outbox 跑 N 跳，断言 engine 语句数的上限
- 语义不变：分区锁、至少一次投递、digest 折叠、claim TTL 回收——现有测试全绿
同形状，请评估（不强求同卡）
service-queue 的 DbQueueAdapter.claimBatch（默认 1s 轮询）按 priority, scheduled_for 排序，而声明索引是 ['queue','status','scheduled_for']——排序首键 priority 不在索引里；空闲时同样恒定轮询。你若判断应该另立卡，说出来。
驱动缺索引、未配置渠道写死信两张关联卡的编号见本卡评论。
Activity
hotlong commentedon Sep 11, 2026
关联卡：
indexes— every declared secondary index is absent on production Turso tenant databases, so hot polling queries full-scan #17609 —— 驱动侧：remote 模式不建对象级声明索引（本卡的 32 条语句因此每条都是全表扫描）emaildelivery for a tenant with no email transport — it dead-letters on its first attempt and is kept 90d, so the hot outbox grows ~1 dead row per notification #17611 —— 投递表因必死的 email 行持续增长（本卡的每跳成本随之线性上涨）hotlong commentedon Sep 11, 2026
Claim: maintainer direct dispatch (cloud ops session)
Session:
c5c0ce54-bb9c-478c-9e5b-cf44b80d4569Time: 2026-09-11T03:34Z
Branch:
claude/issue-17610-dispatcher-idle-costWorktree:
~/Documents/GitHub/objectstack-issue-17610Domain:
domain:servicesFile surface:
packages/services/service-messaging/src/dispatcher.ts,sql-outbox.ts(+ tests)Clause-②: no — dispatcher scheduling/idle behaviour; delivery semantics unchanged; no
packages/spec/src/**surface expected (path limb re-checked on the actual diff at review)os-litant commentedon Sep 11, 2026
分诊:半状态治愈 —— 补
pm:queue;type 设为Bug。⛔priority:p1与domain:services是维护者所写,本席不动。分诊席 ·
session_017VGfRocA8VjczSe84fgjY3· R+178 · 2026-09-11T03:3xZ · 本评论来自分诊座位. 读数取origin/main@3ef96b47120a989d9333aeeeb6181bc371bc9b52.卡上原有
domain:services+priority:p1而无 pm 状态 ⇒ 落在 ③ 析取,对队列不可见。补上这一半。✅ 车道正确:packages/services/service-messaging在domain:services行。⭐ 维护者点名的那个判断:另立卡了 —— #17612
卡末原话:「
service-queue的DbQueueAdapter.claimBatch(默认 1s 轮询)按priority, scheduled_for排序,而声明索引是['queue','status','scheduled_for']……你若判断应该另立卡,说出来。」判断:另立。 两条读数先复现,⛔ 不转述:
packages/services/service-queue/src/db-queue-adapter.ts:478-487——where: { queue, status: 'pending' },orderBy: [priority asc, scheduled_for asc]packages/platform-objects/src/audit/sys-job-queue.object.ts:199-203——indexes: [['queue','status','scheduled_for'], ['idempotency_key','queue'], ['status']]⇒ ✅ 属实:等值前缀用得上索引,排序首键
priority不在里面 ⇒ 第三列对排序失效,只能走临时 B-tree。另立的三条理由(任一足够):① 不同包、不同文件面、不同 changeset;② 不同修法 —— 本卡修发多少条语句,那卡修一条语句多贵,且那卡的方向本身有分叉(
priority进索引 vs 从排序里拿掉,后者是行为变化);③ ⭐ 那卡的可测影响排在 #17609 之后 —— 你在 #17609 里实测 remote 上sys_job_queue声明的 3 个索引一个都没有,所以「排序首键落空」是 #17609 落地之后才开始有代价。⇒ #17612 已挂pm:blocked+Blocked-by: #17609。⛔ 围栏
indexes— every declared secondary index is absent on production Turso tenant databases, so hot polling queries full-scan #17609 或 service-messaging: fan-out writes anemaildelivery for a tenant with no email transport — it dead-letters on its first attempt and is kept 90d, so the hot outbox grows ~1 dead row per notification #17611。 三件事叠乘,三张卡各修一件。indexes— every declared secondary index is absent on production Turso tenant databases, so hot polling queries full-scan #17609,表的增长归 service-messaging: fan-out writes anemaildelivery for a tenant with no email transport — it dead-letters on its first attempt and is kept 90d, so the hot outbox grows ~1 dead row per notification #17611。查重 —— 2026-09-11T03:3xZ 跑于 581 张 open 板
NotificationDispatchersql-outbox|claimDigestservice-messagingoutbox(控制)⛔ 无重复。 #11453(
pm:on-hold)是INotificationOutbox的取消语义与ack()行为,同一个 outbox、不同的面。#17611(needs-user-decision,p2)是三件事里的第三件(未配置渠道写死信),已在决策箱且标签齐全 ⇒ ⛔ 本席不动。Size/model suggestion:M —— reap 解耦是 S,退避带上界与计数测试是这张卡的主体。分诊席位 ·
session_017VGfRocA8VjczSe84fgjY3· R+178 · 2026-09-11T03:3xZ · 本评论来自分诊座位Generated by Claude Code
os-litant commentedon Sep 11, 2026
⛔ 更正:上一条评论的
pm:queue把你的认领挤掉了约 90 秒,已恢复 —— 本卡现为pm:dispatched,assigneehotlong分诊席 ·
session_017VGfRocA8VjczSe84fgjY3· R+178 · 2026-09-11T03:4xZ · 本评论来自分诊座位发生了什么
本席 03:33Z 取板时,本卡是
domain:services+priority:p1+ 无 pm 状态(③ 半状态)。在我起草上一条评论期间,你认领了它(assigneehotlong,挂上pm:dispatched)。我的写入把pm:queue写了进去并挤掉了pm:dispatched⇒ 一张已认领的卡被打回队列态,约 90 秒。已恢复:pm:dispatched,assignee 未动。⇒ ⛔ 那一行「补
pm:queue」作废,⛔ 不是要你把卡放回队列。 上一条评论的其余内容(#17612 的另立判断与其两条复现读数、四条围栏、查重)全部照常有效。⛔ 这是本席的错,不是你的,成因也说清楚
同一个 fire 里这是第二次:#17609 也在我起草期间被
domain:engine席认领,产生了pm:dispatched+pm:queue并存约 40 秒。⇒ 现在的规则是:写入六态之一时,若卡上已有一个我读取之后才出现的六态,直接抛错、零写入,要求重读后再决定。⛔ 位移别人的写入不再是一个可能的结果。已双向验证。
⭐ 判据是「那不是我的写」:一个并发出现的状态是别的席位刚做的决定,只有他们能移动它。分诊补半状态的授权面只覆盖没有人在动的卡。
分诊席位 ·
session_017VGfRocA8VjczSe84fgjY3· R+178 · 2026-09-11T03:4xZ · 本评论来自分诊座位Generated by Claude Code
hotlong commentedon Sep 11, 2026
os-dev-report
{
"issue": 17610,
"status": "done",
"branch": "claude/issue-17610-dispatcher-idle-cost",
"pr": "#17622",
"premise_still_valid": true,
"summary": "The dispatcher now runs the visibility-timeout reap ONCE per tick, before any claim and outside the partition locks. The new optional INotificationOutbox.reap() and ClaimOptions.skipReap make that possible; before, the reap ran inside both claims of every partition. The loop also backs off exponentially while idle: maxIdleIntervalMs, default 30 s, plugin option dispatchMaxIdleIntervalMs. MessagingService.emit() wakes it through setOutbox(outbox, { onEnqueued }), which calls NotificationDispatcher.wake(). An idle tick went from 32 to 17 statements, and 10 idle minutes from 1,201 to 24 ticks. Premise measured exactly as the card states; no objection. Local emits add no latency; unannounced work (deferred retries, quiet hours, digests, rows from other processes, crashed-node claims) is picked up within maxIdleIntervalMs, stated as the bound in the PR. No packages/spec/src change and no change to what any contract accepts or rejects, so Clause-② stays no. The interface additions are optional, so the changeset is minor and not declared breaking; check-adr-0087-registration rejected my first shape, which made reap a required member. The assignee was already set (hotlong) by the dispatch. check-tenant-audit-census went red on this diff, because the reap consolidation removed one write call site (223 to 222); commit ff0ba83 re-measures the census page and its prose counts.",
"tests": "Premise probe on 3ef96b4 (ObjectQL+SqlDriver, engine calls = knex statements): 32 statements per idle tick, 16 reap UPDATE + 8 claim SELECT + 8 digest SELECT; the same probe after the change reads 17 (1 + 16). At f368fd0: pnpm --filter @objectstack/service-messaging exec vitest run --maxWorkers=2 -> 37 files, 389 tests passed. typecheck -> exit 0; the tsconfig includes the src tests. eslint --no-inline-config --format json over the 11 touched .ts files -> 11 files, 0 errors, 0 warnings; eslint.config.mjs enables no type-aware linting, so no untouched file's verdict can move. turbo build of service-automation^..., then its 3 notify integration files over the rebuilt dist -> 18 tests passed. Ablations at f368fd0, each confirmed on disk by occurrence counts and restored by git checkout HEAD, proven by blob hash plus an empty git diff HEAD: A1, per-claim reap restored -> count leg red (1 failed, 4 passed); A4, skipReap forced with no reap() -> legacy-outbox leg red (1 failed, 4 passed); A2, onEnqueued call removed -> both wake-on-emit legs red, fake-timer and composed-plugin (2 failed, 8 passed); A3, backoff disabled -> 2 backoff legs red (2 failed, 7 passed); restoration leg 15/15 green. Gates: check-adr-0087-registration, check-changeset-no-major and check-empty-changeset, each with its --self-test, all green. dispatch-gates --commands derived 94 families, all run with exit codes recorded. dispatch-gates --ran reconcile: 94 derived, 91 run, 3 NOT-MEASURED, 0 UNRUN (NOT-MEASURED derived from recorded exit 3). check-tenant-audit-census and its self-test exit 0 at ff0ba83 after the census re-measure. NOT MEASURED, as prerequisite refusals: check:i18n, check:type-check-debt, check:dual-build-cjs-loads (exit 3; they need the CLI closure or a full-repo build, which this dispatch forbids), and check:skill-examples (exit 1 with the gate's own refusal: client-react dist has no .d.ts). check:merge-driver exit 1 fails the same 2 check-regen-pending --self-test cases on a clean origin/main 0918c44 checkout on this host, so it is not this diff. os-verify-lock ran UNLOCKED (declared: the host has no flock). CI at report time: in_progress on ff0ba83.",
"mcp_calls": "0 — every GitHub read and write went through the gh CLI",
"open_questions": [
{
"question": "Default idle backoff ceiling (maxIdleIntervalMs / dispatchMaxIdleIntervalMs). It bounds how late the dispatcher notices unannounced work while idle: deferred retries, quiet hours, digests, rows from other processes, and crashed-node claims, which are recovered within claimTtlMs + ceiling.",
"options": [
"A 30 s (shipped): an idle kernel issues 17 statements per 30 s, about 0.57/s against the measured ~28/s",
"B claimTtlMs-sized, 5 s: crash recovery within 2 × TTL, about 2.8 statements/s idle",
"C 10 s: about 1.5 statements/s idle"
],
"recommendation": "A. Emits in the dispatcher's own process wake it, so the ceiling only delays work nobody announced, and 30 s is invisible for retries, digests and quiet hours while cutting idle round trips about 50x. Operators can lower it per plugin."
}
],
"out_of_scope_findings": [
"not filed (per dispatch: the PM files): HttpDispatcher / SqlHttpOutbox in the same plugin has the identical idle shape. A throwaway probe (ObjectQL+SqlDriver, 8 partitions) measured 16 statements per idle tick on sys_http_delivery (8 reap UPDATE + 8 SELECT), on a fixed 500 ms interval with no backoff. The same reap?/skipReap plus backoff/wake-from-enqueueHttp fix applies. A targeted search for 'HttpDispatcher sys_http_delivery' found no open card; the control query 'NotificationDispatcher' hit #17610.",
"DbQueueAdapter evaluation: #17612 (open, p2, Blocked-by #17609) already covers the sort key missing from the index and the constant 1 s poll, so no second card is needed for that. If #17612 wants to reuse this PR's backoff, note that it lives inside NotificationDispatcher (a setTimeout chain doubling to a cap, plus wake on enqueue) and is not extracted as a helper.",
"not filed (for the PM to add to #17612 or file separately): DbQueueAdapter.claimBatch head-of-line starvation, which #17612's body does not mention. The due filter on scheduled_for is applied in JS AFTER LIMIT batchSize×3, in priority-first order. Reproduced with service-queue's own fake engine, which honours orderBy and limit, at batchSize 10: with 30 future-scheduled priority-1 jobs, pollOnce() processes 0 while a due priority-100 job waits until they come due; with 29 (the control) it processes 1.",
"noted, not filed: check:merge-driver (check-regen-pending --self-test, 2 'runner not installed' cases) fails on this macOS host on origin/main 0918c44 as well; it looks host-environmental. Picked up by: none."
]
}
hotlong commentedon Sep 11, 2026
状态（UTC 2026-09-11T04:48Z）：PR #17622 CI 全绿（31 pass · 4 skipping · 0 fail）→ PM 复核通过（空闲退避上限裁定 30s，见 PR 评论）→ 已转 ready → 已开启自动合并，满足条件后自动入队（timeline
auto_merge_enabled）。合并后本卡随Closes关闭；同形状的 HttpDispatcher 见 #17623，DbQueueAdapter 队头饥饿已补到 #17612。hotlong commentedon Sep 11, 2026
已合并（UTC 2026-09-11 05:11:02）：PR #17622 → 合并提交
690f083f8。dispatchMaxIdleIntervalMs可调）HttpDispatcher由 service-messaging: HttpDispatcher issues 16 statements per tick on an EMPTYsys_http_deliveryoutbox on a fixed 500 ms interval — same idle shape as #17610, never backs off #17623 跟进（本 PR 合并解除了它的同文件串行约束）DbQueueAdapter队头饥饿已补到 service-queue:DbQueueAdapter.claimBatch按priority, scheduled_for排序,而sys_job_queue的声明索引不含priority—— 排序首键落空,恒定 1s 轮询每跳吃一次临时 B-tree 排序 #17612fix(service-queue,platform-objects): the claim path's sort keys join …
fix(service-queue,core): the idle job-queue poll backs off, on the on…