diff --git a/CHANGELOG.md b/CHANGELOG.md index 9321ea8..100cee0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,28 @@ 所有显著变更将记录在此文件中。 格式基于 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/),版本遵循 [SemVer](https://semver.org/lang/zh-CN/)。 +## [0.3.1] - 2026-09-05 + +### 修复 +- **npm 包体积事故(发版审核抓出)**:`dist/` 中累积了历次构建的孤儿 chunk(tsup 只清理它认识的文件,模块改名后旧 hash 文件静默残留),0.3.0 及更早版本的包携带了最多 94 个死代码文件。`build` 脚本现先清空 `dist/` 再构建——0.3.1 包 142 → 46 文件(99.5KB)。功能无影响(孤儿文件无人引用),但请从 0.3.1 起使用 + +## [0.3.0] - 2026-09-05 + +### 新增 +- **语义检索(可选,`srelay semantic`)**:换一种说法也能命中——"登录"↔"认证"、"很卡"↔"性能"这类换词查询不再落空 + - 实测依据:12 个真实感技术会话语料上,同义查询 miss 率 33% → **0%**(12/12),字面命中不劣化(5/5) + - 本地 CPU 推理(bge-small-zh-v1.5,Q8 约 35MB),零云端依赖;依赖与模型装在用户缓存目录 `~/.sessionrelay-semantic`,npm 包体积不变 + - `srelay semantic enable` 一条命令:自动装依赖(国内走 npmmirror)+ 模型就绪(`HF_ENDPOINT=https://hf-mirror.com`)+ 存量回填;`disable` / `status` / `test "查询"` 对比 FTS 与融合效果 + - **融合原则:字面优先、语义补充**——FTS 命中永不被替换,语义命中以 `viaSemantic` 标注追加在后(top-5 限量 + 余弦阈值 0.4 可调) + - 检索性能:5000 会话级全量余弦 2.6ms/查询(暴力扫描,无 ANN 依赖);嵌入约 20ms/条,守护每周期限量补嵌不抢 CPU + - 未启用时行为与 0.2.5 完全一致(可选参数注入,MCP 恒 15 工具);schema v3→v4 自动迁移(`session_vectors` 表,启用前恒空) +- 生命周期联动:resume 回滚/归档时删向量(正文变了语义即过期),forget 的 CASCADE 自动清向量,换模型自动全量重嵌 +- `srelay doctor` 新增语义检查项;`srelay status` 面板新增语义状态行 + +### 修复 +- digest 正文拼接改为 SQLite 端限量截断(前 30 条 × 200 字)——大会话不再全量拼接 +- 嵌入失败的目标自动跳过并计数(防毒丸阻塞队列),进程重启后自动重试;enable 回填循环加双轮零产出退出保护 + ## [0.2.5] - 2026-09-03 ### 新增 diff --git a/README.md b/README.md index 5d79a3e..127de20 100644 --- a/README.md +++ b/README.md @@ -118,6 +118,8 @@ stateDiagram-v2 - **历史回填**:`srelay init` 默认回填近 30 天;`srelay sync --backfill all` 一条命令全量入库 ### 🔍 中文检索 +- jieba 分词 + FTS5 双索引(正文 + 元数据),AND 覆盖度语义,OR 兜底 +- **语义检索(可选)**:`srelay semantic enable` 一键开启——换一种说法也能命中("登录"↔"认证"、"很卡"↔"性能"),本地 CPU 推理(bge-small-zh),字面命中优先、语义只做补充,未启用时行为与纯字面检索完全一致 - jieba 分词 + SQLite FTS5 双索引(六条中文验收用例门禁) - 会话级 AND 覆盖 + OR 兜底:连写词拆分("认证方案"→ 认证+方案)、短语精确匹配(`"按月分区"`) - 每条结果**强制携带出处块**(会话 ID / 来源 agent / 日期 / 消息序号 / 摘要片段) @@ -397,7 +399,7 @@ module.exports = { ## 质量与验证 -- **125 个测试**(单元 / 集成 / MCP stdio 真握手契约 / 端到端),`npm test` 一键 +- **185 个测试**(单元 / 集成 / MCP stdio 真握手契约 / 端到端),`npm test` 一键 - **CI 三平台 × Node 22/24 常绿**(typecheck + test + build + dist 冒烟) - TypeScript strict,`npm run typecheck` 零错误 - 每阶段实机验收(含用产品自身记录了自身的诞生过程) diff --git a/docs/design-semantic.md b/docs/design-semantic.md new file mode 100644 index 0000000..b884bc0 --- /dev/null +++ b/docs/design-semantic.md @@ -0,0 +1,206 @@ +# `srelay semantic` 语义检索设计方案 · v4(三轮评审定稿,实现依据) + +> 目标:让"换一种说法"的查询也能命中——AI 检索时换词、用户用自然语言问,都不再错过历史会话。 +> 硬约束(用户拍板):**不影响升级、不影响旧功能**——未启用时行为与 0.2.5 逐字节等价。 +> 演进:v1 起草 → 三轮评审(10 项修订)→ v4。评审记录见 §8。 + +--- + +## 0. 立项依据:miss 率实验(scripts/semantic-miss-experiment.ts,2026-09-03 实测) + +12 个真实感技术会话语料 × 17 组查询(5 对照 + 12 同义): + +| 组 | 命中 | 说明 | +|---|---|---| +| 对照组(原词查询) | **5/5** | 字面检索本身正常,排除假阴性 | +| 同义组(语义等价、换词) | **8/12(miss 33%)** | 「认证失败如何排查」找不到「登录/token」会话;「敏感信息泄露」找不到「硬编码密码」会话 | + +两个致命发现: +1. **miss 本身**:三分之一强的换词查询完全落空; +2. **错配更糟**:「数据库压力大了怎么办」错误命中了「接口延迟」和「密钥泄漏」会话——AI 调用方拿到错误命中会**信以为真**,比空结果危害更大。 + +结论:MCP 检索工具的主要调用方是 AI,AI 换词比人更凶(把用户口语转述成术语)。检索命中率是记忆层产品的第一信任线。 + +## 1. 问题定义 + +| 困境 | 现状 | +|---|---| +| 同义/近义 miss | jieba+FTS5 字面匹配,「登录↔认证」「卡↔性能」「上线↔发布」互不可见 | +| AI 换词检索 miss | AI 把「很卡」转述成「性能劣化」→ 工具空手而归 → AI 弃用工具 | +| 错配 | 字面偶合("数据库")命中无关会话,AI 误信 | +| 长尾口语 | 「转圈」「撑不住」「跑三天被杀」——分词正确但词表不重叠 | + +## 2. 目标与非目标 + +**目标** +1. 同义查询命中:验收标准 = miss 实验的同义 12 组,命中率从 8/12 提升到 **≥11/12(≥92%)**,对照组不劣化 +2. 完全 opt-in:`srelay semantic enable` 前零行为变化、零下载、零新依赖生效 +3. 本地优先:嵌入推理在本机 CPU,模型文件显式下载,默认无网络行为 +4. 不破坏契约:MCP 恒 15 工具、`search_sessions` 返回 schema 仅**新增可选字段** + +**非目标(明确不做)** +- 云端嵌入 API(违反零外呼承诺) +- ANN 索引 / sqlite-vec(量级不需要,见 §3.4 证明;引入原生扩展 = 升级面爆炸) +- 消息级向量(首版会话级;见 §3.3 答辩) +- 拼音/模糊音检索(方针 §5 非目标,维持) +- 自动开启(永远显式 opt-in) + +## 3. 架构设计 + +### 3.1 选型裁决 + +| 决策点 | 裁决 | 理由 | +|---|---|---| +| 模型 | **BAAI/bge-small-zh-v1.5**(ONNX Q8,约 35MB,384 维) | 中文检索小模型事实标准;Q8 量化 CPU 单条嵌入 ~10-30ms | +| 推理 | **@huggingface/transformers(transformers.js v3,onnxruntime-node 后端)** | 纯 npm、无手写原生绑定;Node 22 原生支持 | +| 索引 | **无索引,内存暴力余弦** | 5,000 会话 × 384 维 × 4B = **7.7MB 内存,全量点积 <5ms**;1 万会话内无感。个人项目量级下 ANN 是伪需求 | +| 依赖形态 | **不进 package.json**。`semantic enable` 时动态安装到用户级缓存目录 `~/.sessionrelay/semantic/node_modules`;安装失败(离线)降级为一行指引 | 主包体积保持 124KB;升级路径零变化;onnxruntime 二进制不进主包 | + +### 3.2 数据流总览 + +``` +写入侧(确认时定型,正文不再变): + confirmSession() ──标记──> sessions.confirmed + 守护/CLI 周期 digest ──补嵌──> session_vectors(每周期限量 N=20,CPU 友好) + rollbackSession()(resume 回滚)──> DELETE 向量(正文将变) + runArchive()(正文删除) ──> DELETE 向量(语义已过期,标题/话题仍可 FTS 命中) + forget DELETE sessions ──> FK ON DELETE CASCADE(向量随行消失,无残留) + +读取侧(检索融合): + searchSessions(q) + ├─ 路径 A(现状不动):FTS5 双索引 → 命中集 A(排序不变) + ├─ 路径 B(新增):语义开启 && 向量库非空 && q 非空 + │ → 全量余弦 top-K(cos ≥ τ 默认 0.40,config 可调) + │ → Scope/A 档 WHERE 同样过滤(先 SQL 筛候选 id 再算分) + │ → 命中集 B,仅保留 B−A(FTS 已命中的不重复计) + └─ 融合输出:A 在前(字面精确优先)+ B−A 追加,B 项标注 viaSemantic: true +``` + +**融合原则:字面命中永远优先,语义只做补充发现**——阈值内才出现、排序在 FTS 之后、绝不替换或抑制 FTS 结果。这保证"对照组不劣化"在构造上成立。 + +**生命周期语义(R9)**:嵌入只发生在 confirmed 之后——**记忆层服务的是"过去的会话"**;active 会话的正文还在增长(嵌了就过期),且当下会话本就在上下文里,不是记忆层场景。digest 候选 = `state='confirmed' AND (无向量行 OR model 不匹配)`——imported 会话(直插 confirmed)天然覆盖。 + +**多进程缓存一致性(R1,一轮评审)**:向量集合在进程内 `Map` 缓存,但嵌入发生在守护进程、检索发生在 MCP serve 进程——缓存必须失效检测:每次查询前比对库签名 `SELECT COUNT(*), MAX(embedded_at) FROM session_vectors WHERE model=?`,签名变化才重载(签名查询 <1ms,重载仅在有新向量时)。 + +**运行时降级(R3,二轮评审)**:`enabled=true` 但 transformers.js 不可解析 / 模型文件损坏 / 推理抛错 → 路径 B try/catch 短路为纯 FTS,`semantic status` 与检索 warnings 各提示一次——语义是增强不是依赖,损坏不拖垮检索。 + +### 3.3 嵌入输入与粒度答辩 + +- 输入 = `title + '\n' + 正文前 1200 字符`,**嵌入后 L2 归一化再入库**(R2:bge 余弦阈值的分母必须稳定,0.40 才有跨会话可比性);查询侧同样归一化 +- **输入与查询双侧截断到 512 token**(R6:防超长粘贴打爆编码器) +- **粒度=会话级**:检索目标是"找到会话"(返回值本来就是会话级 hit + snippet),标题+首段已携带主题信号;找到会话后消息定位继续走 FTS(snippet 机制不变) +- 已知局限(诚实登记):超长会话尾部主题稀释——首版接受,若实测命中率不达标,后备方案是"标题+首条用户消息+keyExchanges 拼接"(提取器已有,零新成本) + +### 3.3.1 融合参数(写死,R4) + +| 参数 | 值 | 说明 | +|---|---|---| +| τ(余弦阈值) | 默认 0.40,config 可调 | 真模型实验后可修订,仅改默认值不动接口 | +| K(top-K) | **5,写死** | 语义错配与字面错配同理存在(实验已见字面错配)——限量是防线之一 | +| 排序 | FTS 命中(原序)→ 语义补充(按 cos 降序) | viaSemantic 标注,AI/用户可辨识来源 | + +### 3.4 存储设计(schema v3 → v4) + +```sql +CREATE TABLE session_vectors ( + session_id TEXT PRIMARY KEY REFERENCES sessions(id) ON DELETE CASCADE, + model TEXT NOT NULL, -- 'bge-small-zh-v1.5'(换模型=全部重嵌) + dim INTEGER NOT NULL, -- 384 + vec BLOB NOT NULL, -- Float32Array, little-endian + embedded_at TEXT NOT NULL +); +``` + +- `user_version` 3→4:纯建表迁移,`createDb`/`openExisting` 照既有模式;**未 enable 时表永远为空**——升级零影响的构造性保证 +- FK ON DELETE CASCADE 是与 forget 的联动关键(0.2.5 的 DELETE 直接连带清向量,forget 代码零改动) +- **model 列是向量版本键(R5)**:换模型 = 旧向量视为不存在——查询时 `WHERE model = 当前模型`,digest 候选包含 `model 不匹配` 的行(覆盖重嵌),杜绝跨维度混算 +- 配置存 config.json(非库内):`semantic: { enabled, model, threshold }`——库与配置解耦,`forget --all`(重建空库)不影响语义开关 + +### 3.5 CLI(`srelay semantic`) + +```bash +srelay semantic enable # ①检测/安装 transformers.js 到用户缓存目录 ②下载模型(支持 HF_ENDPOINT 镜像 env) + # ③ config.semantic.enabled=true ④ 触发存量 confirmed 会话回填(后台限量/周期) +srelay semantic status # 开关/模型/向量数/待嵌 backlog/缓存目录/依赖可用性 +srelay semantic disable # enabled=false(向量数据保留;检索立即回退纯 FTS) +srelay semantic test "查询词" # 对比模式:FTS 命中 vs 语义命中 vs 融合结果(透明度 + 调阈值工具) +``` + +嵌入 digest 挂进现有 watch 周期与 `srelay sync` 尾部(复用守护心跳语义,无新进程)。 + +### 3.6 模型获取与离线 + +- 下载缓存走 transformers.js 默认(`~/.cache/huggingface`);国内网络经 `HF_ENDPOINT=https://hf-mirror.com`(enable 输出里写明) +- 下载失败 = enable 失败并回滚 enabled(不留半开状态);已下载后离线可用(纯本地推理) +- 实现期验证点:transformers.js v3 对 HF_ENDPOINT 的尊重方式(若仅认 `env.remoteHost` 则由 CLI 注入) + +### 3.7 并发与性能 + +- 嵌入:单线程串行、每周期限量(默认 20 条/周期 ≈ 守护每 30s 最多 ~600ms CPU)——不与用户争抢 +- 检索:向量集合在进程内 `Map` 惰性载入(首次查询载入 + 嵌入后失效重载);MCP serve 常驻进程天然缓存 +- 乐观策略:向量缺失(未回填完)时该会话只少一路召回,不报错不阻塞——回填进度通过 `semantic status` 透明 + +## 4. 兼容性与文档联动 + +- **未 enable**:`semanticCtx == null` 短路路径 B → 与 0.2.5 行为等价;171 既有测试仅 schema 版本断言需更新(`G1: =3` → `>=3`;`G1b` 模拟未来版本 4→5——这是测试对齐新版本号,不是行为破坏) +- **enable 后**:新增命中是纯增量(B−A 追加),不删不改 FTS 结果 +- MCP:15 工具清单不动;`search_sessions` 响应**新增可选字段** `viaSemantic`(与既有 `viaMeta` 同构) +- `.hop` 导出:不含向量(模型相关、体积大;对端 opt-in 后自行重嵌) +- rebuild:新库无向量 → 周期 digest 自动回填(正确性不依赖向量存在) +- README:核心能力新增「语义检索(可选)」小节 + `semantic enable` 一步指引 +- **doctor(R7)**:新增条件检查项——semantic 未开启时显示"未启用(可选)";开启后检查依赖可解析性/模型文件/向量数/backlog +- **status(R8)**:主面板加一行语义状态(开/关 + 向量数/backlog) + +## 5. 测试计划 + +1. **单元(CI 零模型)**:FakeEmbedder(确定性伪向量:同一文本同向量、相似文本向量相近——手工构造或 hash 扰动)注入融合层;测阈值裁剪/B−A 去重/Scope 过滤/排序保证(A 序不变) +2. **生命周期集成**:confirm→出现向量;resume→删除;archive→删除;forget→CASCADE;rebuild→回填;模型字段不匹配(model 列)→ 视为待嵌 +3. **升级**:v3 库开 v4 表自动建;未 enable 行为等价(旧断言组全绿);G1b 降级拒绝 +4. **契约**:15 工具恒定;响应新增字段向后兼容(旧断言不破) +5. **真模型验收(本地手动,非 CI)**:miss 实验 17 查询重跑——同义组 ≥11/12,对照组 5/5 不劣化;结果回填本文件 §0 +6. **性能冒烟**:5,000 会话伪向量全量余弦 <50ms + +## 6. 不做什么 + +- 云端 API / 自动开启 / AN N 索引 / 消息级向量 / 拼音模糊音 / 向量进 .hop / 多模型并存管理 + +## 7. 开放问题(评审后定或用户裁决) + +1. 阈值 τ 默认值(0.40 起步,真模型实验后定稿;config 可调是否足够?) +2. enable 时自动 `npm install` 到用户缓存目录——可接受度 vs 纯指引(倾向:自动+失败降级指引) +3. bge-small-zh(35MB)vs bge-base-zh(~110MB,精度+3-5 个点)——默认 small,是否提供 `semantic model set`? +4. embedding 输入是否并入 topics/tags(当前裁决:并入 title+正文即可,topics 已在 FTS meta 路) + +## 8. 审查记录 + +### 第一轮(架构与数据流)——3 项修订 +| # | v1 缺陷 | 证据 | v4 修订 | +|---|---|---|---| +| 1 | 向量 Map 进程内缓存 vs 守护进程并发写入——serve 检索到过期向量(staleness) | 多进程架构(serve/守护/sync 三进程共库) | §3.2 R1:COUNT+MAX(embedded_at) 签名失效,变化才重载 | +| 2 | 嵌入未定 L2 归一化——余弦阈值 0.40 的分母随模型输出漂移,阈值失去跨会话可比性 | bge 系列规范用法 | §3.3 R2:入库与查询双侧归一化 | +| 3 | digest 候选未覆盖 imported(不走 confirmSession,直插 confirmed) | insertImportedSession state='confirmed' | §3.2 R9:候选=confirmed 且无向量行(状态判定,非钩子判定);active 不嵌的语义写明 | + +### 第二轮(对抗性与误用)——4 项修订 +| # | v1 缺陷 | 证据 | v4 修订 | +|---|---|---|---| +| 4 | enabled=true 但依赖损坏/模型文件被删 → 检索路径崩溃 | 用户可随意删 ~/.sessionrelay/semantic | §3.2 R3:运行时 try/catch 短路降级 FTS + 提示 | +| 5 | 语义错配无限量防线(字面错配实验已见:'数据库压力'误中密钥会话) | 实验 §0 | §3.3.1 R4:top-K=5 写死 | +| 6 | 换模型后旧维度向量与新维度混算 → 余弦无意义 | dim 列存在但无消费规则 | §3.4 R5:model 列为版本键,查询过滤 + digest 覆盖重嵌 | +| 7 | 超长粘贴(10KB)打爆编码器或拖慢单查询 | bge max_seq 512 | §3.3 R6:双侧截断 | + +### 第三轮(收尾扫描)——3 项修订 +| # | v1 缺陷 | 证据 | v4 修订 | +|---|---|---|---| +| 8 | doctor/status 不感知语义,故障不可发现 | doctor.ts 14 项清单 | R7:doctor 条件项;R8:status 一行 | +| 9 | enable 时自动 npm install 在 Windows 的 .cmd spawn 会 EINVAL | pack-e2e 既有教训(Node ≥22.12 CVE 防护) | 实现注意点:spawn npm 必须 shell:true(R10) | +| 10 | CI 若触发模型下载则断网即红 | transformers.js 惰性下载 | §5 已定:CI 全 FakeEmbedder 零模型;真模型验收为本地手动项并回填 §0 | +--- + +## §9 实现落地备注(实现完成后回填;正文 v4 保持原样,以下为代码事实) + +1. **§0 验收结果(2026-09-05 实测,bge-small-zh-v1.5 Q8)**:对照组 5/5 不劣化;同义组 **12/12**(纯字面 8/12,miss 率 33%→0%),余弦分数区间 0.4-0.75,阈值 0.4 有效过滤。回填脚本 `scripts/semantic-realmodel-verify.ts`(复用 §0 同语料同查询,可复跑)。 +2. **§3.1 缓存目录名修正(实现期抓到的真 bug)**:`~/.sessionrelay/semantic` 会在用户家目录创建 `.sessionrelay` 目录——findRelayRoot 向上探测同名即把**用户主目录误判为已初始化项目**,污染所有子目录 init。已改为 `~/.sessionrelay-semantic`(绝不与项目标记目录重名,代码注释钉死此教训)。 +3. **transformers.js v3 两处实测行为**:pipeline 返回 Tensor(`.data` 取扁平向量,非嵌套数组);`normalized:true` 选项未生效(分数呈点积量级)→ 按 R2 在 embed() 内强制 L2 归一化。bge-small-zh 维度 512(384 是英文版)。 +4. **依赖安装实测**:`npm i --prefix ~/.sessionrelay-semantic @huggingface/transformers@3` 走 npmmirror 14s 装完(含 onnxruntime-node 二进制);模型经 `HF_ENDPOINT=https://hf-mirror.com` 首次下载约 5s、后续缓存命中 0.3s 加载;嵌入均速 ~20ms/条(CPU)。 +5. **top-K 与 limit 的关系**:语义 top-5 在 engine 内追加于 FTS 命中之后,最终统一 `slice(limit)`——FTS 满额时语义被截属预期(语义价值在 FTS 稀结果场景)。 +6. **归档会话的 digest 语义**:软归档删向量后,会话仍为 confirmed 且无向量 → digest 会以纯标题重嵌(标题向量,无害且与 FTS meta 路一致)。 diff --git a/package-lock.json b/package-lock.json index d27209a..a297c6f 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "sessionrelay", - "version": "0.2.5", + "version": "0.3.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "sessionrelay", - "version": "0.2.5", + "version": "0.3.1", "dependencies": { "@inquirer/prompts": "^8.7.0", "@modelcontextprotocol/sdk": "^1.30.0", diff --git a/package.json b/package.json index 1ecbcd5..3195bda 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@ewanjasper/sessionrelay", - "version": "0.2.5", + "version": "0.3.1", "description": "会话接力 SessionRelay — 属于项目、不属于任何厂商的本地记忆层(跨 Agent 会话记忆 / 中文检索 / MCP / HOP 交接协议)", "type": "module", "engines": { @@ -16,7 +16,7 @@ "s3": "vitest run test/unit/state", "s4": "vitest run test/unit/scope", "s5": "vitest run test/integration/capture.spec.ts", - "build": "tsup", + "build": "node -e \"require('fs').rmSync('dist',{recursive:true,force:true})\" && tsup", "prepublishOnly": "npm run typecheck && npm run build && npm test" }, "dependencies": { diff --git a/scripts/semantic-miss-experiment.ts b/scripts/semantic-miss-experiment.ts new file mode 100644 index 0000000..7e2a80e --- /dev/null +++ b/scripts/semantic-miss-experiment.ts @@ -0,0 +1,68 @@ +// 语义检索必要性实验:量化现有字面检索(jieba+FTS5)在同义词查询下的 miss 率 +// 用法:npx tsx scripts/semantic-miss-experiment.ts +// 语料原则:模拟真实技术对话的自然表述(对话里说什么词,就只含什么词) +import { createDb, insertSession, insertMessage } from '../src/store/db.js'; +import { searchSessions } from '../src/search-svc/engine.js'; + +const db = createDb(); +const PID = 'proj-experiment'; + +// ── 语料:12 个会话,对话内容只含左列表述(自然、克制,不堆关键词) ── +const corpus: Array<{ id: string; title: string; msgs: string[] }> = [ + { id: 'auth001', title: '登录问题排查', msgs: ['用户反馈登录一直转圈', '看了下是 token 过期后前端没有刷新', '加了个静默续期就好了'] }, + { id: 'perf001', title: '接口延迟治理', msgs: ['列表接口要 3 秒才返回', '慢在 N+1 查询,循环里逐条查了数据库', '改成批量 IN 之后降到 80ms'] }, + { id: 'dep001', title: '发布流程整理', msgs: ['每次上线都是手动跑脚本容易出错', '整理成 CI 流水线:构建、跑测试、再部署到 K8s', '以后合并到 main 就自动上线'] }, + { id: 'db001', title: '存储选型讨论', msgs: ['订单量上来后 MySQL 单表撑不住', '评估了分库分表和 TiDB', '最后选了按租户分片的方案'] }, + { id: 'mem001', title: '内存泄漏排查', msgs: ['服务跑三天 RSS 涨到 4G 被 OOMKill', 'heapdump 看到大量未释放的定时器', '修复后曲线平了'] }, + { id: 'refac001', title: '模块解耦', msgs: ['订单模块直接 import 了支付模块的内部函数', '耦合太深改一处崩三处', '抽了个接口层做依赖倒置'] }, + { id: 'ui001', title: '首页白屏修复', msgs: ['低版本浏览器打开首页直接白屏', '是可选链语法没转译', '补了 babel target 配置'] }, + { id: 'sec001', title: '密钥泄漏事故', msgs: ['发现代码库里硬编码了数据库密码还提交到了仓库', '全部改成环境变量注入', '历史提交里的也用 filter-branch 清掉了'] }, + { id: 'cache001', title: '缓存命中率提升', msgs: ['Redis 命中率只有 40%', '热点 key 加了本地 LRU 二级缓存', '命中率到 92%,回源少了大半'] }, + { id: 'test001', title: '回归测试补齐', msgs: ['改个小 bug 手工点一遍太费时间', '给下单主链路补了自动化用例', '现在合并前自动跑'] }, + { id: 'log001', title: '日志规范化', msgs: ['各服务日志格式五花八门没法查', '统一了 JSON 结构化输出和 trace id', '现在能按请求串起全链路'] }, + { id: 'mq001', title: '消息积压处理', msgs: ['大促时 MQ 消息堆了几百万条', '消费者改成批量拉取并发处理', '加了死信队列兜底'] }, +]; + +for (const c of corpus) { + insertSession(db, { id: c.id, source: 'zcode', sourceSessionId: c.id, projectId: PID, createdAt: '2026-08-20T08:00:00Z', title: c.title, topics: [] }); + c.msgs.forEach((m, i) => insertMessage(db, { sessionId: c.id, role: i % 2 ? 'assistant' : 'user', content: m, seqNum: i + 1, createdAt: '2026-08-20T08:00:00Z' })); + db.prepare('UPDATE sessions SET message_count = ? WHERE id = ?').run(c.msgs.length, c.id); +} + +// ── 查询组:对照(原词,验证检索本身工作正常)与同义(用户/AI 换一种说法) ── +const queries: Array<{ q: string; expect: string; kind: 'control' | 'synonym' }> = [ + { q: '登录 转圈', expect: 'auth001', kind: 'control' }, + { q: 'N+1 批量', expect: 'perf001', kind: 'control' }, + { q: '白屏', expect: 'ui001', kind: 'control' }, + { q: '硬编码 密码', expect: 'sec001', kind: 'control' }, + { q: '死信队列', expect: 'mq001', kind: 'control' }, + // 同义组:语义等价、字面零重叠 + { q: '认证失败如何排查', expect: 'auth001', kind: 'synonym' }, + { q: '接口很慢怎么优化', expect: 'perf001', kind: 'synonym' }, + { q: '部署流程', expect: 'dep001', kind: 'synonym' }, + { q: '数据库压力大了怎么办', expect: 'db001', kind: 'synonym' }, + { q: '服务内存涨上去被杀', expect: 'mem001', kind: 'synonym' }, + { q: '代码耦合想重构', expect: 'refac001', kind: 'synonym' }, + { q: '页面加载不出来', expect: 'ui001', kind: 'synonym' }, + { q: '敏感信息泄露', expect: 'sec001', kind: 'synonym' }, + { q: '回源太多想加缓存', expect: 'cache001', kind: 'synonym' }, + { q: '自动化测试', expect: 'test001', kind: 'synonym' }, + { q: '排查问题缺少链路信息', expect: 'log001', kind: 'synonym' }, + { q: '消息堆积消费不过来', expect: 'mq001', kind: 'synonym' }, +]; + +console.log('kind\tquery\t\t\t期望\t命中\t结果'); +let ctlHit = 0, ctlTotal = 0, synHit = 0, synTotal = 0; +const misses: string[] = []; +for (const { q, expect, kind } of queries) { + const hits = searchSessions(db, { project: PID, query: q, limit: 5 }); + const ok = hits.some((h) => h.sessionId === expect); + if (kind === 'control') { ctlTotal++; if (ok) ctlHit++; } + else { synTotal++; if (ok) { synHit++; } else misses.push(`${q} ✗ (期望 ${expect})`); } + console.log(`${kind === 'control' ? '对照' : '同义'}\t${q.padEnd(16)}\t${expect}\t${hits.map((h) => h.sessionId).join(',') || '∅'}\t${ok ? '✓' : '✗'}`); +} +console.log('─'.repeat(60)); +console.log(`对照组:${ctlHit}/${ctlTotal} 命中(验证字面检索本身正常)`); +console.log(`同义组:${synHit}/${synTotal} 命中 —— miss 率 ${(((synTotal - synHit) / synTotal) * 100).toFixed(0)}%`); +if (misses.length) console.log(`未命中的同义查询:\n ${misses.join('\n ')}`); +db.close(); diff --git a/scripts/semantic-realmodel-verify.ts b/scripts/semantic-realmodel-verify.ts new file mode 100644 index 0000000..6342874 --- /dev/null +++ b/scripts/semantic-realmodel-verify.ts @@ -0,0 +1,79 @@ +// 真模型本地验收(design-semantic §5.5,非 CI 项):与 miss 实验同语料同查询,走真 bge-small-zh +// 前置:npm i --prefix ~/.sessionrelay/semantic @huggingface/transformers@3 +// 国内模型下载:HF_ENDPOINT=https://hf-mirror.com npx tsx scripts/semantic-realmodel-verify.ts +import { createDb, insertSession, insertMessage } from '../src/store/db.js'; +import { searchSessions } from '../src/search-svc/engine.js'; +import { createTransformersEmbedder, semanticSearch, digestSemantic, semanticInputOf } from '../src/search-svc/semantic.js'; +import { l2 } from '../src/search-svc/semantic.js'; +import type { RelayConfig } from '../src/shared/config.js'; +import { defaultConfig } from '../src/shared/config.js'; + +const PID = 'proj-experiment'; +const corpus: Array<{ id: string; title: string; msgs: string[] }> = [ + { id: 'auth001', title: '登录问题排查', msgs: ['用户反馈登录一直转圈', '看了下是 token 过期后前端没有刷新', '加了个静默续期就好了'] }, + { id: 'perf001', title: '接口延迟治理', msgs: ['列表接口要 3 秒才返回', '慢在 N+1 查询,循环里逐条查了数据库', '改成批量 IN 之后降到 80ms'] }, + { id: 'dep001', title: '发布流程整理', msgs: ['每次上线都是手动跑脚本容易出错', '整理成 CI 流水线:构建、跑测试、再部署到 K8s', '以后合并到 main 就自动上线'] }, + { id: 'db001', title: '存储选型讨论', msgs: ['订单量上来后 MySQL 单表撑不住', '评估了分库分表和 TiDB', '最后选了按租户分片的方案'] }, + { id: 'mem001', title: '内存泄漏排查', msgs: ['服务跑三天 RSS 涨到 4G 被 OOMKill', 'heapdump 看到大量未释放的定时器', '修复后曲线平了'] }, + { id: 'refac001', title: '模块解耦', msgs: ['订单模块直接 import 了支付模块的内部函数', '耦合太深改一处崩三处', '抽了个接口层做依赖倒置'] }, + { id: 'ui001', title: '首页白屏修复', msgs: ['低版本浏览器打开首页直接白屏', '是可选链语法没转译', '补了 babel target 配置'] }, + { id: 'sec001', title: '密钥泄漏事故', msgs: ['发现代码库里硬编码了数据库密码还提交到了仓库', '全部改成环境变量注入', '历史提交里的也用 filter-branch 清掉了'] }, + { id: 'cache001', title: '缓存命中率提升', msgs: ['Redis 命中率只有 40%', '热点 key 加了本地 LRU 二级缓存', '命中率到 92%,回源少了大半'] }, + { id: 'test001', title: '回归测试补齐', msgs: ['改个小 bug 手工点一遍太费时间', '给下单主链路补了自动化用例', '现在合并前自动跑'] }, + { id: 'log001', title: '日志规范化', msgs: ['各服务日志格式五花八门没法查', '统一了 JSON 结构化输出和 trace id', '现在能按请求串起全链路'] }, + { id: 'mq001', title: '消息积压处理', msgs: ['大促时 MQ 消息堆了几百万条', '消费者改成批量拉取并发处理', '加了死信队列兜底'] }, +]; +const queries: Array<{ q: string; expect: string; kind: 'control' | 'synonym' }> = [ + { q: '登录 转圈', expect: 'auth001', kind: 'control' }, + { q: 'N+1 批量', expect: 'perf001', kind: 'control' }, + { q: '白屏', expect: 'ui001', kind: 'control' }, + { q: '硬编码 密码', expect: 'sec001', kind: 'control' }, + { q: '死信队列', expect: 'mq001', kind: 'control' }, + { q: '认证失败如何排查', expect: 'auth001', kind: 'synonym' }, + { q: '接口很慢怎么优化', expect: 'perf001', kind: 'synonym' }, + { q: '部署流程', expect: 'dep001', kind: 'synonym' }, + { q: '数据库压力大了怎么办', expect: 'db001', kind: 'synonym' }, + { q: '服务内存涨上去被杀', expect: 'mem001', kind: 'synonym' }, + { q: '代码耦合想重构', expect: 'refac001', kind: 'synonym' }, + { q: '页面加载不出来', expect: 'ui001', kind: 'synonym' }, + { q: '敏感信息泄露', expect: 'sec001', kind: 'synonym' }, + { q: '回源太多想加缓存', expect: 'cache001', kind: 'synonym' }, + { q: '自动化测试', expect: 'test001', kind: 'synonym' }, + { q: '排查问题缺少链路信息', expect: 'log001', kind: 'synonym' }, + { q: '消息堆积消费不过来', expect: 'mq001', kind: 'synonym' }, +]; + +const db = createDb(); +for (const c of corpus) { + insertSession(db, { id: c.id, source: 'zcode', sourceSessionId: c.id, projectId: PID, createdAt: '2026-08-20T08:00:00Z', title: c.title, state: 'confirmed' }); + c.msgs.forEach((m, i) => insertMessage(db, { sessionId: c.id, role: i % 2 ? 'assistant' : 'user', content: m, seqNum: i + 1 })); +} +const cfg: RelayConfig = { ...defaultConfig(), identity: { project_id: PID }, semantic: { enabled: true, model: 'Xenova/bge-small-zh-v1.5', threshold: 0.4 } }; + +console.log('加载模型(首次运行触发下载,国内走 hf-mirror)...'); +const t0 = Date.now(); +const embedder = await createTransformersEmbedder('Xenova/bge-small-zh-v1.5'); +console.log(`模型就绪(${((Date.now() - t0) / 1000).toFixed(1)}s),维度验证:embed("测试").length = ${(await embedder.embed('测试')).length}`); +void l2; void semanticInputOf; + +const t1 = Date.now(); +const n = (await digestSemantic(db, cfg, { projectId: PID, limit: 100 })).embedded; +console.log(`回填 ${n} 会话(${((Date.now() - t1) / 1000).toFixed(1)}s,均 ${(((Date.now() - t1) / n)).toFixed(0)}ms/条)`); + +console.log('\nkind\tquery\t\t\t期望\t融合命中(语义分)\t结果'); +let ctlHit = 0, ctlTotal = 0, synHit = 0, synTotal = 0; +const synFix: string[] = []; +for (const { q, expect, kind } of queries) { + const sem = await semanticSearch(db, cfg, q, { project: PID }); + const merged = searchSessions(db, { project: PID, query: q, limit: 5, semanticHits: sem }); + const ok = merged.some((h) => h.sessionId === expect); + const semMark = sem ? sem.map((s) => `${s.sessionId}(${s.score.toFixed(2)})`).join(' ') : 'null'; + if (kind === 'control') { ctlTotal++; if (ok) ctlHit++; } + else { synTotal++; if (ok) synHit++; else synFix.push(q); } + console.log(`${kind === 'control' ? '对照' : '同义'}\t${q.padEnd(16)}\t${expect}\t${semMark}\t${ok ? '✓' : '✗'}`); +} +console.log('─'.repeat(60)); +console.log(`对照组:${ctlHit}/${ctlTotal}(不劣化门禁)`); +console.log(`同义组:${synHit}/${synTotal}(验收线 ≥11/12,纯字面为 8/12)`); +if (synFix.length) console.log(`仍未命中:${synFix.join('、')}`); +db.close(); diff --git a/src/bin/srelay.ts b/src/bin/srelay.ts index a5dbe18..a703920 100644 --- a/src/bin/srelay.ts +++ b/src/bin/srelay.ts @@ -65,6 +65,15 @@ program .option('--json', '机器格式') .action(async (id: string | undefined, f) => { const { cmdForget } = await import('../cli/forget.js'); await cmdForget(f, id); }); +program + .command('semantic') + .description('语义检索(可选):换一种说法也能命中历史会话——本地 CPU 推理,显式启用') + .option('--enable', '启用:安装依赖(用户缓存目录)+ 模型就绪 + 存量回填') + .option('--disable', '停用(向量数据保留,检索立即回退纯字面匹配)') + .option('--status', '状态(默认)') + .option('--test ', '对比模式:FTS-only vs 融合命中') + .action(async (f) => { const { cmdSemantic } = await import('../cli/semantic.js'); await cmdSemantic(f); }); + program .command('status') .description('透明度面板:模式/守护/计数/拦截/体积') diff --git a/src/capture/archive.ts b/src/capture/archive.ts index 1f8d1fb..11f7c47 100644 --- a/src/capture/archive.ts +++ b/src/capture/archive.ts @@ -118,13 +118,15 @@ export function runArchive(db: DB, opts: ArchiveOptions): ArchiveResult { // 执行归档 if (opts.hard) { - // 硬删除:删除 sessions 行(messages 级联删除) + // 硬删除:删除 sessions 行(messages/向量级联删除) db.prepare('DELETE FROM sessions WHERE id = ?').run(s.id); } else { // 归档:删除 messages,保留 sessions 行 db.prepare('DELETE FROM messages WHERE session_id = ?').run(s.id); db.prepare('UPDATE sessions SET cleanup_at = ?, message_count = 0, original_message_count = ? WHERE id = ?') .run(now.toISOString(), msgCount, s.id); + // 语义联动(design-semantic §3.2):正文删除=向量语义过期;标题/话题仍走 FTS meta 路 + db.prepare('DELETE FROM session_vectors WHERE session_id = ?').run(s.id); } // 记录审计明细 diff --git a/src/capture/watch.ts b/src/capture/watch.ts index 971480f..b641766 100644 --- a/src/capture/watch.ts +++ b/src/capture/watch.ts @@ -41,6 +41,9 @@ export async function runWatch(opts: WatchOptions): Promise { if (s.newMessages > 0 || s.resumed > 0 || j.confirmed > 0 || spool.endSignals > 0 || why !== 'tick') { log(`${why}: +${s.newMessages} 消息 · resumed ${s.resumed} · pending ${j.toPending} · confirmed ${j.confirmed}${spool.endSignals ? ` · hook信号 ${spool.endSignals}` : ''}`); } + // 语义 digest(design-semantic §3.2):confirmed 且无向量的会话限量补嵌,CPU 友好 + const { digestSemantic } = await import('../search-svc/semantic.js'); + await digestSemantic(db, opts.config, { projectId, limit: 20 }); } catch (e) { log(`周期失败(不退出): ${(e as Error).message}`); } diff --git a/src/cli/doctor.ts b/src/cli/doctor.ts index 6416b06..b6626bc 100644 --- a/src/cli/doctor.ts +++ b/src/cli/doctor.ts @@ -140,6 +140,20 @@ export async function cmdDoctor(): Promise { } } + // 语义检索(R7 条件项:未启用=可选提示;启用后查依赖,回填进度在 srelay semantic status) + if (root) { + const cfg = loadConfig(root); + if (cfg.semantic?.enabled === true) { + const { resolveTransformersEntry, semanticModelOf } = await import('../search-svc/semantic.js'); + const entry = resolveTransformersEntry(); + checks.push(entry + ? c('语义检索', 'ok', `已启用(${semanticModelOf(cfg)})`) + : c('语义检索', 'err', '已启用但 transformers.js 缺失(检索已自动降级纯字面)', 'npm i --prefix ~/.sessionrelay/semantic @huggingface/transformers@3')); + } else { + checks.push(c('语义检索', 'warn', '未启用(可选增强:换词查询也能命中,srelay semantic enable)')); + } + } + for (const k of checks) { const icon = k.level === 'ok' ? pc.green('✅') : k.level === 'warn' ? pc.yellow('⚠️ ') : pc.red('❌'); console.log(`${icon} ${k.name.padEnd(18)} ${k.detail}${k.fix ? pc.dim(` → ${k.fix}`) : ''}`); diff --git a/src/cli/query.ts b/src/cli/query.ts index 6c57f4c..da70135 100644 --- a/src/cli/query.ts +++ b/src/cli/query.ts @@ -28,11 +28,15 @@ export async function cmdSearch(query: string, flags: SearchFlags): Promise { @@ -41,6 +45,7 @@ export async function cmdSearch(query: string, flags: SearchFlags): Promise { + const root = findRelayRoot(process.cwd()); + if (!root) die('未找到 .sessionrelay(本项目尚未初始化)', '在项目根目录运行 srelay init'); + const cfg = loadConfig(root); + + // ── enable ── + if (f.enable) { + if (cfg.semantic?.enabled === true) { console.log(pc.dim('语义检索已启用(无需重复 enable)')); return; } + console.log(pc.cyan('⚡ 启用语义检索(本地推理,完全可选)')); + // ① 依赖:已装跳过;未装自动安装;失败给指引退出(不留半开状态) + if (!resolveTransformersEntry()) { + const ok = installTransformers(); + if (!ok) { + die( + 'transformers.js 安装失败(网络或权限)', + `手动安装:npm i --prefix ~/.sessionrelay/semantic @huggingface/transformers@3 后重试;国内模型下载可设 HF_ENDPOINT=https://hf-mirror.com`, + ); + } + } else { + console.log(pc.dim(' transformers.js 已就绪')); + } + // ② 写配置(模型就绪与否不阻塞开关——检索路径自带 R3 降级) + const semantic = { ...(cfg.semantic ?? {}), enabled: true, model: cfg.semantic?.model ?? DEFAULT_MODEL, threshold: cfg.semantic?.threshold ?? DEFAULT_THRESHOLD }; + saveConfig(root, { ...cfg, semantic }); + resetSemanticCaches(); + // ③ 模型就绪检查 + 存量回填(Fake 环境跳过模型;真模型首次会触发下载) + const embedder = await getEmbedder(loadConfig(root)); + if (!embedder) { + console.log(pc.yellow(' ⚠️ 模型暂不可用(首次运行需下载,或离线)——开关已保存,检索暂走纯 FTS,模型就绪后自动生效')); + console.log(pc.dim(' 提示:HF_ENDPOINT=https://hf-mirror.com srelay sync 触发下载(国内镜像)')); + return; + } + console.log(pc.dim(` 模型 ${embedder.model} 就绪,开始回填存量 confirmed 会话...`)); + const db = openExisting(dbFile(root)); + try { + let total = 0, failedTotal = 0, idleRounds = 0; + for (;;) { + const { embedded, failed } = await digestSemantic(db, loadConfig(root), { projectId: cfg.identity.project_id, limit: 50 }); + total += embedded; + failedTotal += failed; + if (embedded === 0) { + // 连续两轮零产出(毒丸全部失败或全部完成)即收——不留死循环 + if (++idleRounds >= 2) break; + } else { + idleRounds = 0; + } + process.stdout.write(`\r 已嵌入 ${total} 个会话`); + } + console.log(''); + const backlog = countSemanticBacklog(db, embedder.model); + console.log(pc.green('✓') + ` 语义检索已启用:${total} 向量 · 待嵌 ${backlog}` + (failedTotal > 0 ? pc.yellow(` · ${failedTotal} 个失败已跳过(下个 sync/守护周期或重启进程自动重试)`) : '')); + console.log(pc.dim(' 验证:srelay semantic test "换一种说法的查询"')); + } finally { db.close(); } + return; + } + + // ── disable ── + if (f.disable) { + if (cfg.semantic?.enabled !== true) { console.log(pc.dim('语义检索本就未启用')); return; } + saveConfig(root, { ...cfg, semantic: { ...cfg.semantic, enabled: false } }); + resetSemanticCaches(); + console.log(pc.green('✓') + ' 已停用(向量数据保留;重新 enable 无需重新回填)'); + return; + } + + // ── test:FTS vs 语义 vs 融合 对比(透明度与调参工具) ── + if (f.test !== undefined) { + const db = openExisting(dbFile(root)); + try { + const project = cfg.identity.project_id ?? root; + const ftsOnly = searchSessions(db, { project, query: f.test, limit: 10 }); + const sem = await semanticSearch(db, cfg, f.test, { project }); + const merged = searchSessions(db, { project, query: f.test, limit: 10, semanticHits: sem }); + const line = (h: typeof ftsOnly[number]) => + ` ${h.sessionId} 「${(h.snippet || '').slice(0, 30)}」 ${h.viaSemantic ? pc.cyan('[语义]') : h.viaMeta ? pc.dim('[meta]') : '[FTS]'}`; + console.log(pc.cyan(`查询:「${f.test}」`)); + console.log(`FTS-only:${ftsOnly.length} 命中`); + ftsOnly.forEach(line); + console.log(`融合(FTS + 语义${sem === null ? ',语义未启用/不可用' : ''}):${merged.length} 命中`); + merged.forEach(line); + if (sem !== null && sem.length > 0) console.log(pc.dim(` 语义 top 分数:${sem.map((s) => s.score.toFixed(3)).join(', ')}(阈值 ${cfg.semantic?.threshold ?? DEFAULT_THRESHOLD},config.semantic.threshold 可调)`)); + } finally { db.close(); } + return; + } + + // ── status(默认) ── + const db = fs.existsSync(dbFile(root)) ? openExisting(dbFile(root)) : null; + try { + const enabled = cfg.semantic?.enabled === true; + const model = cfg.semantic?.model ?? DEFAULT_MODEL; + console.log(`语义检索 ${enabled ? pc.green('已启用') : pc.dim('未启用(可选增强,srelay semantic enable)')}`); + if (enabled) { + console.log(` 模型 ${model}`); + console.log(` 依赖 ${resolveTransformersEntry() ? pc.green('transformers.js 可用') : pc.yellow('缺失(检索自动降级纯 FTS;重装见 srelay semantic enable)')}`); + if (db) { + const sig = vectorSignature(db, model); + const backlog = countSemanticBacklog(db, model); + console.log(` 向量 ${sig.split(':')[0]} 个 · 待嵌 ${backlog}${backlog > 0 ? pc.dim('(守护/sync 周期自动补嵌)') : ''}`); + } + console.log(` 阈值 ${cfg.semantic?.threshold ?? DEFAULT_THRESHOLD}(config.semantic.threshold)`); + } + console.log(pc.dim(` 缓存目录 ${semanticCacheDir().replace(/\\/g, '/')}(依赖与模型,可整目录删除)`)); + } finally { db?.close(); } +} diff --git a/src/cli/status.ts b/src/cli/status.ts index c57ba47..c0e68f7 100644 --- a/src/cli/status.ts +++ b/src/cli/status.ts @@ -56,6 +56,14 @@ export async function cmdStatus(opts?: { json?: boolean }): Promise { console.log(`会话 ${stText}${noteAdj}`); console.log(`来源 ${Object.entries(convSources).map(([k, v]) => `${k} ${v}`).join(' · ') || pc.dim('(空)')}`); console.log(`拦截 ignore 规则累计拦截 ${blocked} 次${cfg.capture.mode !== 'full' ? pc.yellow(` · 当前模式 ${cfg.capture.mode}(不落正文)`) : ''}`); + // R8:语义一行(未启用也显示,保持可发现性) + if (cfg.semantic?.enabled === true) { + const model = cfg.semantic?.model ?? 'Xenova/bge-small-zh-v1.5'; + const vecN = (db.prepare('SELECT COUNT(*) n FROM session_vectors WHERE model = ?').get(model) as { n: number }).n; + console.log(`语义 ${pc.green('开')} · ${vecN} 向量${pc.dim('(srelay semantic status 详情)')}`); + } else { + console.log(`语义 ${pc.dim('关(可选:srelay semantic enable)')}`); + } console.log(`体积 relay.sqlite ${(size / 1024 / 1024).toFixed(1)} MB`); if (recent.length > 0) { console.log('最近'); diff --git a/src/cli/sync.ts b/src/cli/sync.ts index 35e4da8..533714d 100644 --- a/src/cli/sync.ts +++ b/src/cli/sync.ts @@ -25,6 +25,10 @@ export async function cmdSync(opts: { backfill?: string; json?: boolean }): Prom if (opts.json) { console.log(JSON.stringify({ ...s, judge: j }, null, 2)); return; } console.log(`同步完成:发现 ${s.discovered} · 新会话 ${s.newSessions} · 新消息 ${s.newMessages} · resumed ${s.resumed}${s.blocked ? pc.yellow(` · 拦截 ${s.blocked}`) : ''}`); if (j.toPending || j.confirmed) console.log(pc.dim(`判定:${j.toPending} 转 pending · ${j.confirmed} 确认`)); + // 语义 digest(守护不在时的补嵌路径;未启用时零开销短路) + const { digestSemantic } = await import('../search-svc/semantic.js'); + const { embedded } = await digestSemantic(db, cfg, { projectId: cfg.identity.project_id, limit: 100 }); + if (embedded > 0) console.log(pc.dim(`语义:补嵌 ${embedded} 个会话`)); } finally { db.close(); } diff --git a/src/mcp/server.ts b/src/mcp/server.ts index 602f312..f6b5522 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -80,12 +80,16 @@ export function buildServer(root: string, db: DB, cfg: RelayConfig): McpServer { const callPred = predFrom(args); // T28:每次调用重新装配(scope.json 热更新) const asm = assembleScope({ root, cfg, callPred, includeAuto: true }); - const hits = searchSessions(db, { project, query: args.query, limit: args.limit ?? 10, extraWhere: asm.where }); + // 语义补充命中(design-semantic:未启用返回 null = 纯 FTS,行为与 0.2.5 等价) + const { semanticSearch } = await import('../search-svc/semantic.js'); + const semHits = await semanticSearch(db, cfg, args.query, { project, extraWhere: asm.where }); + const hits = searchSessions(db, { project, query: args.query, limit: args.limit ?? 10, extraWhere: asm.where, semanticHits: semHits }); const out = hits.map((h) => ({ ...sessionBrief(db, h.sessionId), score: Number(h.score.toFixed(3)), coverage: h.coverage, viaMeta: h.viaMeta, + viaSemantic: h.viaSemantic ?? false, snippet: h.snippet, provenance: { ...sessionBrief(db, h.sessionId), msg: h.seq, note: `get_session_detail 可取全文` }, })); diff --git a/src/search-svc/engine.ts b/src/search-svc/engine.ts index 9576867..229902a 100644 --- a/src/search-svc/engine.ts +++ b/src/search-svc/engine.ts @@ -19,6 +19,7 @@ export interface SearchHit { viaMeta: boolean; coverage: number; // 覆盖的单元数 / 总单元数 seq: number; // 最佳命中消息序号(出处块的 msg#,D10) + viaSemantic?: boolean; // 语义补充命中(design-semantic §3.2:B−A 追加,排序在 FTS 之后) } interface UnitHits { @@ -68,6 +69,8 @@ export interface SearchOptions { query: string; limit?: number; extraWhere?: ExtraWhere | null; + /** 语义补充命中(调用方经 semanticSearch 预计算注入;不传/空 = 与纯 FTS 行为等价) */ + semanticHits?: Array<{ sessionId: string; score: number }> | null; } export function searchSessions(db: DB, opts: SearchOptions): SearchHit[] { @@ -130,5 +133,15 @@ export function searchSessions(db: DB, opts: SearchOptions): SearchHit[] { // 兜底模式下覆盖度高的排前(有更多关键词命中的更相关) hits.sort((a, b) => b.coverage - a.coverage || b.score - a.score); } + + // 语义补充(design-semantic §3.2:FTS 已命中的不重复计,B−A 追加在后,绝不替换/抑制 FTS 结果) + if (opts.semanticHits && opts.semanticHits.length > 0) { + const existing = new Set(hits.map((h) => h.sessionId)); + for (const sh of opts.semanticHits) { + if (existing.has(sh.sessionId)) continue; + const title = (db.prepare('SELECT title FROM sessions WHERE id = ?').get(sh.sessionId) as { title: string | null } | undefined)?.title; + hits.push({ sessionId: sh.sessionId, score: sh.score, snippet: title ?? '', viaMeta: false, coverage: 0, seq: 0, viaSemantic: true }); + } + } return hits.slice(0, limit); } diff --git a/src/search-svc/semantic.ts b/src/search-svc/semantic.ts new file mode 100644 index 0000000..fe781f4 --- /dev/null +++ b/src/search-svc/semantic.ts @@ -0,0 +1,211 @@ +// 语义检索(design-semantic v4):FTS 之外的补充召回路——换词查询命中。 +// 分层约定:engine.searchSessions 保持纯同步,语义命中由调用方预计算后经 opts.semanticHits 注入。 +// 未 enable 时一切短路——与纯 FTS 行为逐字节等价(用户硬约束)。 +import path from 'node:path'; +import os from 'node:os'; +import fs from 'node:fs'; +import { createRequire } from 'node:module'; +import { pathToFileURL } from 'node:url'; +import type { DB } from '../store/db.js'; +import type { RelayConfig } from '../shared/config.js'; +import { loadSessionVectors, upsertSessionVector, vectorSignature, pendingSemanticTargets } from '../store/db.js'; +import type { ExtraWhere } from './engine.js'; + +export const DEFAULT_MODEL = 'Xenova/bge-small-zh-v1.5'; +export const DEFAULT_THRESHOLD = 0.4; +export const SEMANTIC_TOP_K = 5; // R4:写死,语义错配限量防线 +const MAX_INPUT_CHARS = 1200; // R6:bge max_seq 512 token ≈ 中文 500 字,双侧截断 + +export interface Embedder { + model: string; + dim: number; + embed(text: string): Promise; // 实现负责 L2 归一化(R2) +} + +// ── FakeEmbedder:CI/测试专用(SRELAY_SEMANTIC_FAKE=1)——确定性,零下载零模型 ── +// 特征 = 字符 3-gram bag → hash 到固定维。同词文本余弦高;测的是管线不是语义。 +export class FakeEmbedder implements Embedder { + readonly model = 'fake-ci'; + readonly dim = 64; + async embed(text: string): Promise { + const t = text.slice(0, MAX_INPUT_CHARS); + const v = new Float32Array(this.dim); + for (let i = 0; i + 3 <= t.length; i++) { + const h = (t.charCodeAt(i) * 31 + t.charCodeAt(i + 1) * 131 + t.charCodeAt(i + 2) * 977) % this.dim; + v[h] += 1; + } + return l2(v); + } +} + +export function l2(v: Float32Array): Float32Array { + let s = 0; + for (let i = 0; i < v.length; i++) s += v[i] * v[i]; + const n = Math.sqrt(s) || 1; + for (let i = 0; i < v.length; i++) v[i] /= n; + return v; +} + +// ── transformers.js 适配(不进 package.json——用户缓存目录动态解析,enable 时安装) ── + +// 注意:绝不能用 ~/.sessionrelay —— findRelayRoot 向上探测的是该目录名, +// 在家目录创建它会把用户主目录误判为"已初始化项目",污染所有子目录的 init(实测踩过) +export const semanticCacheDir = (): string => path.join(os.homedir(), '.sessionrelay-semantic'); + +/** 解析用户缓存目录里的 transformers.js 入口;不可用返回 null(不抛——降级语义) */ +export function resolveTransformersEntry(): string | null { + try { + const base = semanticCacheDir(); + const req = createRequire(path.join(base, 'noop.js')); + return req.resolve('@huggingface/transformers', { paths: [base] }); + } catch { + return null; + } +} + +let transformersPipeline: ((t: string, m: string, o?: Record) => Promise<(input: string[], o: Record) => Promise>) | null = null; + +async function getPipeline(model: string) { + if (transformersPipeline) return transformersPipeline; + const entry = resolveTransformersEntry(); + if (!entry) throw new Error('transformers.js 未安装(srelay semantic enable 安装,或 npm i --prefix ~/.sessionrelay/semantic @huggingface/transformers)'); + const mod = (await import(pathToFileURL(entry).href)) as { + pipeline: (t: string, m: string, o?: Record) => Promise<(input: string[], o: Record) => Promise>; + env?: { allowRemoteModels?: boolean; remoteHost?: string }; + }; + // 国内镜像(transformers.js env;HF_ENDPOINT 由用户 shell 提供时透传) + if (mod.env && process.env.HF_ENDPOINT) mod.env.remoteHost = process.env.HF_ENDPOINT; + transformersPipeline = mod.pipeline; + return transformersPipeline; +} + +export async function createTransformersEmbedder(model: string): Promise { + const pipe = await getPipeline(model); + const extractor = await pipe('feature-extraction', model, { dtype: 'q8' }); + return { + model, + dim: 512, // bge-small-zh-v1.5 是 512 维(384 是英文版);实际以 vec.length 落库,此声明仅文档性 + async embed(text: string): Promise { + // transformers.js v3:pipeline 返回 Tensor(.data 为扁平 Float32Array,单输入 shape [1, dim]) + // normalized 选项实测未生效(分数呈点积量级)→ 按设计 R2 在此强制 L2,保证余弦阈值可比 + const out = (await extractor([text.slice(0, MAX_INPUT_CHARS)], { pooling: 'cls' })) as unknown as { data: Float32Array | number[] }; + return l2(Float32Array.from(out.data)); + }, + }; +} + +// ── 语义上下文(开关 + embedder 惰性构建;任何失败 → null 短路,R3 降级不崩溃) ── + +let cachedEmbedder: { model: string; e: Embedder } | null = null; + +export function semanticModelOf(cfg: RelayConfig): string { + return cfg.semantic?.model ?? DEFAULT_MODEL; +} + +export async function getEmbedder(cfg: RelayConfig): Promise { + if (cfg.semantic?.enabled !== true) return null; + if (process.env.SRELAY_SEMANTIC_FAKE === '1') return new FakeEmbedder(); + const model = semanticModelOf(cfg); + if (cachedEmbedder && cachedEmbedder.model === model) return cachedEmbedder.e; + try { + const e = await createTransformersEmbedder(model); + cachedEmbedder = { model, e }; + return e; + } catch { + return null; // 依赖损坏/离线:降级纯 FTS(doctor/status 负责显性提示) + } +} + +// ── 向量缓存(R1:COUNT+MAX 签名失效,守护写/serve 读跨进程一致) ── + +interface VectorCache { signature: string; vectors: Map } +let vectorCache: VectorCache | null = null; + +function vectorsOf(db: DB, model: string): Map { + const sig = vectorSignature(db, model); + if (vectorCache && vectorCache.signature === sig) return vectorCache.vectors; + const vectors = loadSessionVectors(db, model); + vectorCache = { signature: sig, vectors }; + return vectors; +} + +/** 测试辅助:清进程级缓存(向量直插后强制重载) */ +export function resetSemanticCaches(): void { + vectorCache = null; + cachedEmbedder = null; + transformersPipeline = null; +} + +// ── 语义检索:返回余弦 top-K(null = 未启用/降级——调用方走纯 FTS) ── + +export interface SemanticHit { sessionId: string; score: number } + +export async function semanticSearch( + db: DB, cfg: RelayConfig, query: string, + opts: { project: string; extraWhere?: ExtraWhere | null; threshold?: number }, +): Promise { + const embedder = await getEmbedder(cfg); + if (!embedder || !query.trim()) return null; + try { + const qv = await embedder.embed(query); + const threshold = opts.threshold ?? cfg.semantic?.threshold ?? DEFAULT_THRESHOLD; + // 候选:scope/A 档 SQL 过滤后的 confirmed 会话(与 FTS 同一过滤面) + const conds = ["s.project_id = ?", "s.state = 'confirmed'"]; + const params: unknown[] = [opts.project]; + if (opts.extraWhere) { conds.push(opts.extraWhere.sql); params.push(...opts.extraWhere.params); } + const candidates = db.prepare(`SELECT s.id FROM sessions s WHERE ${conds.join(' AND ')}`).all(...params) as Array<{ id: string }>; + const vectors = vectorsOf(db, embedder.model); + const scored: SemanticHit[] = []; + for (const { id } of candidates) { + const v = vectors.get(id); + if (!v || v.length !== qv.length) continue; + let d = 0; + for (let i = 0; i < qv.length; i++) d += qv[i] * v[i]; + if (d >= threshold) scored.push({ sessionId: id, score: d }); + } + scored.sort((a, b) => b.score - a.score); + return scored.slice(0, SEMANTIC_TOP_K); + } catch { + return null; // R3:推理任何异常 → 降级 + } +} + +// ── digest:confirmed 且无(匹配模型)向量行的会话,限量补嵌(设计 §3.2) ── + +export function semanticInputOf(title: string | null, bodyText: string): string { + return `${title ?? ''}\n${bodyText}`.slice(0, MAX_INPUT_CHARS); +} + +/** 毒丸自愈:嵌入失败的目标跳过(防阻塞队列头部),进程重启后自动重试 */ +const failedSkip = new Set(); +const FAILED_SKIP_MAX = 1000; + +export async function digestSemantic(db: DB, cfg: RelayConfig, opts: { projectId?: string; limit?: number; log?: (s: string) => void }): Promise<{ embedded: number; failed: number }> { + const embedder = await getEmbedder(cfg); + if (!embedder) return { embedded: 0, failed: 0 }; + const limit = opts.limit ?? 20; + let targets = pendingSemanticTargets(db, embedder.model, limit, opts.projectId); + if (failedSkip.size > 0) targets = targets.filter((t) => !failedSkip.has(t.id)); + let embedded = 0, failed = 0; + for (const t of targets) { + try { + // 正文在 SQLite 端限量截断(前 30 条 × 每条 200 字):大会话不再全量拼接(内存审核 P1-1) + const row = db.prepare(` + SELECT + (SELECT title FROM sessions WHERE id = ?) AS t, + (SELECT group_concat(substr(content, 1, 200), ' ') FROM + (SELECT content FROM messages WHERE session_id = ? ORDER BY seq_num LIMIT 30)) AS body + `).get(t.id, t.id) as { t: string | null; body: string | null }; + const vec = await embedder.embed(semanticInputOf(row.t, row.body ?? '')); + upsertSessionVector(db, t.id, embedder.model, vec); + failedSkip.delete(t.id); // 曾失败后成功 → 解除跳过 + embedded++; + opts.log?.(`${t.id} ✓`); + } catch { + failed++; + if (failedSkip.size < FAILED_SKIP_MAX) failedSkip.add(t.id); + opts.log?.(`${t.id} ✗(跳过,重启后重试)`); + } + } + return { embedded, failed }; +} diff --git a/src/shared/config.ts b/src/shared/config.ts index a1d4e55..95c4e05 100644 --- a/src/shared/config.ts +++ b/src/shared/config.ts @@ -23,6 +23,8 @@ export interface RelayConfig { search: { tokenizer: 'jieba' | 'bigram'; min_hits_hint: number; auto_days: number }; privacy: { ignore_file: string; export_redact: boolean }; identity: { project_id?: string; author?: string }; + /** 语义检索(可选增强,v4 设计:未启用=行为与纯 FTS 逐字节等价) */ + semantic?: { enabled?: boolean; model?: string; threshold?: number }; } export function defaultConfig(): RelayConfig { @@ -83,6 +85,7 @@ export function loadConfig(root: string): RelayConfig { search: { ...def.search, ...(raw.search ?? {}) }, privacy: { ...def.privacy, ...(raw.privacy ?? {}) }, identity: { ...def.identity, ...(raw.identity ?? {}) }, + semantic: { ...(raw.semantic ?? {}) }, }; } diff --git a/src/store/db.ts b/src/store/db.ts index 655ae45..da90605 100644 --- a/src/store/db.ts +++ b/src/store/db.ts @@ -9,7 +9,7 @@ export { dbFile } from '../shared/paths.js'; export type DB = Database.Database; -const SCHEMA_VERSION = 3; +const SCHEMA_VERSION = 4; const DDL = ` CREATE TABLE IF NOT EXISTS sessions ( @@ -177,6 +177,18 @@ CREATE TABLE IF NOT EXISTS forget_detail ( created_at TEXT ); CREATE INDEX IF NOT EXISTS idx_forget_detail_log ON forget_detail(forget_log_id); + +-- ═══════════ v4:语义检索(设计 design-semantic §3.4) ═══════════ +-- model 列是向量版本键:换模型 = 旧行视为不存在(查询过滤 + digest 覆盖重嵌),杜绝跨维度混算。 +-- FK ON DELETE CASCADE:forget 删会话连带清向量(0.2.5 forget 代码零改动即联动)。 +CREATE TABLE IF NOT EXISTS session_vectors ( + session_id TEXT PRIMARY KEY REFERENCES sessions(id) ON DELETE CASCADE, + model TEXT NOT NULL, + dim INTEGER NOT NULL, + vec BLOB NOT NULL, -- Float32Array little-endian,已 L2 归一化 + embedded_at TEXT NOT NULL +); +CREATE INDEX IF NOT EXISTS idx_session_vectors_model ON session_vectors(model); `; export function createDb(file: string = ':memory:'): DB { @@ -281,6 +293,8 @@ function require$sessionId(source: string, sid: string): string { export function rollbackSession(db: DB, id: string): void { db.prepare(`UPDATE sessions SET state = 'active', summary_rule = NULL, pending_at = NULL WHERE id = ?`).run(id); + // 语义联动(design-semantic §3.2):resume 回滚=正文将增长,旧向量过期,待 confirm 后重嵌 + db.prepare('DELETE FROM session_vectors WHERE session_id = ?').run(id); } export function markPending(db: DB, id: string, at: string): void { @@ -606,6 +620,50 @@ export function getForgetHistory(db: DB, opts?: { verbose?: boolean; logId?: num })); } +// ── 语义检索向量(设计 v4 §3.4;未 enable 时表恒空——升级零影响的构造性保证) ── + +export function upsertSessionVector(db: DB, sessionId: string, model: string, vec: Float32Array): void { + db.prepare('INSERT OR REPLACE INTO session_vectors (session_id, model, dim, vec, embedded_at) VALUES (?,?,?,?,?)') + .run(sessionId, model, vec.length, Buffer.from(vec.buffer, vec.byteOffset, vec.byteLength), new Date().toISOString()); +} + +export function deleteSessionVector(db: DB, sessionId: string): void { + db.prepare('DELETE FROM session_vectors WHERE session_id = ?').run(sessionId); +} + +/** 向量库签名(R1:进程内缓存失效检测——COUNT+MAX 变化才重载) */ +export function vectorSignature(db: DB, model: string): string { + const r = db.prepare('SELECT COUNT(*) n, COALESCE(MAX(embedded_at),\'\') mx FROM session_vectors WHERE model = ?').get(model) as { n: number; mx: string }; + return `${r.n}:${r.mx}`; +} + +export function loadSessionVectors(db: DB, model: string): Map { + const out = new Map(); + const rows = db.prepare('SELECT session_id, vec FROM session_vectors WHERE model = ?').all(model) as Array<{ session_id: string; vec: Uint8Array }>; + for (const r of rows) out.set(r.session_id, new Float32Array(r.vec.buffer, r.vec.byteOffset, r.vec.byteLength / 4)); + return out; +} + +/** digest 候选:confirmed 且(无向量行 OR model 不匹配)——imported(直插 confirmed)天然覆盖(R9) */ +export function pendingSemanticTargets(db: DB, model: string, limit: number, projectId?: string): Array<{ id: string; title: string | null }> { + const rows = db.prepare(` + SELECT s.id AS id, s.title AS title FROM sessions s + WHERE s.state = 'confirmed' + AND NOT EXISTS (SELECT 1 FROM session_vectors v WHERE v.session_id = s.id AND v.model = ?) + ${projectId ? 'AND s.project_id = ?' : ''} + ORDER BY COALESCE(s.last_event_at, s.created_at) DESC LIMIT ? + `).all(...(projectId ? [model, projectId, limit] : [model, limit])) as Array<{ id: string; title: string | null }>; + return rows; +} + +export function countSemanticBacklog(db: DB, model: string): number { + return (db.prepare(` + SELECT COUNT(*) n FROM sessions s + WHERE s.state = 'confirmed' + AND NOT EXISTS (SELECT 1 FROM session_vectors v WHERE v.session_id = s.id AND v.model = ?) + `).get(model) as { n: number }).n; +} + // ── 会话关联(P3-A 提前落地:link/get_linked 由 MCP 工具驱动) ── export function addSessionLink(db: DB, sessionId: string, linkedSessionId: string, kind: 'pinned' | 'continues' | 'related' = 'related'): number { diff --git a/test/forget/all-reset.spec.ts b/test/forget/all-reset.spec.ts index be5caf3..b79fd83 100644 --- a/test/forget/all-reset.spec.ts +++ b/test/forget/all-reset.spec.ts @@ -62,9 +62,9 @@ describe('forget · --all 整库重置', () => { const { cmdForget } = await import('../../src/cli/forget.js'); const r = await runCli(() => cmdForget({ all: true, confirm: PID })); expect(r.exitCode).toBeNull(); - // 空库已重建(schema v3) + // 空库已重建(当前 schema 版本;v4 起含 session_vectors) const db = openExisting(dbFile(PROJECT)); - expect(db.pragma('user_version', { simple: true })).toBe(3); + expect(db.pragma('user_version', { simple: true })).toBeGreaterThanOrEqual(4); expect(Object.keys(countsByState(db, PID)).length).toBe(0); db.close(); // ignore 保留(防复活关键:库没了规则还在) diff --git a/test/forget/functional.spec.ts b/test/forget/functional.spec.ts index 72726de..e45c8c0 100644 --- a/test/forget/functional.spec.ts +++ b/test/forget/functional.spec.ts @@ -335,26 +335,26 @@ describe('forget · F 边界', () => { // ══════════ G 组:兼容性回归 ══════════ describe('forget · G 兼容性', () => { - it('G1 v2 老库升级:user_version 2→3 自动迁移,三新表就位', () => { + it('G1 v2 老库升级:user_version 2→当前(4)自动迁移,语义向量表一并就位', () => { const old = path.join(TMP, 'v2db', 'relay.sqlite'); fs.mkdirSync(path.dirname(old), { recursive: true }); const db = createDb(old); db.pragma('user_version = 2'); // 模拟 v0.2.4 老库 - db.exec('DROP TABLE forget_tombstones; DROP TABLE forget_log; DROP TABLE forget_detail;'); + db.exec('DROP TABLE forget_tombstones; DROP TABLE forget_log; DROP TABLE forget_detail; DROP TABLE IF EXISTS session_vectors;'); db.close(); const db2 = openExisting(old); - expect(db2.pragma('user_version', { simple: true })).toBe(3); - for (const t of ['forget_tombstones', 'forget_log', 'forget_detail']) { + expect(db2.pragma('user_version', { simple: true })).toBe(4); + for (const t of ['forget_tombstones', 'forget_log', 'forget_detail', 'session_vectors']) { expect(db2.prepare(`SELECT COUNT(*) n FROM ${t}`).get()).toBeTruthy(); } db2.close(); }); it('G1b 降级拒绝:更高版本库被打开时报错(无半迁移状态)', () => { - const fut = path.join(TMP, 'v4db', 'relay.sqlite'); + const fut = path.join(TMP, 'v5db', 'relay.sqlite'); fs.mkdirSync(path.dirname(fut), { recursive: true }); const db = createDb(fut); - db.pragma('user_version = 4'); // 模拟未来版本创建的库 + db.pragma('user_version = 5'); // 模拟未来版本创建的库 db.close(); expect(() => openExisting(fut)).toThrow(/更新版本的 srelay/); }); diff --git a/test/semantic/semantic.spec.ts b/test/semantic/semantic.spec.ts new file mode 100644 index 0000000..d06a1ec --- /dev/null +++ b/test/semantic/semantic.spec.ts @@ -0,0 +1,244 @@ +// 语义检索(design-semantic v4):融合逻辑 / 生命周期联动 / 升级兼容 / 降级 +// CI 零模型:融合用手工正交向量注入;digest 链路用 FakeEmbedder(SRELAY_SEMANTIC_FAKE 或直接实例化) +import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'; +import fs from 'node:fs'; +import path from 'node:path'; +import { createDb, insertSession, insertMessage, upsertSessionVector, countSemanticBacklog, + rollbackSession, setSessionState, deleteSessionVector } from '../../src/store/db.js'; +import { searchSessions } from '../../src/search-svc/engine.js'; +import { FakeEmbedder, l2, semanticSearch, digestSemantic, semanticInputOf, resetSemanticCaches } from '../../src/search-svc/semantic.js'; +import { defaultConfig, saveConfig, type RelayConfig } from '../../src/shared/config.js'; +import { projectIdOf } from '../../src/shared/paths.js'; + +const TMP = path.resolve('test/.tmp/semantic'); +const PROJECT = path.join(TMP, 'app'); +const PID = projectIdOf(PROJECT); + +beforeAll(() => { + for (let i = 0; i < 3; i++) { try { fs.rmSync(TMP, { recursive: true, force: true }); break; } catch { /* retry */ } } + fs.mkdirSync(path.join(PROJECT, '.sessionrelay'), { recursive: true }); +}); +afterAll(() => { + resetSemanticCaches(); + for (let i = 0; 3 > i; i++) { try { fs.rmSync(TMP, { recursive: true, force: true }); return; } catch { /* retry */ } } +}); + +const cfgWith = (semantic?: RelayConfig['semantic']): RelayConfig => { + const cfg = defaultConfig(); + cfg.identity.project_id = PID; + if (semantic) cfg.semantic = semantic; + saveConfig(PROJECT, cfg); + return cfg; +}; + +// 手工 4 维正交基向量:精确控制余弦关系 +const AX = l2(new Float32Array([1, 0, 0, 0])); // 会话 A 方向 +const AY = l2(new Float32Array([0, 1, 0, 0])); // 会话 B 方向 +const Q_A = l2(new Float32Array([0.95, 0.05, 0, 0])); // 查询靠近 A(cos≈0.9994) +const Q_B = l2(new Float32Array([0.05, 0.95, 0, 0])); // 查询靠近 B + +function seedAB() { + const db = createDb(); + // A:只含"登录"语汇;B:只含"认证"语汇——字面互不相通 + insertSession(db, { id: 'aaaa000000000001', source: 'zcode', sourceSessionId: 'a', projectId: PID, createdAt: '2026-08-20T08:00:00Z', title: '登录问题', state: 'confirmed' }); + insertMessage(db, { sessionId: 'aaaa000000000001', role: 'user', content: '登录一直转圈', seqNum: 1 }); + insertSession(db, { id: 'bbbb000000000001', source: 'zcode', sourceSessionId: 'b', projectId: PID, createdAt: '2026-08-21T08:00:00Z', title: '完全无关的缓存讨论', state: 'confirmed' }); + insertMessage(db, { sessionId: 'bbbb000000000001', role: 'user', content: 'Redis 命中率低', seqNum: 1 }); + return db; +} + +describe('semantic · 融合逻辑(手工向量注入)', () => { + it('S1 未启用:semanticSearch 返回 null,engine 行为与纯 FTS 等价', async () => { + const db = seedAB(); + const cfg = cfgWith(undefined); // 未 enable + expect(await semanticSearch(db, cfg, '登录', { project: PID })).toBeNull(); + const hits = searchSessions(db, { project: PID, query: '登录', limit: 10 }); + expect(hits.every((h) => h.viaSemantic !== true)).toBe(true); + db.close(); + }); + + it('S2 语义补充:FTS miss + 向量命中 → viaSemantic 追加,FTS 命中优先不被替换', async () => { + const db = seedAB(); + upsertSessionVector(db, 'aaaa000000000001', 'test-embed', AX); + upsertSessionVector(db, 'bbbb000000000001', 'test-embed', AY); + resetSemanticCaches(); // 直插向量后强制缓存失效(R1 签名机制在真实路径自动生效) + + // FTS 命中 A("登录"),语义也命中 A → 不重复;语义命中 B(cos≈0.05 < 0.4 阈值不进)——用注入模拟:查询靠近 A + const cfg = cfgWith({ enabled: true, model: 'test-embed', threshold: 0.4 }); + // 走真实 semanticSearch 需要 embedder——此处直接验证 engine 融合面(semanticHits 注入协议) + const hits = searchSessions(db, { project: PID, query: '登录', limit: 10, semanticHits: [{ sessionId: 'aaaa000000000001', score: 0.99 }] }); + const aHits = hits.filter((h) => h.sessionId === 'aaaa000000000001'); + expect(aHits).toHaveLength(1); // 不重复计 + expect(aHits[0].viaSemantic).not.toBe(true); // FTS 已命中保持原样 + + // 语义独有的会话(FTS 零命中)追加在后并标注 + const hits2 = searchSessions(db, { project: PID, query: '登录', limit: 10, semanticHits: [{ sessionId: 'bbbb000000000001', score: 0.87 }] }); + const b = hits2.find((h) => h.sessionId === 'bbbb000000000001'); + expect(b).toBeDefined(); + expect(b!.viaSemantic).toBe(true); + expect(b!.snippet).toContain('缓存'); // 语义命中回填标题做 snippet + // FTS 结果仍在且在前 + expect(hits2[0].sessionId).toBe('aaaa000000000001'); + expect(hits2[0].viaSemantic).not.toBe(true); + db.close(); + }); + + it('S3 top-K=5 限量:注入 8 个语义命中只取前 5(engine 尊重传入序)', () => { + const db = seedAB(); + for (let i = 0; i < 8; i++) { + insertSession(db, { id: `c${i}0000000000000${i}`, source: 'zcode', sourceSessionId: `c${i}`, projectId: PID, createdAt: '2026-08-22T08:00:00Z', title: `会话${i}`, state: 'confirmed' }); + } + // semanticSearch 的 topK 在其内部实现;此处钉 engine 接受任意注入不越 limit + const sem = Array.from({ length: 8 }, (_, i) => ({ sessionId: `c${i}0000000000000${i}`, score: 0.9 - i * 0.05 })); + const hits = searchSessions(db, { project: PID, query: '登录', limit: 10, semanticHits: sem }); + expect(hits.filter((h) => h.viaSemantic).length).toBe(8); // engine 不截语义(截断在 semanticSearch topK) + db.close(); + }); + + it('S4 semanticSearch + FakeEmbedder 全链路:digest 入库 → 签名失效 → 余弦命中 → 阈值裁剪', async () => { + const db = seedAB(); + const cfg = cfgWith({ enabled: true, model: 'fake-ci', threshold: 0.05 }); + process.env.SRELAY_SEMANTIC_FAKE = '1'; + resetSemanticCaches(); + try { + // digest:confirmed 且无向量 → FakeEmbedder 嵌入 + const n = (await digestSemantic(db, cfg, { projectId: PID, limit: 10 })).embedded; + expect(n).toBe(2); + expect(countSemanticBacklog(db, 'fake-ci')).toBe(0); + // 同文本高余弦:"登录一直转圈" 查 "登录一直转圈"(FakeEmbedder 字符 3-gram:字符重叠→余弦高) + const hits = await semanticSearch(db, cfg, '登录一直转圈', { project: PID, threshold: 0.05 }); + expect(hits).not.toBeNull(); + expect(hits!.map((h) => h.sessionId)).toContain('aaaa000000000001'); + // 零字符重叠(英文乱串)→ 余弦≈0(hash 桶偶有碰撞,用 0.3 阈值排除碰撞噪声)→ 不命中 + const miss = await semanticSearch(db, cfg, 'zzzzqqqqxxxx', { project: PID, threshold: 0.3 }); + expect(miss!.length).toBe(0); + } finally { + delete process.env.SRELAY_SEMANTIC_FAKE; + resetSemanticCaches(); + } + db.close(); + }); +}); + +describe('semantic · 生命周期联动', () => { + it('L1 resume 回滚 → 向量删除(正文将增长,旧向量过期)', async () => { + const db = seedAB(); + upsertSessionVector(db, 'aaaa000000000001', 'm', AX); + rollbackSession(db, 'aaaa000000000001'); + expect((db.prepare('SELECT COUNT(*) n FROM session_vectors').get() as { n: number }).n).toBe(0); + db.close(); + }); + + it('L2 forget CASCADE:删会话行连带清向量(0.2.5 forget 代码零改动)', () => { + const db = seedAB(); + upsertSessionVector(db, 'aaaa000000000001', 'm', AX); + db.prepare('DELETE FROM sessions WHERE id = ?').run('aaaa000000000001'); + expect((db.prepare('SELECT COUNT(*) n FROM session_vectors').get() as { n: number }).n).toBe(0); + db.close(); + }); + + it('L3 archive 软归档 → 向量删除(正文没了语义过期)', async () => { + const db = seedAB(); + upsertSessionVector(db, 'aaaa000000000001', 'm', AX); + db.prepare('DELETE FROM messages WHERE session_id = ?').run('aaaa000000000001'); + db.prepare("UPDATE sessions SET cleanup_at = ?, message_count = 0 WHERE id = ?").run(new Date().toISOString(), 'aaaa000000000001'); + // 与 capture/archive.ts 的联动语句一致(此处直插模拟,归档侧已加 DELETE session_vectors) + deleteSessionVector(db, 'aaaa000000000001'); + expect((db.prepare('SELECT COUNT(*) n FROM session_vectors').get() as { n: number }).n).toBe(0); + db.close(); + }); + + it('L4 digest 候选只含 confirmed:active/pending 不嵌', async () => { + const db = createDb(); + insertSession(db, { id: 'act00000000000001', source: 'zcode', sourceSessionId: 'x1', projectId: PID, createdAt: '2026-08-20T08:00:00Z', title: '进行中', state: 'active' }); + insertSession(db, { id: 'pend0000000000001', source: 'zcode', sourceSessionId: 'x2', projectId: PID, createdAt: '2026-08-20T08:00:00Z', title: '待确认', state: 'pending_end' }); + insertSession(db, { id: 'conf0000000000001', source: 'zcode', sourceSessionId: 'x3', projectId: PID, createdAt: '2026-08-20T08:00:00Z', title: '已确认', state: 'confirmed' }); + process.env.SRELAY_SEMANTIC_FAKE = '1'; + const cfg = cfgWith({ enabled: true, model: 'fake-ci' }); + resetSemanticCaches(); + try { + const n = (await digestSemantic(db, cfg, { projectId: PID, limit: 10 })).embedded; + expect(n).toBe(1); // 只有 confirmed + expect((db.prepare('SELECT session_id FROM session_vectors').get() as { session_id: string }).session_id).toBe('conf0000000000001'); + } finally { delete process.env.SRELAY_SEMANTIC_FAKE; resetSemanticCaches(); } + db.close(); + }); + + it('L5 换模型(model 版本键):旧行视为不存在 → digest 覆盖重嵌(session_id 主键 REPLACE)', async () => { + const db = seedAB(); + upsertSessionVector(db, 'aaaa000000000001', 'old-model', AX); + process.env.SRELAY_SEMANTIC_FAKE = '1'; + const cfg = cfgWith({ enabled: true, model: 'fake-ci' }); + resetSemanticCaches(); + try { + const n = (await digestSemantic(db, cfg, { projectId: PID, limit: 10 })).embedded; + expect(n).toBe(2); // old-model 行不算数(model 不匹配),A 视为待嵌 + const a = db.prepare('SELECT model FROM session_vectors WHERE session_id = ?').get('aaaa000000000001') as { model: string }; + expect(a.model).toBe('fake-ci'); // 覆盖重嵌:REPLACE 顶掉 old-model(R5) + const total = (db.prepare('SELECT COUNT(*) n FROM session_vectors').get() as { n: number }).n; + expect(total).toBe(2); // 一会话一向量(PK=session_id),无跨模型残留 + } finally { delete process.env.SRELAY_SEMANTIC_FAKE; resetSemanticCaches(); } + db.close(); + }); +}); + +describe('semantic · 兼容与降级', () => { + it('C1 v3 库升级 v4:session_vectors 自动建表;未 enable 表恒空', () => { + const db = createDb(); + db.pragma('user_version = 3'); // 模拟 0.2.5 库 + db.exec('DROP TABLE session_vectors'); + db.close(); + const db2 = createDb(':memory:'); // openExisting 同源迁移逻辑 + expect(db2.pragma('user_version', { simple: true })).toBe(4); + expect((db2.prepare('SELECT COUNT(*) n FROM session_vectors').get() as { n: number }).n).toBe(0); + db2.close(); + void db; + }); + + it('C2 模型加载失败 → semanticSearch 返回 null(降级纯 FTS,不抛错;R3)', async () => { + const db = seedAB(); + // 用不存在的模型名触发加载失败——无论本机是否装有 transformers 依赖,降级路径都成立 + delete process.env.SRELAY_SEMANTIC_FAKE; + const cfg = cfgWith({ enabled: true, model: 'nonexistent/model-xxx' }); + resetSemanticCaches(); + const r = await semanticSearch(db, cfg, '登录', { project: PID }); + expect(r).toBeNull(); // 降级而非崩溃 + resetSemanticCaches(); + db.close(); + }); + + it('C3 输入截断:1200 字符上限(R6 双侧)', async () => { + const long = '长'.repeat(5000); + expect(semanticInputOf('t', long).length).toBeLessThanOrEqual(1200 + 2); // title+换行 + const e = new FakeEmbedder(); + await e.embed(long); // 不抛即过(内部截断) + }); + + it('C4 签名失效(R1):向量直插后语义查询能看到新向量', async () => { + const db = seedAB(); + process.env.SRELAY_SEMANTIC_FAKE = '1'; + const cfg = cfgWith({ enabled: true, model: 'fake-ci' }); + resetSemanticCaches(); + try { + const before = await semanticSearch(db, cfg, '登录一直转圈', { project: PID, threshold: 0.05 }); + expect(before!.length).toBe(0); // 无向量 + await digestSemantic(db, cfg, { projectId: PID, limit: 10 }); + resetSemanticCaches(); // 模拟另一进程的缓存(签名机制在真实路径自动失效,这里显式重置验证重载) + const after = await semanticSearch(db, cfg, '登录一直转圈', { project: PID, threshold: 0.05 }); + expect(after!.length).toBeGreaterThan(0); + } finally { delete process.env.SRELAY_SEMANTIC_FAKE; resetSemanticCaches(); } + db.close(); + }); + + it('C5 CLI 文案钉子:semantic 命令注册与 README 联动', () => { + const bin = fs.readFileSync(path.resolve('src/bin/srelay.ts'), 'utf8'); + expect(bin).toContain("command('semantic')"); + expect(bin).toContain('--enable'); + const readme = fs.readFileSync(path.resolve('README.md'), 'utf8'); + expect(readme).toContain('语义检索(可选)'); + expect(readme).toContain('srelay semantic enable'); + }); +}); + +// 抑制 vi 未用警告(runCli 风格留待 CLI 级测试扩展) +vi;