Optional decision-provider for memory importance classification (cache-neutral, default off)

Short description

A pluggable producer for the classify-memories dreamer task's importance/scope/shareability output — first provider: TypeSafe's Jev (decision.provider: "jev" | "off", default off). This issue also maps the full set of decision points where the same provider seam could later apply, with measured evidence where it exists. Cache neutrality is preserved throughout: decisions are computed out-of-band and written through the existing cache-neutral setMemoryClassification path.

Problem: what does this solve?

The incumbent importance classifier is good but saturates its own scale, which degrades injection selection once the memory budget binds:

In my production store (1,852 active memories), 71% of importances land in the 40–59 band, and 724 memories share importance = 50 exactly.
memorySelectionOrder (inject-compartments.ts:1053) breaks ties by id — so once the injection budget binds, that 39% of the pool is ordered by creation order, which carries no relevance signal.
Every new memory needs a dreamer classifier round-trip (agent spawn + LLM call) before it is classified.

I verified this histogram with:

SELECT (importance/20)*20 AS band, COUNT(*) FROM memories WHERE status='active' GROUP BY band;

Proposed solution

The proposal: Jev as the classify-memories producer

Keep classify-memories exactly as it is — same task, same lease, same setMemoryClassification write — but let a decision provider fill the columns when configured:

memory:
  classification:
    provider: "off"   # off (default) | jev

Jev (TypeSafe's System One model, api.typesafe.ai/v1/systemone) returns typed probabilities (noul/choice/score) with flat ~1s latency for 1–200 batched questions. Measured on my store:

40 batched keep/drop judgments in one 1.06s request, $0.000136; latency is flat from 1→200 questions (~1.0–1.6s).
Full-pool re-score of 1,852 memories ≈ $0.02 (upper bound; operating design scores the budget-relevant subset at fold time — pool-relative scores make cross-batch stitching invalid).
Hand-labeled 30-memory evaluation (I labeled each memory keep/drop as the domain expert): Jev top-15 overlap with my keep-set 10/15 vs incumbent 13/15; r(Jev, incumbent) = 0.668; Jev produced 22 distinct scores vs the incumbent's 13 on the same pool. Note the incumbent is the system under evaluation, so agreement with it is not the quality signal — outcome-based evaluation (did the agent re-derive a dropped fact?) is the real test for a trial.
Key design driver: scores are pool-relative (the same memory scored 0.463 alone vs 0.533 in a 12-item pool), so the persisted artifact is a decision band + ordinal rank captured at the fold — never a raw scalar; hysteresis (keep ≥ 0.60 / drop ≤ 0.40 / hold) prevents threshold flapping between folds.

Design constraints (all verified against current source):

Transform-path unchanged. Zero LLM/network calls added to the transform. Classification stays a dreamer task under the existing memory:<project> lease.
Cache-neutral writes only. Decisions land via setMemoryClassification (column-only + classified_at stamp — the existing "does not affect rendered m[0] bytes" path). No epoch bump, no mutation-log rows, no new HARD trigger.
Persisted decision, not a live score. Decision band + rank-at-fold + provenance columns (raw score, pool hash, model id, scored_at); never re-derived at render.
Hysteresis. keep ≥ 0.60 / drop ≤ 0.40 / hold (prior decision preserved inside the dead zone).
Failure semantics: degrade to incumbent. Provider outage → fall back to the current classifier path; stored decisions replay. Never blocks a dreamer run; never originates a bust.
Ride-only. Changed importances affect the m[1] render on the next natural pass — same as any classification change today.
Jurisdiction + disclosure. Provider calls happen only in the dreamer lane; for the jev provider, memory content is sent to a hosted endpoint — that's why default is off and the config key is explicit.

The full decision-point map (all surveyed, with evidence)

Where else the same provider shape was surveyed in this codebase — measured where noted:

A. Compartment-chunk rerank (search.ts:1331-1391) — the chunk lane is raw cosine with a ×0.8 single-source penalty and no second stage. Measured on 60 real chunks from my store: with synonym-disjoint queries (zero shared tokens), a judged rerank of the lexical top-10 shortlist got 2/5 vs lexical 0/5; the 3 misses were shortlist/excerpt-evidence misses, not judge errors. Query-time, so it cannot touch the cache. Also verified the hybrid fusion constants live at search.ts:51-54 (SEMANTIC_WEIGHT 0.7 / FTS_WEIGHT 0.3) and a rerank would sit after compareUnifiedResults.

B. Compaction keep/drop gate (historian output) — the same decision shape as the fast-jev-compaction project (two nouls per tool call: keep the call? keep the result verbatim?) — but that project rewrites the transcript at Claude Code's compaction event, a natural single bust point. MC's transplant: land verdicts through the existing tags.drop_mode ("full" | "truncated" | "edit_marker", module-state-sync.ts:83) + updateTagStatus replay, bust passes only. Measured decision shape: 40 judgments in one 1.06s request, $0.000136. Jev cannot generate — it gates what the historian's output keeps; the historian itself is untouched.

C. Auto-search hint gate (hook.ts:1260, static score_threshold: 0.6, below-threshold cached) — a learned gate could replace the fixed threshold. Enumerated, not measured. Low priority: the current one is fail-safe and cached.

D. Cross-source boost table (search.ts:51-54, hand constants 1.3 / 1.275 / 1.2 / 1.25) — judged relevance per source. Enumerated, not measured. Most invasive; last resort.

E. Primer clustering (PRIMER_CLUSTER_THRESHOLD 0.85 + 0.02 hysteresis in primer-clustering.ts) — judged cluster membership for candidate promotion. Enumerated, not measured — no live primer-pool evidence gathered this round.

F. Curate dedup prefilter — pairwise similarity judgments before the curate agent merges via ctx_memory. Enumerated, not measured. Deferred: the agent already does this; a prefilter adds cost before judgment.

The classification proposal above comes first because setMemoryClassification is already a documented cache-neutral write path; the rest are listed so the seam's applicability is visible in one place. Each could be its own follow-up issue if the maintainer prefers.

Alternatives considered

Tune the incumbent prompt for wider spread. Cheapest, keeps everything in-band. Doesn't fix the per-memory agent round-trip cost, and spread remains uncalibrated (no probability semantics).
Batch Classify via one big LLM call outside the dreamer. Loses the lease/idempotence machinery and re-introduces per-run nondeterminism the manifest protocol currently prevents.
Defer entirely. The 724-way tie is stable and deterministic (id tiebreak), so "incumbent is good enough" is a legitimate outcome; this issue proposes an opt-in improvement, not a replacement.

Area

Memory / historian / dreamer

Additional context

Evidence base: live probes on my production store (source line references throughout), full measurement matrix available on request. Also referenced: the fast-jev-compaction project (decision-shape precedent for point B) and jev-sift (batched relevance-screening contract).
I'm the author of merged fix(pi): resolve native embedding routing before registration #462; happy to implement behind the config flag with tests covering the hysteresis band, provider-failure fallback, and the cache-neutrality contract (no epoch/mutation-log writes from the provider path).

Thanks for the evidence — the histogram query is the right instrument, and the tie-break observation is a real defect regardless of what classifies: once the injection budget binds, ordering the importance = 50 band by id carries no signal. We will fix that part in-band (a secondary key with actual signal — last reinforcement / verification recency — rather than creation order); it needs no provider.

Before comparing classifiers, though, one question that may reframe the issue: how many of your 1,852 memories have classified_at set?

SELECT classified_at IS NOT NULL AS classified, COUNT(*), SUM(importance = 50) AS at_50
FROM memories WHERE status = 'active' GROUP BY 1;

I ask because on our own store the answer split cleanly. Classified memories spread across the whole scale (0–96, mode 60–79, 271 at exactly 50 out of 7,092 — about 4%). Unclassified ones sit at 50 or NULL and nowhere else. And while checking that I found classify-memories has been failing on several of our own projects for a week, silently from the user's point of view — the symptom is exactly "everything is at 50", which reads like a classifier that cannot discriminate when it is a classifier that never ran. If your classified_at column is mostly NULL, the incumbent has not been evaluated on your pool at all, and we have a task-health bug to fix rather than a scale problem. If it is set and the pool still peaks at 50, the classifier model is the lever: which model ran it? A small local or flash-class model collapsing to the midpoint is a prompt-and-model fix, and a per-band histogram in /ctx-status would make either case visible without SQL. Both are things we would take.

On the hosted decision provider: we are not going to add one, even default-off. Memory content leaving the machine is a new trust surface, and every optional provider is a permanent one to document, test across three harnesses and keep working. The direction for pluggable dreamer behaviour is the policy-pack work after the shared-engine consolidation (#358), where a classification policy would be a pack concern with the same review as everything else in a pack — not a per-task provider key. Your points about persisted decisions, hysteresis and ride-only writes are the right constraints for any classifier, ours included, and I have noted them there.

Leaving this open as the tracking point for the tie-break fix, the histogram, and the classify-health fix; will update when they ship.

Answering the question first, because it reframes the issue exactly as you suspected — though as a split, not a clean "mostly NULL".

Your query on my store

classified    count 1282   at_50  163
unclassified  count  704   at_50  692

So 65% classified / 35% not. Enough of the pool is unclassified to matter, and the unclassified rows are 98% at exactly 50 (692 of 704) — which is the default, not a score.

That means my headline number was wrong and I want to correct it here rather than let it stand. The issue says "71% of importances land in the 40–59 band, 39.3% at exactly 50". That was the whole active pool, and the pile inflating it is the default value:

band	classified	unclassified
00–19	17	1
20–39	181	3
40–49	337	2
50 exactly	163	692
51–59	286	1
60–79	261	4
80+	37	1
total	1282	704

Classified-only: 61% in 40–59, 12.7% at exactly 50 — against your 3.8%. Still peaked relative to yours, but it is a spread, not a collapse, and it is 72 distinct values on the classified side versus 11 on the unclassified side. The classifier that ran does discriminate. "The incumbent has not been evaluated on your pool at all" is too strong; "a third of the pool was never evaluated, and the default is what I measured" is the accurate version.

But your task-health finding reproduces here, and it is worse than classify

classify-memories last ran on this store 2026-09-17 00:27. Of the last 59 classify runs, 25 had pending > 0 and processed = 0. The two error shapes:

6x  provider_error: "Pi dreamer subagent failed (no_assistant):
    pi assistant produced empty text (provider error: This model requires
    a ClinePass subscription. Free-tier models (marke...)"
    + "classify-memories incomplete: N remain (was N at run start; processed 0 this run)"

The incomplete: N remain (was N at run start; processed 0 this run) string — where the pending count is identical at start and end — is 15 projects' current last_error. That is the shape you described: it reads as a classifier that cannot discriminate and is a classifier that never ran.

Two things I found while checking, both probably more useful to you than the classifier question:

The whole dreamer stopped, not just classify. Newest dream_runs row is 2026-09-17 01:08 — four days ago. 605 of 637 task_schedule_state rows are currently next_due_at <= now. So this is not a schedule problem; nothing has been dispatched. last_status across all 637 rows is None: 364, skipped: 156, completed: 79, failed: 38. The 364 None rows have never run at all.
Content edits clear classified_at but keep importance. The 12 unclassified rows not at 50 are 12 of 12 with updated_at > created_at — i.e. every one was content-edited, and rewriteMemoryContent clears classified_at without resetting importance (by design, per the comment). So the stragglers outside the 50 pile are the edit path, not a classifier artifact. Worth knowing that importance != 50 AND classified_at IS NULL is a reliable "was edited after classification" signal.

On the model question: this store's OMP dreamer model is Atria/Atria-Dawn-Preview at thinking_level: medium, with an apinex/cline-pass/opencode-zen fallback chain. The provider_error above is the fallback chain exhausting — the ClinePass line is a cline-pass/deepseek/deepseek-v4.1-flash attempt. So on this store it is model availability, not model capability.

On the two things you took

The tie-break fix is the right call and it has a larger blast radius than it looks here: with 692 of 704 unclassified rows sitting at exactly 50, that band is the unclassified pile on my store, so ordering it by recency means the newest unclassified memories survive the budget. That is a better default than creation order, and it also makes the pile visible — which is the real fix.

The histogram in /ctx-status would have caught this without SQL, and I would have caught my own error before filing. Please do.

On the provider

Understood and accepted — no hosted provider, and the reasoning (permanent trust surface and a permanent three-harness maintenance cost for something optional) is stronger than my default-off argument. Policy packs after #358 is a better home for it than a per-task provider key, and a pack is where the persisted-decision/hysteresis/ride-only constraints belong anyway.

I will leave the provider direction alone and follow #358 rather than re-arguing it. If the classify-health fix wants a reproducer, my store has 704 rows that fit the failing shape and I can export the dream_runs rows for the 25 zero-progress runs.

Follow-up to my reply above, and it changes the picture for your classify-health investigation.

I found why classify-memories has not run on this store since 2026-09-17, and it is not the classifier and not the schedule. The whole dreamer is dead on the OMP harness — filed as #496.

runTick does message-history maintenance before per-project work, in one try block, and sweepOrphanedOpenCodeMessageIndexes calls openCodeSweepHarness() as its first statement, which throws on any non-OpenCode harness:

function openCodeSweepHarness(): "opencode" | "opencode2" {
    const harness = getHarness();
    if (harness === "opencode" || harness === "opencode2") return harness;
    throw new Error(`OpenCode orphan sweep cannot read a ${harness} host store`);
}

The catch is in runTick, so it is contained — it logs and the tick ends. Which means the per-project loop below it never runs: no runProjectMaintenance, no runDueTasksForProject, no task dispatch at all. 714 occurrences in the log, and the last dream_runs row is 2026-09-17 01:08.

That reframes the answer I gave you above. The 704 unclassified rows on this store are not a classifier that stalled — they are the newest memories, created after the last run, because nothing has classified anything for four days. 89 of the 91 unclassified rows in the project I checked were created after the last classify run; only 2 predate it. And of the 25 zero-progress classify runs I quoted, the provider_error half is a genuine fallback-chain exhaustion, but the incomplete: N remain (was N at run start; processed 0 this run) half — 15 projects' current last_error — may be this same abort rather than a classifier failure, since a task that never gets dispatched cannot report progress.

So the histogram I posted is a snapshot of a store whose classifier has been switched off for four days. The 61% figure and the 72 distinct values are still the honest reading of what did get classified, but "peaked relative to yours" is now better explained by the pool having grown unclassified under a stalled task than by the model's spread.

Two consequences for the fixes you took:

The histogram in /ctx-status would have caught this immediately, and it is the more valuable half of the two. A tick that aborts before per-project work currently looks exactly like a dreamer with nothing to do.
The tie-break fix matters more on this store than I first argued, because the importance = 50 band is the unclassified pile here — 692 of 704 — so it is precisely the newest memories being ordered by id.

I have not touched the local config or the installed bundle; the issue is filed with the reproducer and the source lines. If it is useful I can also export the 25 dream_runs rows so you can separate the provider-error half from the abort half.

Thank you for correcting your own headline in public and then finding the actual cause — the split table plus the classified_at query is exactly what closes this thread, and #496 is the real finding. Confirmed at source and acknowledged there; the fix is in progress and goes out as a patch release.

On the classifier itself, your classified-only column settles it: 72 distinct values and 12.7% at exactly 50 is a discriminating classifier on a pool that grew under a dead scheduler. Nothing further to change on scoring.

Two things you can rely on for the export offer: the incomplete: N remain (was N at run start; processed 0 this run) string is written by a run that was dispatched and processed nothing, so on your store it predates the Sep 17 abort (it cannot be produced by a tick that never reaches the task); the abort itself leaves last_status untouched, which is the None: 364. So the 25 zero-progress runs split cleanly by date against 2026-09-17 21:03: before it, provider-chain exhaustion (the ClinePass line) and the incomplete runs are both real classify outcomes; after it, nothing ran. No export needed unless you see a zero-progress row dated after the abort.

Your importance != 50 AND classified_at IS NULL signal for edited-after-classification is correct and worth keeping; that is by design in rewriteMemoryContent. Both fixes you took stand: the recency tie-break is on master, and the histogram with an unclassified denominator is on master and will show the "never ran" state explicitly once #496 lands, since that state is being surfaced in the same /ctx-status view.

Independent confirmation from a third store (small pool, 96 active), with one clarification that may save the next reader a wrong turn:

The importance != 50 AND classified_at IS NULL signal is real here too — but on inspection every row matching it was content-edited after the most recent classify run, and classify's backlog is exactly classified_at IS NULL. So edited memories are not skipped; they re-enter the next run's backlog and get re-scored there. What looked like "stale scores forever" was just "stale scores until the next daily classify" — worth stating explicitly since the failure mode (default-looking 50s, or an old score on rewritten content) reads exactly like a classifier defect in between runs.

One asymmetry that is by design but worth documenting alongside the histogram: a content edit triggers an immediate re-embed (observed: same-second proactively embedded memory N after each ctx_memory update), while importance re-scoring waits for the next classify cycle — so for up to a day, search sees the new content but injection ordering still weighs the old judgment. Fine for a daily batch design; just the two halves of "edit" running on different clocks.

Thanks, that's the right reading. An edit clears classified_at, so the memory goes back into the next classify run's backlog and is re-scored there; until that run it keeps its previous score. The re-embed happens right away because search needs the new content to be findable, while importance is a batch judgement over the pool. So for up to a day after an edit, search sees the new text and injection ordering still uses the old score, as you describe. We'll mention that in the memory docs next to the histogram.

Everything this issue tracked has shipped, all in 0.43.0:

equal-importance memories tie-break on verification and last-seen recency on OpenCode, Pi and the Rust module, not on creation order;
/ctx-status shows a per-band importance histogram with an unclassified count;
the dreamer runs on Pi and OMP again (Dreamer never runs on the OMP harness: an unguarded harness throw aborts the whole timer tick #496), so unclassified memories get classified;
the memory docs say that an edit is re-embedded right away and re-scored at the next classify run.

The hosted decision provider stays declined, as discussed. Pluggable classification belongs to the policy-pack work in #358.

Navigation Menu

Repository navigation

Optional decision-provider for memory importance classification (cache-neutral, default off) #474

Short description

Problem: what does this solve?

Proposed solution

The proposal: Jev as the classify-memories producer

The full decision-point map (all surveyed, with evidence)

Alternatives considered

Area

Additional context

Your query on my store

But your task-health finding reproduces here, and it is worse than classify

On the two things you took

On the provider

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Optional decision-provider for memory importance classification (cache-neutral, default off) #474

Description

Short description

Problem: what does this solve?

Proposed solution

The proposal: Jev as the classify-memories producer

The full decision-point map (all surveyed, with evidence)

Alternatives considered

Area

Additional context

Activity

magic-alfonso commented on Sep 20, 2026

Qiiks commented on Sep 21, 2026

Your query on my store

But your task-health finding reproduces here, and it is worse than classify

On the two things you took

On the provider

Qiiks commented on Sep 21, 2026

magic-alfonso commented on Sep 21, 2026

G0-0000 commented on Sep 23, 2026

magic-alfonso commented on Sep 23, 2026

magic-alfonso commented on Sep 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions