live.setkernel.com: show when an engineer's VM stops reporting, instead of calling it Idle

Why

live.setkernel.com is the only demo (setkernel/hq decision 0008, point 4). Today it can't tell a dead VM from an idle one. Checked on main:

The page knows five engineer states: WORKING, REVIEW, DISCUSS, IDLE and PAUSED (src/logic.js:13; setkernel/hq observability/wall/public.py:36). None of them means "not reporting".
sk-obs01 reads each engineer's state from the sk-agent journal. After 10 minutes with no journal line or Claude call, it sets the engineer to IDLE (setkernel/hq observability/wall/server.py:45, :518-520).
Idle cycles write nothing to the log (setkernel/hq agents/vm/sk-agent-watch:8). So a VM whose sk-agent has stopped looks exactly like one with nothing to do.
An engineer missing from the snapshot also shows as idle (public/lib/words.js:201).
The only liveness on the page covers the whole office: "No word from the office since …" after 120 s with no push (words.js:37, :230).
Per engineer, the last-event time (seen) only sorts the headline (words.js:233). The Now column says "Idle" with no age (public/lib/day.js, moment()).
server.py keeps a per-host last, but it is the time the line was received, even during the 30-minute history replay (:368, :467). It is never published.
Real case: vega and orion ran no cycle for about 12 hours on 2026-10-16 (setkernel/hq#246). By this code the public page showed them both as Idle the whole time.

What is already tracked:

Live: Buzz #infra alert when updates stop (Cron Trigger) #23 (the PR for Live: Buzz #infra alert when updates stop (Cron Trigger) #9, waiting on Raghav's Buzz identity) alerts #infra when the whole publisher or house stops pushing. It does nothing per VM and nothing on the page.
setkernel/hq#251 alerts #infra when one VM's sk-agent has no run for 15 minutes, from sk_agent_last_run_end_seconds in sk-agent-watch's .prom file. That alert is internal only. Its 2026-10-20 comment says those series aren't in Prometheus yet, because agent-vm.sh hasn't been re-run on the hosts.
Untracked (this issue): the public page saying, per engineer, that the VM has stopped reporting and when it last ran.

Do

Prototype first (agent): two or three options for how a not-reporting engineer looks on Day. Cover the Now column (for example "Not reporting since 14:05"), the lane, the headline sentence and ?wall. Add a ?demo fixture with one engineer down. Put screenshots at 1440×900, 375×812 and ?wall 1920×1080 in the PR, with the prototype openable.
Raghav decides the option, and whether this new public fact may leave sk-obs01. It would be per engineer and coarse: no hostnames, no metric values, and minute-rounded times only. He comments the choice here.
setkernel/hq PR (agent), wall server (setkernel/hq#396): setkernel/hq#398, merged and deployed on sk-obs01 2026-10-24 12:38 UTC
- Read per-host liveness from Prometheus, by host, as the wall's other queries do (server.py:406-410). The series are sk_agent_last_run_end_seconds, sk_agent_running, sk_agent_paused and sk_agent_watch_checked_seconds.
- Mark an engineer not reporting when no run has finished in 15 minutes and none is in progress (the setkernel/hq#251 threshold), or when the watch itself has stopped writing.
- Use the metric's own time, not the receipt time.
- If Prometheus or the series are missing, keep today's behaviour and never claim a VM is down.
- public.py sends only the new state and a minute-rounded last-run time.
- Tests in test_server.py and test_public.py.
setkernel-live PR (agent): live.setkernel.com: show when an engineer's VM stops reporting, instead of calling it Idle #57, merged and deployed 2026-10-24 12:34 UTC
- Accept the new state in src/logic.js (STATES, plus the optional time, validated). Don't name it OFFLINE: the page already uses "offline" for the whole office.
- Carry it through public/lib/state.js view().
- Render the chosen option in words.js and day.js.
- Update STATE_KEYS in scripts/smoke.mjs if the shape changes.
- Tests in test/logic.test.mjs, test/page-logic.test.mjs and test/day.test.mjs.
- If the v4 showcase (Live v4: the showcase that wins customers (prototype first) #52) lands first, carry the state into it.
Dependency, Raghav's step in setkernel/hq#251: re-run agent-vm.sh on the hosts, so the sk_agent_* series reach Prometheus. The two PRs can merge before this, because missing data fails safe.
Drill, with Raghav's OK on which seat: stop sk-agent.timer on one VM for 20 minutes. Record on this issue when the page showed that engineer as not reporting, and when it went back after the restart.

Done when

With one VM's sk-agent stopped, live.setkernel.com shows that engineer as not reporting, with when it last ran, within 20 minutes. The other engineers still show Idle or their work. It returns to normal within one cycle after the restart. The drill is recorded on this issue.

Refs setkernel/hq#251, setkernel/hq#246, #23, setkernel/hq#105, setkernel/hq#213

— Claude Lead

Released (agent-ready). This round only: box 1, the prototype options with inline screenshots (phone and desktop, light and dark). Then stop for Raghav's pick: he decides from the screenshots, so build nothing under public/ yet.

— Claude Lead

@rigel-setkernel

Picked up. Working on branch agent/54-live-setkernel-com-show-when-an-engineer. Automated: @rigel-setkernel's agent on rigel-dev.

Raghav's picks, 2026-10-24. He made them at 04:12–04:13 UTC on his v4c Owner Desk page in the Claude Lead session, after reading setkernel-live#56's screenshots. They're relayed word for word, and he added no notes.

How Day shows a VM that stopped reporting: "C · Flagged". The PR recommended B.
May "not reporting since HH:MM" leave sk-obs01 and appear on the public page: "Yes".
The new state's name: "SILENT".
An outage longer than a day: "Keep Day's time format".
After the VM comes back: "No: the hatch goes".
The ?demo fixture: "Prototype switcher now, ?demo=down in the build".

setkernel-live#56, the prototype, is approved and merges next. The build round on this issue follows from these picks, with a separate setkernel/hq change for sk-obs01's SILENT signal. Claude Lead hands both out once #56 is on main.

— Claude Lead

Build round for setkernel-live#54, released (agent-ready). The prototype round is done: #56 merged at fa2b898, and its files are in docs/design/v3/not-reporting/. The earlier instruction to build nothing under public/ no longer applies. This round builds Raghav's picks under public/ and src/. sk-obs01's half, sending SILENT, is a separate issue: setkernel/hq#396.

Raghav's picks (2026-10-24, on his Owner Desk page, relayed word for word in the comment above):

How Day shows a VM that stopped reporting: "C · Flagged". The PR recommended B; build C.
May "not reporting since HH:MM" leave sk-obs01 and appear on the public page: "Yes".
The new state's name: "SILENT".
An outage longer than a day: "Keep Day's time format".
After the VM comes back: "No: the hatch goes".
The ?demo fixture: "Prototype switcher now, ?demo=down in the build".

What to build (option C, docs/design/v3/not-reporting/README.md:31-45)

src/logic.js:
- Add SILENT to STATES (:13).
- In agentInfo (:52-58), keep an optional lastRun for SILENT only, by the prototype's cleanSilent rule (src/scenario.js:14-23): an integer, a whole minute, not in the future and not older than 7 days. Otherwise drop the time and keep the state, so the page says "Not reporting" with no time.
- The rule covers both batch agents (:104) and agent events (:82).
- cleanBatch takes no clock today (:98), and the future and 7-day checks need one.
public/lib/words.js:
- The state's words: "Not reporting" in STATE_WORD (:35). SILENT stays out of ACTIVE (:36). Give it a text glyph in GLYPH (:124) too, which only stateWords() uses (:243). The page draws the dotted ring from the #s-SILENT symbol (below).
- The headline takes C's form (src/proto-words.js:38-62). The stopped engineers come first ("Orion stopped reporting at 14:05."), then the others as today.
- Two names share one sentence, each with its own time. With no time, it says "Orion is not reporting."
- "The engineers are between tasks." becomes "The other engineers are between tasks."
- Offline and waiting say only what they say today (:227-232). The headline also reaches document.title (app.js:51), and the scrub readout (#readout, aria-live, index.html:49) says "Not reporting" (below). The #announce region keeps speaking only GitHub events and requests (words.js:262-272).
public/lib/day.js:
- The Now column (moment, :193-210) says "Not reporting", then "Since 14:05 · 2 h 35 min" (src/proto-day.js:14-31).
- When the office is offline, the "Last word, …" column shows the stopped engineer as they were at the last word.
- When scrubbed, the ten-minute columns after lastRun say "Not reporting", and so does the readout.
- The key (keyItems, :235-244) gets "Not reporting" with the red hatch, only when one is drawn.
Pick 4: write the time with Day's own dayOf/whenDay (day.js:164-165): "14:05", "yesterday 21:40" or "Fri 16 Oct 04:10". Add nothing for outages longer than a day. Don't carry over the prototype's copy of that code (src/proto-words.js:11-17).
public/lib/chart.js (Orion's nit on live.setkernel.com: show when an engineer's VM stops reporting, instead of calling it Idle #56):
- Draw the hatch in chart.js with its own geometry: extend its hatch() helper (:103-110 wide, :241 tall) with a signal class. Don't carry over src/proto-chart.js's recomputed GRID, row heights and top + 68 track offset.
- On the lane, the faint track stops at lastRun with a red end cap.
- The red hatch runs from lastRun (or the window's start when there's no time) to now. When the office is offline it stops at the office's last word, where the office's own hatch takes over (:112).
- The words inside the lane are as in proto-chart.js:53-62.
- Lyra's nit: in the wall shot, the Claude/chat wave overlapped the start of the hatch by a few pixels. The wave should stop at lastRun.
- On the phone, draw the column's hatch and end cap as in proto-chart.js:70-94.
public/index.html and public/style.css:
- Add three defs next to #hatch (index.html:72-76), from the prototype's build.mjs:74-77: the s-SILENT symbol (the dotted ring, which chart.js draws with <use href="#s-${state}"> at :170 and :302), the hatch-sig pattern, and the key swatch m-silent-c (drawn by #m-${it.kind} at :345).
- Add .f-hatch-sig, .hatch-sig-line, .s-hatch-edge and the Now column and Now list classes (src/proto.css:15-17, :25-27).
- Every SVG colour is a class, never a style attribute (AGENTS.md:85-87).
Pick 6, ?demo=down (public/app.js:17, :198-233; public/lib/demo.js):
- It is ?demo's busy day with Orion stopped 2 h 35 min before now, built as src/scenario.js:28-50 builds it.
- His own events end at the stop, but a merge of his pull request still arrives from GitHub.
- He reads IDLE for the first 15 minutes, then SILENT with lastRun.
- ?demo, ?demo=quiet, ?demo=offline and ?demo=waiting don't change. The prototype and its switcher in docs/ stay as they are.
Not this round:
- No dark page (README.md:16-17; test/page.test.mjs:119-125).
- No replay column, per pick 5. state.js, the 128 KiB budget and STATE_KEYS in scripts/smoke.mjs don't change (README.md:107-109, :127-129).

Contrast

C makes --signal a text colour: 5.50:1 on the page's --surface and 5.99:1 on the wall's (README.md:96-99), and 6.03:1 and 6.60:1 on --bg (computed from style.css:13-15, :25-27). That passes AA, and is under the 7:1 that test/page.test.mjs:101 asks of --ink and --ink-2.
Raghav's Owner Desk showed him that cost ("Red text is 5.5:1"), with C's other two (README.md:96-97): it reads as an incident banner, and its red competes with the red of review feedback and failed CI. He picked C knowing all three, so build it as shown, with --signal as the text colour. Don't swap in another red. If a reviewer wants 7:1 here, ask Raghav on this issue.
Add signal to TEXT (test/page.test.mjs:92), so it is checked as text at 4.5:1 on both grounds, on the page and the wall. Today it is only in MARKS (:93), so fill: var(--signal) on text passes the "every text colour is checked" test (:112-117) at 3:1.

The data: sk-obs01's new SILENT signal (setkernel/hq#396).

Per engineer, it sends { "state": "SILENT", "lastRun": <seconds, on a whole minute> }.
It is sent when no run has ended in 15 minutes and none is in progress, or when the watch has stopped writing.
Nothing sends it yet, so after this PR every engineer still reads as today, and this PR can merge first. (If the hq PR merges first, today's page drops the unknown state, src/logic.js:104-105, and the engineer reads Idle, words.js:201.)

Tests (both allowlist suites, AGENTS.md:65-67)

test/logic.test.mjs and test/page-logic.test.mjs:
- SILENT with and without lastRun;
- lastRun dropped and the state kept when it isn't an integer, isn't a whole minute, is in the future or is older than 7 days;
- extra fields dropped, NEVER still dropping the agent, and lastRun on any other state dropped.
test/page-logic.test.mjs:
- ?demo=down's events pass cleanEvent unchanged and its agents pass cleanBatch unchanged, as busyDay's do (:267-271);
- C's headline with one engineer down, two down and no time, with offline and waiting unchanged.
test/day.test.mjs: the Now column with a time today, yesterday and on an older date; the scrubbed columns after lastRun; the key entry only when drawn; the other engineers unchanged.
test/page.test.mjs: signal in TEXT, checked at 4.5:1 on both grounds, on the page and the wall.
test/worker/: one batch with SILENT and lastRun reaches /api/state unchanged.
Run npm run validate (check, test, test:worker, wrangler deploy --dry-run) and report each step's result.

Screenshots:

Add ?demo=down to docs/design/v3/day/built/shoot.mjs (:48-60) and commit its shots: 1440×900, 375×812, 375 full page, ?demo=down&wall 1920×1080, and one scrubbed to after the stop.
Put them inline in the PR beside the prototype's c-*.png, so the build can be checked against the pick.
The existing ?demo shots and og.png don't change.

PR body: Refs setkernel/setkernel-live#54. The drill closes the issue, not this PR.

— Claude Lead

@rigel-setkernel

Picked up. Working on branch agent/54-live-setkernel-com-show-when-an-engineer. Automated: @rigel-setkernel's agent on rigel-dev.

feat: accept SILENT, an engineer whose VM stopped reporting, with its…

feat: show a VM that stopped reporting on Day, option C, and ?demo=do…

docs: ?demo=down screenshots, and README and AGENTS for SILENT (#54)

Both PRs are merged and deployed, and their two boxes are ticked:

setkernel-live#57: merged at 12:34 UTC. The deploy's smoke check passed.
setkernel/hq#398: merged at 12:35 UTC. sk-obs01 deployed it at 12:38 UTC, and the wall's healthz answers 200.

Still open:

setkernel/hq#251's host step (Raghav): until it runs, the sk_agent_* series aren't in Prometheus, so no tile can read SILENT. Missing data fails safe.
The drill: it waits for step 1 and Raghav's OK on which seat.

— Claude Lead

fix: restore the frozen #54 prototype, trim only its font stacks (#50)

Navigation Menu

Repository navigation

live.setkernel.com: show when an engineer's VM stops reporting, instead of calling it Idle #54

Why

Do

Done when

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

live.setkernel.com: show when an engineer's VM stops reporting, instead of calling it Idle #54

Description

Why

Do

Done when

Activity

realRaghavGupta commented on Oct 21, 2026

rigel-setkernel commented on Oct 21, 2026

realRaghavGupta commented on Oct 24, 2026

realRaghavGupta commented on Oct 24, 2026

rigel-setkernel commented on Oct 24, 2026

realRaghavGupta commented on Oct 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions