Skip to content

cloudflare-export: alert on Workers AI neurons/day over threshold — the Oct 24–25 burn (4.11M neurons, ~$45) was invisible for days #2287

Closed

Description

@dezren39

Problem

The fleet burned 4,114,515 Workers AI neurons in October 2026 (~$44.95 estimated) and nobody noticed for days. Workers AI is pay-as-you-go beyond the 10k free neurons/day allowance — it is not a plan pool. A same-day alert on the dataset the new collector already reads would have caught this on day one.

Verified facts (Workers AI cost investigation, 2026-10-29)

  • 4,114,515 neurons total for October; 99.2% on one model — @cf/deepseek-ai/deepseek-v4-pro-0813 — reached via the cloudflare-gateway provider in the laptop's OpenCode config.
  • Two-day burst: Oct 24 ≈ 2.99M neurons, Oct 25 ≈ 1.11M neurons. Source: OpenCode sessions under ~/work/id-train/ (the "id train" and "pm waves" sessions) with the Cloudflare model explicitly selected.
  • Mechanism: every agent turn re-sent a ~200–500k-token context, and Workers AI bills the full input on every call — 336M cache-read tokens were processed and billed, not free.
  • The three Cloudflare providers (cloudflare-gateway, cloudflare-workers-ai, cloudflare-ai-gateway) are now in disabled_providers; usage has been zero since Oct 25. This issue is about detection, not the (already stopped) burn.

Ask

The Cloudflare collector (#1048, pm-stack/cloudflare-export/, PR #2228) already reads the aiInference dataset every 15 minutes, but it has no threshold alert. Add one:

  • Compute Workers AI neurons/day from the same dataset the collector already ingests.
  • When a day's total crosses a configurable threshold (suggested default: >50k neurons/day — 5× the free daily allowance, far below the 2.99M single-day burn), surface it loudly the same day through the collector's existing reporting path.
  • Document the threshold, how to change it, and where the alert lands.

Acceptance criteria

  • Alert fires on a synthetic/seeded over-threshold day (test fixture or replayed Oct 24-shaped data).
  • No alert on normal days — days at or under the 10k free allowance stay silent.
  • Threshold is configurable without a code change; the default and the alert destination are documented (collector README or config comments).
  • Alert identifies the day total and the top model(s) by neurons, so the next burn is attributable immediately.

Related: #1048 (collector), PR #2228.

Activity

  1. added
    task/triageQueue packet kind: verify, reproduce, and route this item
    source/queue-workerFiled or worked by the queue/worker pilot
    priority/highImportant and time-sensitive
    queue/queuedHas a queued task in the Supabase agent queue
    effort/sSmall: expected under 2 hours of work
    on Oct 29, 2026
  2. developing-today commented on Oct 29, 2026

    @developing-today
    Contributor
  3. dezren39 commented on Oct 29, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · filed

  4. added
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 1, 2026
  5. 112 remaining items

  6. added
    status/parkedDeliberately paused with a recorded resume condition
    and removed
    queue/claimedQueue worker lifecycle state
    on Nov 2, 2026
  7. dezren39 commented on Nov 2, 2026

    @dezren39
    MemberAuthor

    agent-note:v1 — parked
    Agent: repo-worker-code (OpenChamber scheduled lane)
    Task: repo-health-code-pilot task 5234
    At: 2026-11-02T12:50:20Z
    State: parked
    Fresh evidence: #2287 OPEN and assigned to dezren39; task-bound PR #4126 OPEN, non-draft, authored/assigned to dezren39.
    Disposition: parked for 30 minutes at the Drew-owned hard stop. No PR contents, branch, worktree, or checks inspected. work/assigned remains set.
    Resume: explicit ownership handoff/release; revalidate fresh before action.
    Links: #2287 · #4126 · task 5234

  8. added
    queue/claimedQueue worker lifecycle state
    and removed
    status/parkedDeliberately paused with a recorded resume condition
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    on Nov 2, 2026
  9. dezren39 commented on Nov 2, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · picked up

  10. added a commit that references this issue on Nov 2, 2026
  11. dezren39 commented on Nov 2, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

app/pm-stackpm-stack (Pad/Grist) in developing-today/codeeffort/sSmall: expected under 2 hours of workkind/infrastructureInfrastructure / host-portability workpriority/highImportant and time-sensitivesource/queue-workerFiled or worked by the queue/worker pilottask/triageQueue packet kind: verify, reproduce, and route this item

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions