Raised while measuring #335. Not folded into that PR — the exposure is created by the migration mechanism and the Helm chart, not by any one migration.
What happens now
runMigrationsOn (app/migrations/migrations.go:60-76) executes each statement as a bare target.Exec: no transaction, no timing, no logging. migrations.Run is called synchronously from cmd/run.go before the HTTP server binds, so a slow statement is boot latency with zero operator feedback — the process looks hung.
Index builds are the slow statements. Measured on a 2M-row / 8.1GB telemetry database (#335): a single CREATE INDEX on exception_stack_traces took ~10s, and the DROP before it another ~2s.
Why it is worth fixing now
#334 (0020) builds three indexes and #335 (0021) builds a fourth. An operator upgrading across both runs all four in one boot, on the same database, before anything listens on a port.
helm/traceway/templates/deployment.yaml:54-61 has no startupProbe — verified, there is no startupProbe anywhere in helm/. There is only:
livenessProbe:
initialDelaySeconds: 10
periodSeconds: 30
failureThreshold: 3
That is roughly a 70-100s budget from container start before the kubelet kills the pod. Four index builds on a large database plus normal startup can plausibly reach it.
The failure is not self-correcting. The version row is recorded by a separate Exec after the statements (migrations.go:73), outside any transaction, so a kill mid-CREATE INDEX never records the version. The pod restarts, re-runs the same build, and is killed again at the same point. Both statements are idempotent so there is no corruption — but the crash-loop does not resolve on its own, and it looks like a failing image rather than a slow migration.
Related smaller hole: 0021's DROP and CREATE are separate statements with no transaction, so for the seconds between them the table carries no (project_id, exception_hash) index at all. Only reachable if the process dies in that window, and the rerun repairs it.
What "done" looks like
Roughly in cost order:
config.Logf per migration in runMigrationsOn, with elapsed time. Turns a silent stall into a log line. Cheapest thing here and useful regardless of the rest.- A
startupProbe in the Helm chart, with a failureThreshold sized for a large-database migration, so the liveness budget only starts after boot completes. This is what startupProbe is for.
- Optionally reconsider recording the version in the same transaction as the statements, so a killed migration is atomic. Note SQLite DDL is transactional, so this is feasible — but it changes behaviour for every migration, not just index builds, so it deserves its own look.
Docker/compose deployments have no equivalent probe and just see a slow start, so (1) is the fix that helps every deployment shape.
Raised while measuring #335. Not folded into that PR — the exposure is created by the migration mechanism and the Helm chart, not by any one migration.
What happens now
runMigrationsOn(app/migrations/migrations.go:60-76) executes each statement as a baretarget.Exec: no transaction, no timing, no logging.migrations.Runis called synchronously fromcmd/run.gobefore the HTTP server binds, so a slow statement is boot latency with zero operator feedback — the process looks hung.Index builds are the slow statements. Measured on a 2M-row / 8.1GB telemetry database (#335): a single
CREATE INDEXonexception_stack_tracestook ~10s, and theDROPbefore it another ~2s.Why it is worth fixing now
#334 (
0020) builds three indexes and #335 (0021) builds a fourth. An operator upgrading across both runs all four in one boot, on the same database, before anything listens on a port.helm/traceway/templates/deployment.yaml:54-61has nostartupProbe— verified, there is nostartupProbeanywhere inhelm/. There is only:That is roughly a 70-100s budget from container start before the kubelet kills the pod. Four index builds on a large database plus normal startup can plausibly reach it.
The failure is not self-correcting. The version row is recorded by a separate
Execafter the statements (migrations.go:73), outside any transaction, so a kill mid-CREATE INDEXnever records the version. The pod restarts, re-runs the same build, and is killed again at the same point. Both statements are idempotent so there is no corruption — but the crash-loop does not resolve on its own, and it looks like a failing image rather than a slow migration.Related smaller hole:
0021'sDROPandCREATEare separate statements with no transaction, so for the seconds between them the table carries no(project_id, exception_hash)index at all. Only reachable if the process dies in that window, and the rerun repairs it.What "done" looks like
Roughly in cost order:
config.Logfper migration inrunMigrationsOn, with elapsed time. Turns a silent stall into a log line. Cheapest thing here and useful regardless of the rest.startupProbein the Helm chart, with afailureThresholdsized for a large-database migration, so the liveness budget only starts after boot completes. This is whatstartupProbeis for.Docker/compose deployments have no equivalent probe and just see a slow start, so (1) is the fix that helps every deployment shape.