Skip to content

Webhook retries have no jitter, so failures synchronise #341

Open

Description

@Miracle656

Background

src/alerts/delivery.ts backs off deterministically:

function retryDelayMs(attempt: number): number {
  return Math.pow(2, attempt - 1) * 1000
}

Every delivery that fails at the same moment retries at exactly the same moment
— 1s, then 2s, then 4s. A price movement is precisely the event that triggers
many alerts at once, so when a subscriber's endpoint has a wobble, Lens responds
by hitting it with a synchronised burst, three times, which is a good way to
turn a wobble into an outage.

What to build

  • Add jitter. Full jitter — a random value between zero and the computed delay —
    is the standard choice and is one line.
  • Keep the existing retry count and the existing rule that 4xx is not retried,
    which is correct.
  • Make the test deterministic by injecting the randomness rather than asserting
    on a range.

Acceptance criteria

  • Retry delays are jittered
  • Two deliveries failing simultaneously retry at different times, asserted by a test
  • 4xx responses are still not retried
  • The attempt cap is unchanged
  • The randomness source is injectable so the test is not flaky

Drips Wave · Complexity: Easy · 100 points


Repo: Divinity-Labs/Lens. Node + TypeScript, Fastify, Prisma on Postgres,
Redis for queues, Prometheus via prom-client.

Run the tests: npm test. The tests are vitest, not jest. Running jest
against this repo reports "66 suites, 0 tests" and means nothing. A single file
is npx vitest run src/__tests__/yourfile.test.ts.

Load test (prices) is already failing on main and has been for some
time. It is not your PR that broke it. Every other check should be green.

This service prices real assets on Stellar mainnet. A number that is wrong
is worse than a number that is missing, because a caller cannot tell the
difference. Prefer failing loudly over returning a plausible value.

Never elide an address, a contract id or a key in code, a comment or a test
fixture.
A previous batch wrote a shortened contract id in an issue text and
five contributors invented plausible-looking ids to fill the gap. Write the
whole thing or do not write it at all.

The house pattern for an outbound deadline already exists — see
src/alerts/delivery.ts, which passes signal: AbortSignal.timeout(10_000) to
fetch. Where an issue below asks for a timeout, copy that, do not invent a
second convention.


Required: Before submitting, join the contributor Telegram so your work can be tracked and counted toward the Stellar Wave: https://t.me/+fxHXq8f1SwlkZDBk

Activity

  1. Richard-tobi commented on Oct 16, 2026

    @Richard-tobi

    @Richard-tobi has applied to work on this issue as part of the Stellar Wave Program's 10th wave.

    Hello, I’m interested in contributing to this issue and would appreciate the opportunity to work on it. I understand the need to introduce full jitter into the webhook retry delays to prevent simultaneous failures from causing synchronized retry bursts. My approach would be to update the retry logic in src/alerts/delivery.ts, inject the randomness source to ensure deterministic testing, and preserve the existing retry count and rule that 4xx responses are not retried. I’ll also add tests to verify that simultaneous failures retry at different times and that the attempt cap remains unchanged. I’m committed to delivering a clean, reliable, and well-tested solution that meets the acceptance criteria. Please consider assigning this issue to me. Thank you!

    ℹ️ Repo Maintainers: To accept this application, review their application or assign @Richard-tobi to this issue.

  2. drips-wave commented on Oct 16, 2026

    @drips-wave

    Congratulations, @Richard-tobi! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on October 22, 2026.

    🧑‍💻 @Richard-tobi: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

    Warning

    When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

    🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

    🌊 Happy Wave 🌊

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions