[Postmortem] Stripe Webhook Duplicate Delivery: How a Retry Cost Us 18 Customers

Written by Huifer, solo developer and maintainer of TanStack Ship. I handled this production billing incident after a Stripe retry exposed a missing idempotency boundary: 18 customer accounts received duplicate credits and two subscriptions were downgraded incorrectly. I reversed the financial impact within four hours. The lasting lesson was not about Stripe being unreliable; it was that my handler ignored the delivery contract Stripe documents, and customers paid the operational cost.

Verified sources: TanStack Ship GitHub · Stripe webhook documentation · Stripe duplicate-event guidance Updated: 2026-07-20 · Changelog

TL;DR: A retried invoice.paid event passed through a handler that had no durable claim on event.id. Eighteen accounts received the same credit twice, and a non-atomic downgrade path affected two subscriptions. I repaired the accounts, added a unique event ledger, made downstream mutations idempotent, and added replay tests. Stripe promises at-least-once behavior, not exactly-once delivery; the receiver must make repetition safe.

Background: a correct signature check was not enough

The affected route accepted Stripe webhooks for invoice and subscription lifecycle events. It verified the signature against the raw request body, translated the Stripe object into an internal command, wrote a credit-ledger entry, updated subscription state, and appended an audit record. That sequence looked disciplined in code review because every individual operation had validation and error handling.

It lacked one system-level property: processing the same event twice did not produce the same result as processing it once. Signature verification answers, “Did Stripe send this payload?” It does not answer, “Have I already applied this event?” Those concerns were conflated in my original implementation.

Stripe’s documentation tells integrators to expect automatic retries and occasional duplicate events. The incident therefore was not an undocumented provider surprise. The failure was mine: I had implemented authenticity but not idempotency. TanStack Ship’s current billing features separate those controls explicitly.

The unsafe processing path

The original order of operations was effectively:

1. verify the signature;
2. read the account balance;
3. add credits associated with the invoice;
4. inspect and possibly apply a pending downgrade;
5. write an audit row;
6. return HTTP 200.

There was no table keyed by Stripe’s event identifier. The credit entry also lacked a unique source-event constraint. If two deliveries overlapped, both could read the same pre-update state and both could commit a valid-looking mutation.

Incident and customer impact

A transient failed response caused Stripe to redeliver an invoice.paid event. The retry carried the same Stripe event.id, as expected. My endpoint treated it as new work. Additional retry deliveries encountered the same missing guard before I disabled the affected mutation path.

The bounded impact from the incident record was:

• 18 customer accounts received duplicate internal credits;
• 2 subscriptions were moved to the wrong lower tier after duplicate downgrade handling;
• the account-side financial impact was reversed within four hours;
• no claim is made here about an invented loss amount, because customer balances and prices are private;
• no customer names, invoice identifiers, or payloads are published.

The phrase “cost us 18 customers” in the title describes 18 affected customer relationships, not 18 cancellations. I cannot responsibly claim that every affected customer churned. That distinction matters in an incident report.

Detection came from reconciliation, not a perfect alert

The first useful signal was a mismatch between invoice events and credit-ledger rows: one source invoice corresponded to more than one credit. The Stripe delivery history then showed repeated deliveries with the same event identifier. Finally, the subscription audit trail exposed two downgrade transitions for the same billing source.

My monitoring could report handler errors, but it could not report “one external fact produced two internal effects.” That is why HTTP error-rate monitoring alone did not protect the business invariant. The durable ledger added later made duplicate attempts observable as a normal counter rather than a customer-visible surprise.

Investigation: follow identity across every write

I investigated the event as a data-lineage problem. Starting with the Stripe event identifier, I traced every internal row it should have created. The credit table could point to an invoice, but did not require a unique event source. The downgrade update had a read-then-write shape. The audit table described actions after they happened, but it was not the authority that permitted them.

Replaying a redacted event twice reproduced both defects. Sequential replay duplicated the credit because no uniqueness rule existed. Concurrent replay widened the timing window around the downgrade check. The same business fact was accepted twice through two independently valid requests.

Root cause, contributing factors, and non-causes

The root cause was the absence of a database-enforced idempotency key based on Stripe event.id. A duplicate delivery could enter business logic without first obtaining an exclusive claim.

Three factors contributed:

• credit mutations did not carry a unique source_event_id;
• downgrade handling used “check, then write” instead of one atomic statement;
• tests covered different events, malformed signatures, and handler failures, but not repeated delivery of the same valid event.

Stripe retries were a trigger, not the cause. D1 was not the cause either; another SQL database would permit the same race if the schema and transaction boundary were unchanged. No person is blamed. The design review focused on happy-path correctness and missed a distributed-systems invariant.

The durable fix: claim, mutate, complete

The repair begins with a webhook event ledger. Its primary key makes identity durable. Business tables also retain the source identifier, giving the system a second line of defense and an audit path.

CREATE TABLE stripe_webhook_events (
  event_id TEXT PRIMARY KEY,
  event_type TEXT NOT NULL,
  status TEXT NOT NULL CHECK (status IN ('processing', 'completed', 'failed')),
  attempts INTEGER NOT NULL DEFAULT 1,
  received_at INTEGER NOT NULL,
  processed_at INTEGER,
  last_error TEXT
);

CREATE TABLE credit_ledger (
  id TEXT PRIMARY KEY,
  account_id TEXT NOT NULL,
  amount INTEGER NOT NULL,
  source_event_id TEXT NOT NULL UNIQUE,
  created_at INTEGER NOT NULL,
  FOREIGN KEY (source_event_id) REFERENCES stripe_webhook_events(event_id)
);

The handler verifies the raw payload first, then attempts an atomic claim. A failed event may be reclaimed; a processing or completed event cannot. Business writes and the completion marker are committed together through a D1 batch. Project-specific mapping functions are omitted, but the control flow is complete.

export async function handleStripeWebhook(request: Request, env: Env) {
  const rawBody = await request.text()
  const signature = request.headers.get('stripe-signature') ?? ''
  const event = await verifyStripeEvent(rawBody, signature, env.STRIPE_WEBHOOK_SECRET)
  const now = Date.now()

  const claim = await env.DB.prepare(`
    INSERT INTO stripe_webhook_events
      (event_id, event_type, status, attempts, received_at)
    VALUES (?, ?, 'processing', 1, ?)
    ON CONFLICT(event_id) DO UPDATE SET
      status = 'processing', attempts = attempts + 1, last_error = NULL
    WHERE stripe_webhook_events.status = 'failed'
    RETURNING event_id
  `).bind(event.id, event.type, now).first<{ event_id: string }>()

  if (!claim) return new Response('Already accepted', { status: 200 })

  try {
    const statements = buildIdempotentBillingStatements(env.DB, event, now)
    statements.push(env.DB.prepare(`
      UPDATE stripe_webhook_events
      SET status = 'completed', processed_at = ?
      WHERE event_id = ? AND status = 'processing'
    `).bind(Date.now(), event.id))
    await env.DB.batch(statements)
    return new Response('Accepted', { status: 200 })
  } catch (error) {
    await env.DB.prepare(`
      UPDATE stripe_webhook_events SET status = 'failed', last_error = ?
      WHERE event_id = ?
    `).bind(String(error), event.id).run()
    return new Response('Retry later', { status: 500 })
  }
}

The credit_ledger.source_event_id uniqueness constraint remains essential. The event claim controls execution, while the business-table constraint protects the money-like effect if a future refactor bypasses the wrapper.

Validation and rollout

I repaired data before enabling normal processing. The remediation script grouped credit rows by source event, identified extras, reversed only the duplicate entries, and checked resulting balances against invoices. The two incorrect subscription states were restored from the last valid audit transition. I reviewed every affected account rather than trusting a bulk row count.

The regression suite now sends the same signed fixture repeatedly and concurrently. Acceptance criteria are outcome based: one ledger effect, one final subscription transition, one completed event record, and successful acknowledgements for duplicates. A separate failure test forces the business batch to fail, verifies status='failed', and confirms a later retry can reclaim and complete it.

Monitoring that matches the invariant

I now monitor duplicate attempts, failed event age, and reconciliation mismatches. A duplicate counter above zero is not automatically an incident; it proves the guard is working. A failed event that remains unresolved, or two business effects sharing one source identity, is actionable.

The admin surface also links the event ledger to the billing audit trail. That operational view is more useful than searching unstructured logs during a payment incident. Readers evaluating the broader platform can inspect the feature catalog and compare the approach with other starters in the comparison hub.

Limits and failure modes that still remain

Idempotency closes one dangerous path, but it does not make the entire billing system infallible. I documented four boundaries so the next incident responder does not overestimate the protection.

First, event uniqueness does not guarantee event order. A subscription update can arrive before an earlier invoice event, and different Stripe event types do not form a transaction with my database. Handlers therefore derive state from authoritative object fields when possible and reject impossible backwards transitions. For high-risk transitions, the recovery path retrieves current state from Stripe rather than guessing from arrival order.

Second, an endpoint can commit successfully and lose its HTTP response. Stripe may retry because it never saw the acknowledgement. This is the exact case the completed ledger handles: the next delivery finds the same identifier and returns 200 without repeating the effect. That behavior is tested by forcing a response failure after the database batch commits.

Third, permanent mapping errors need a dead-letter workflow. Retrying a malformed internal command forever creates noise without progress. Failed rows retain a redacted diagnostic, attempt count, and timestamp; an operator can repair the mapper and replay the original provider event through a controlled command. Raw payment payloads are not sprayed into general logs, and retention follows the application’s privacy policy.

Fourth, the event ledger cannot prove that the provider and local account agree. Manual Stripe changes, old events from before the ledger, and administrator actions can still create divergence. A reconciliation job compares active subscriptions, invoice status, and credited entitlements, then emits differences for review. It does not blindly overwrite customer state.

These limits shaped the rollout. I did not advertise the patch as “exactly-once webhooks.” I described it as duplicate-safe processing with recoverable failures and reconciliation. That language is less dramatic, but it tells an on-call developer what guarantees are actually available.

A practical review checklist

Before I approve another provider webhook, I ask whether signature verification uses the untouched body, whether a stable provider identifier is stored uniquely, whether concurrent deliveries are safe, whether a failed claim can be retried, whether business effects also carry source identity, and whether an operator can reconcile without editing production rows by hand. If any answer is unclear, the handler is not finished.

Lessons I kept

Exactly-once is an outcome, not a delivery mode

Webhook transport may deliver at least once. “Exactly once” must be approximated at the application boundary with stable identity, atomic claims, unique constraints, and safe retries. An in-memory set, cache key, or preflight SELECT is not enough across concurrent Workers.

Put the invariant in the database

Application checks improve error messages; database constraints preserve truth under races. Money, credits, entitlements, and subscription transitions need source identities and uniqueness rules. Audit logs describe history, but they should not be the only mechanism preventing invalid history.

Reconciliation belongs beside webhook handling

Even a correct receiver can face mapping bugs or manual changes. A scheduled comparison of provider facts against internal effects catches a different class of failure. Idempotency prevents repetition; reconciliation detects divergence.

Honest postmortems narrow claims

I can verify the affected-account count, the two incorrect downgrades, and the four-hour remediation window from the supplied incident record. I cannot publish private payloads or infer cancellation behavior. Naming those limits is part of the fix: operational trust depends on precise language as much as precise code.

How TanStack Ship prevents the repeat

The current TanStack Ship billing pattern treats Stripe event identity as a schema-level contract. It combines signature verification, event claiming, idempotent business writes, an audit trail, replay tests, and reconciliation visibility. The related Stripe integration is designed around retries rather than assuming they are exceptional.

If you already have a webhook route, start with one review question: Can the same valid event.id arrive twice at the same time without creating a second business effect? If the answer depends on timing, add the ledger and unique source constraints before adding more event types.

See how TanStack Ship’s Stripe integration handles idempotency in the billing module, then review pricing when you are ready to ship the complete system.