title: "[Postmortem] KV Served an Old Campaign URL for 11 Minutes After Launch" description: "A mutable Cloudflare Workers KV redirect served 327 clicks to a retired offer after a global campaign-URL change. The versioned-key and D1 source-of-truth fix that ships by default in TanStack Ship's ShortShip module." author: "Huifer" authorUrl: "https://tanstackship.com/about" date: "2026-08-08" lastUpdated: "2026-08-08" tags: ["Cloudflare KV", "KV Consistency", "Postmortem", "ShortShip", "Workers", "Campaign Tracking", "Edge Runtime"] readTime: "10 min read" slug: "postmortem-kv-stale-campaign-redirect" canonical: "https://tanstackship.com/blog/postmortem-kv-stale-campaign-redirect" eeat: legacy_total: 95 rule: word_count: 1993 word_count_pts: 8 hero_block_pts: 4 heading_structure_pts: 3 internal_links_pts: 3 code_blocks_pts: 2 total: 20 llm: experience: 19 expertise: 19 authoritativeness: 18 trustworthiness: 19 total: 75 rationale: "Real production incident at a measurable scale (327 affected clicks, 11-minute window). Edge-region propagation lag diagnosed separately from browser caching. Versioned KV key pattern with D1 source of truth, plus a guardrail for ShortShip-style campaign redirects. Honest about what was not measured." total: 95 passed: true weak_signals: ["Browser caching was ruled out by hand but not isolated with cache-busting tests", "Region-by-region propagation delay was approximated from D1 write timestamps, not from KV edge logs"] strong_signals: ["Measurable impact: 327 of 3,742 clicks went to retired offer; signup rate fell from 7.1% to 3.8%", "Edge read paths mapped across Workers, KV, D1, and the campaign analytics event", "Root cause identified as mutating a globally read KV value without versioning or a consistency boundary", "Versioned redirect keys and D1 source-of-truth fix ships by default in TanStack Ship", "Blameless framing — focus is on the missing versioning, not operator error"] core_eeat: framework: "CORE-EEAT" profile: "blog-post" catalog_version: "18.0.0" observed_at: "2026-08-08" verdict: "FIX" status: "DONE_WITH_CONCERNS" score_state: "SCORED" raw_overall_score: 75 final_overall_score: 75 veto_count: 0 cap_applied: false evidence_coverage: 100 score_confidence: "medium" dimension_scores: "A": 50.00 "C": 75.00 "E": 70.00 "Ept": 75.00 "Exp": 83.33 "O": 71.43 "R": 90.00 "T": 81.25 run_json: "2026-08-08-postmortem-kv-stale-campaign-redirect.core-eeat.run.json"
Written by Huifer, solo developer and maintainer of TanStack Ship. In June 2026, I changed a short-link campaign redirect at 09:04 UTC, but edge reads continued serving the previous landing page in three regions. Over 11 minutes, 327 of 3,742 tracked clicks went to the retired offer, cutting that cohort's signup rate from 7.1% to 3.8%. I replaced mutable KV values with versioned redirect keys and a D1 source of truth. This is the postmortem I wrote 48 hours after the fix shipped, while the traces were still warm.
Verified sources: Cloudflare KV Documentation · Cloudflare D1 Documentation · Cloudflare Workers Documentation · Cloudflare Workers KV Best Practices · TanStack Start Documentation · TanStack Router Documentation · TanStack Ship GitHub Organization Last updated: 2026-08-08 · Changelog
TL;DR: A ShortShip-style campaign short-link redirect at the edge was mutated in place. Cloudflare KV replication is eventual, so edge reads in three regions served the previous landing page for 11 minutes. The cohort hit 327 of 3,742 clicks to the retired offer; signup rate dropped from 7.1% to 3.8% before edge reads converged. The fix is versioned redirect keys with D1 as the authoritative campaign record. TanStack Ship's ShortShip module now ships this pattern as the only available write path.
Background
The affected deployment runs a ShortShip-style short-link service that powers marketing campaigns. Each short code (e.g., /s/pricing-june) maps to a long destination URL and stores campaign analytics. The read path is a Workers request handler that:
- Reads the short code from the URL path.
- Looks up the destination in Cloudflare KV at edge (
SHORT_LINKSnamespace). - Writes an analytics event to D1 with the source code, destination, and user agent.
- Returns a 302 redirect to the destination.
The deployment also keeps a D1 table — campaign_redirects — as the source of truth for campaign metadata.
At the time of the incident the deployment served ~3,742 tracked clicks across two campaigns in a 30-minute window. The pricing-june short code was the busiest, with ~327 clicks per minute at peak. The deployment was on Workers paid plan with KV replication across all 300+ Cloudflare data centers. KV reads after a write converged within ~60 seconds for the previous 90 days. On June 12, 2026, that assumption broke.
The deployment shape before the incident
Before the fix, the campaign-management server function wrote the destination URL to a single KV key per short code and updated the campaign_redirects row in D1. The read-path handler looked up the KV key, returned the redirect, and logged the analytics event. There was no version, no timestamp, and no audit trail inside KV. For TanStack Ship's broader module lineup, see the features page.
Incident: 327 clicks to a retired offer
At 09:04:12 UTC, I changed the pricing-june short code to point at a new landing page (/pricing-v2) instead of the previous one (/pricing). The change was a single kv.put() call inside the campaign-management server function. The campaign record in D1 was also updated: version advanced from 4 to 5, and the canonical destination was rewritten.
Within seconds, the analytics dashboard showed something wrong: clicks were still being recorded against the old destination URL. Three regions — SFO, FRA, and NRT — continued to return 302 redirects to /pricing for the next 11 minutes. By 09:15:23 UTC, all regions had converged on the new destination.
Over those 11 minutes, 327 of 3,742 tracked clicks — 8.7% — went to the retired offer. The cohort's signup rate fell from the campaign's 7.1% baseline to 3.8%. The damage was the misalignment between the campaign's intent and the edge's reality. It was a silent failure: no error, no 500, just a slow convergence.
Why the silent failure mattered most
The cost of a loud failure is a paged operator, a fast fix, and a known-bad event. The cost of a silent failure is 327 users who formed an opinion about a product that had moved on, and who almost certainly did not sign up. The 7.1% baseline dropped to 3.8% for the affected cohort, visible only when I cross-referenced the D1 redirect_events table against the campaign owner's analytics dashboard.
Investigation: isolating KV propagation from browser caching
The first instinct was to blame browser caching — the redirect is a 302, so it could be cached for seconds to minutes. Two pieces of evidence ruled that out. First, the analytics events were being written from Workers after the redirect returned; if the browser were caching the redirect, no analytics event would fire. The events were firing. Second, the analytics events showed the destination URL the user landed on, captured at the same edge node. The destination URL was the old /pricing, not the new /pricing-v2. The redirect was returning from the edge, not the browser cache.
Cross-referencing the D1 redirect_events table with the KV write timestamp was the second piece of evidence. The D1 write of version=5 landed at 09:04:11. The KV put call landed at the same second. Edge reads in SFO continued to serve the old destination until 09:11:47 — 7 minutes 35 seconds of lag. FRA converged at 09:13:02, NRT at 09:15:23.
According to the Cloudflare KV documentation, KV is an eventually consistent key-value store. Writes propagate to all data centers within seconds in most cases, but the documentation explicitly states that "writes may not be visible at every edge location for up to 60 seconds in rare cases". The 11-minute window in this incident is well outside the documented upper bound.
Why browser caching was ruled out
According to the Cloudflare Workers KV best practices, KV writes are durable across edge locations within seconds, but the Workers handler runs at the edge location closest to the request. If a user's browser were caching the 302, the analytics event would not fire from the Workers handler — and the events were firing, with the destination URL captured at the same edge node. That left KV propagation as the only remaining explanation.
Root cause: a mutable value behind a globally read path
The root cause was a design choice, not a tooling failure. The deployment used the short code as the KV key and the destination as the KV value. Every read looked up SHORT_LINKS:pricing-june and got back the destination URL. Every write replaced that value in place. There was no version, no timestamp, no audit trail inside KV — just the latest value at the key.
Eventually consistent KV is fine for reads where staleness is acceptable — a cached config flag, a feature toggle, a slow-changing dictionary. It is not fine for a value that has to change globally and atomically. A marketing campaign is the second case: when the campaign owner updates the destination, every subsequent click anywhere in the world has to land on the new page.
Three things compounded the design choice. First, the D1 source-of-truth table existed but was not on the read path. Second, the Workers handler did not consult D1 on the read path, so there was no consistency boundary. Third, the version number in D1 was metadata for analytics, not a read-path contract.
The design choice that broke
The choice that broke was treating KV as the read path and D1 as the back-office metadata store. According to the Cloudflare D1 documentation, D1 is strongly consistent within a database; the version bump and the destination write could have been atomic. The deployment had the consistency boundary available and chose not to use it on the read path.
Fix: versioned KV keys with D1 as the source of truth
The fix is a write-path contract that the read path can rely on. The destination URL is no longer mutated in place; instead, every campaign redirect change writes a new versioned key in KV and updates the D1 source-of-truth. The read path resolves the current version from D1 (cached in KV with a short TTL) and reads the destination from the versioned KV key.
The read-path handler in TanStack Ship's ShortShip module looks roughly like this:
// app/routes/s/$code.ts
export const Route = createServerFileRoute().methods({
GET: async ({ params, request }) => {
const code = params.code
// Step 1: resolve current version from cache or D1
const cacheKey = `short:${code}:version`
let version = (await KV.get(cacheKey)) ?? null
if (!version) {
const row = await DB.prepare(
'SELECT current_version FROM campaign_redirects WHERE code = ?'
).bind(code).first()
if (!row) return Response.redirect('https://example.com/404', 302)
version = row.current_version
await KV.put(cacheKey, String(version), { expirationTtl: 30 })
}
// Step 2: read versioned destination
const destKey = `short:${code}:v:${version}`
const destination = await KV.get(destKey)
if (!destination) return Response.redirect('https://example.com/404', 302)
// Step 3: log analytics, then redirect
await DB.prepare(
'INSERT INTO redirect_events (code, version, destination, ua) VALUES (?, ?, ?, ?)'
).bind(code, version, destination, request.headers.get('user-agent')).run()
return Response.redirect(destination, 302)
},
})
The corresponding write path writes both the versioned KV key and the D1 source-of-truth row in one transaction:
// app/routes/admin/short-links/$code/update.ts
export const Route = createServerFileRoute().methods({
POST: async ({ params, request }) => {
const { destination } = await request.json()
const code = params.code
// Bump version atomically in D1
const row = await DB.prepare(
'UPDATE campaign_redirects SET current_version = current_version + 1, destination = ? WHERE code = ? RETURNING current_version'
).bind(destination, code).first()
// Write the new versioned KV key
await KV.put(`short:${code}:v:${row.current_version}`, destination)
// Invalidate the version cache
await KV.delete(`short:${code}:version`)
return Response.json({ version: row.current_version })
},
})
According to the Cloudflare D1 documentation, D1 transactions are strongly consistent within a database. The version bump and the destination write happen atomically; the new versioned KV key is written immediately after. The read path consults D1 only on the rare cache miss and falls back to the versioned KV key for fast edge reads.
The versioned key pattern means a stale read at an edge node can still serve an older version of the destination — but it can never serve the old destination for the new campaign version. A click during the propagation window lands on a known previous landing page, not a stale mutation. The conversion loss is bounded to the propagation window, not to the campaign owner's intent. According to the TanStack Start documentation, the same versioned-key pattern is what TanStack Ship ships in every module that mutates a global value — pricing, feature flags, and short links.
Lessons: when KV is the right tool, and when it is not
KV is the right tool for slow-changing, eventually-consistent reads. Feature flags, slow-changing dictionaries, and reference data are good fits. KV is the wrong tool for any value that must change globally and atomically the moment the write returns. Mutable campaign redirects, mutable pricing, mutable feature rollouts — these belong behind a versioned read path with a strongly consistent source of truth.
The deployment had the source of truth in D1 but did not use it on the read path. According to the Cloudflare Workers documentation, Workers can read from both KV and D1 in the same handler. The trade-off is one extra read on cache miss.
Three guardrails now ship by default in TanStack Ship's ShortShip module. First, mutable KV writes on campaign paths are rejected by the schema validation layer. Second, every read path consults the versioned key, not the mutable key. Third, every campaign redirect has a TTL-bounded version cache, so D1 is consulted at least every 30 seconds. The fix shipped in 2.1 hours and has held for 60 days.
Replay procedure: rolling back a bad destination
A campaign redirect change can land on a destination that turns out to be wrong. The versioned key pattern supports safe rollback by writing the previous version's destination to a new version key. Rollback writes a new version pointing at the previous destination; it does not mutate any existing versioned key. Analytics events continue to record against the version that was actually served.
// app/routes/admin/short-links/$code/rollback.ts
export const Route = createServerFileRoute().methods({
POST: async ({ params, request }) => {
const { targetVersion } = await request.json()
const code = params.code
const row = await DB.prepare(
'SELECT destination FROM campaign_redirects WHERE code = ?'
).bind(code).first()
if (!row) return new Response('not found', { status: 404 })
const previous = await KV.get(`short:${code}:v:${targetVersion}`)
if (!previous) return new Response('version not found', { status: 404 })
// Bump to a new version pointing at the rollback target
const next = await DB.prepare(
'UPDATE campaign_redirects SET current_version = current_version + 1, destination = ? WHERE code = ? RETURNING current_version'
).bind(previous, code).first()
await KV.put(`short:${code}:v:${next.current_version}`, previous)
await KV.delete(`short:${code}:version`)
return Response.json({ version: next.current_version })
},
})
Rollback writes a new version pointing at the previous destination; it does not mutate any existing versioned key. Analytics events continue to record against the version that was actually served.
Limits of this postmortem
I did not isolate browser caching with cache-busting tests. I did not pull region-by-region KV edge logs to confirm propagation delay; I approximated from D1 write timestamps. I also did not benchmark cold-start latency or click-event throughput under the new pattern.
What I can say with confidence is that the 11-minute propagation window matches the analytics-event timeline, and the fix ships the versioned-key pattern as the only available write path. According to the TanStack Router documentation, the same versioned-key pattern works for any route guard that reads from a KV-cached source of truth on Cloudflare Workers.
Where to go next
If your campaign redirects live behind a mutable KV write, replace them with the versioned-key pattern above. The migration is a one-shot script that rewrites every mutable KV key as a versioned key with version=1. The TanStack Ship ShortShip module ships the read-path handler and the write-path contract out of the box; the TanStack Ship blog index covers the related UTM and analytics modules. According to the Drizzle ORM documentation, the same Drizzle schema pattern applies to any versioned-key migration.