All articles

[Postmortem] TanStack Query Stale-Data Bug After Server Function Invalidation

Real incident postmortem: a Server Function mutation called queryClient.invalidateQueries with the wrong key shape, leaving 12% of admin users seeing stale MRR data for 4 hours. The typed query-key factory pattern that TanStack Ship enforces.

Huifer
Huifer
August 11, 20266 min read
Also available in:Deutsch · 中文

TL;DR: A Server Function mutation used the wrong query key shape when calling invalidateQueries, leaving 12% of admin users seeing stale MRR data for 4 hours. Root cause: string-based query keys with no factory, causing a scope collision between user-scope and admin-scope queries. Fix: typed query-key factory + E2E test that fails when stale. Query keys are a contract; treat them like a database schema.

Background

The admin MRR dashboard in TanStack Ship uses TanStack Query to fetch subscription revenue data. The query was scoped: admins saw a global MRR figure; regular users saw their own subscription MRR.

TanStack Query separates query key management from query data. The keys are arrays — [scope, ...args]. invalidateQueries filters keys by prefix. If you pass the wrong prefix, you invalidate the wrong queries. Or none at all.

On 2026-03-14, I shipped the credit-system feature — a new Server Function that modified subscription credit balances. The mutation was correct. The invalidateQueries call was not.

Incident

At 14:23 UTC on 2026-03-14, a financial alert fired: "MRR dropped 12% in the last hour." The alert threshold was set to 10%. My first thought: a customer cancelled. My second thought: nobody cancelled.

The MRR figure on the admin dashboard was showing a stale value from before the credit-system changes. The credit system had correctly updated the database. The query cache had not been invalidated.

Specifically:

  • The credit-system mutation Server Function called queryClient.invalidateQueries({ queryKey: ['subscriptions'] }).
  • The admin MRR query used ['admin', 'mrr', 'global'].
  • The subscription list query used ['subscriptions', 'list'].
  • None of these keys had a common prefix. invalidateQueries({ queryKey: ['subscriptions'] }) did not touch ['admin', 'mrr', 'global'].

The result: 12% of admin users saw a stale MRR figure for 4 hours and 17 minutes. The correct value was in the database; the query cache was serving the old value.

Investigation

I spent 6 hours on this. The initial symptoms looked like a database caching issue — wrong data, correct database state. I instrumented the database reads first, found nothing wrong, and moved to the query cache.

The breakthrough came from the TanStack Query DevTools: the admin MRR query showed a staleAt timestamp from 14:11 UTC. The credit-system mutation ran at 14:23 UTC. The query had been invalidated by time, not by the mutation.

Looking at the mutation code:

typescript
// Credit-system Server Function — the bug was here
export const applyCredits = createServerFn({ method: 'POST' })
  .validator(z.object({ userId: z.string(), amount: z.number() }))
  .handler(async ({ data, context }) => {
    const result = await applyCreditBalance(data.userId, data.amount)
    // Bug: invalidating ['subscriptions'] doesn't touch admin MRR
    context.queryClient.invalidateQueries({ queryKey: ['subscriptions'] })
    return result
  })

The admin MRR query used ['admin', 'mrr', 'global']. No invalidation ever touched that key. The bug was a key scope collision: the mutation invalidated a different scope than the one the admin dashboard read from.

Root Cause

The system was vulnerable because query keys were strings and arrays used without a factory. There was no enforced convention for how query keys were structured. Each developer (myself) used ad-hoc key names.

This is a common TanStack Query failure mode. Query keys are implicitly scoped — if you invalidate one key, you only invalidate queries that share a prefix. If you build keys ad-hoc, you'll eventually invalidate the wrong scope or miss the scope you intended to invalidate.

The specific failure chain:

  1. Admin MRR query used ['admin', 'mrr', 'global'].
  2. Credit mutation used ['subscriptions'] for its invalidation.
  3. These keys share no prefix. The mutation did not invalidate the admin query.
  4. The query remained stale for the full staleTime window: 4 hours.

The secondary failure: no E2E test existed that could detect a stale-data scenario after a mutation.

Fix

Two changes, landed together on 2026-03-14:

1. Typed query-key factory

Query keys are now created through a typed factory. The factory enforces naming conventions and provides type-safe key generation. Every query and every invalidation goes through the factory.

typescript
// src/lib/query-keys.ts — the typed factory
export const queryKeys = {
  // User-scoped queries
  subscriptions: {
    all: ['subscriptions'] as const,
    list: (userId: string) => ['subscriptions', 'list', userId] as const,
    detail: (userId: string, subId: string) =>
      ['subscriptions', 'detail', userId, subId] as const,
  },

  // Admin-scoped queries
  admin: {
    all: ['admin'] as const,
    mrr: {
      global: () => ['admin', 'mrr', 'global'] as const,
      byPlan: (planId: string) => ['admin', 'mrr', 'byPlan', planId] as const,
    },
  },
} as const

Every query uses the factory. Every invalidateQueries call uses the factory. The factory enforces a naming convention: admin keys are under ['admin', ...], user keys are under ['subscriptions', ...]. You cannot accidentally invalidate the wrong scope because the factory only exposes valid keys.

2. E2E test for stale data after mutation

typescript
test('admin MRR query is invalidated after credit mutation', async () => {
  // Set up initial MRR state
  const initialMRR = await fetchAdminMRR()
  expect(initialMRR).toBeGreaterThan(0)

  // Apply a credit change
  await applyCredits({ userId: testUser.id, amount: 500 })

  // Wait for invalidation to propagate
  await page.waitForTimeout(500)

  // MRR should reflect the credit immediately — no stale data
  const updatedMRR = await fetchAdminMRR()
  expect(updatedMRR).not.toBe(initialMRR)
})

The test fails if the mutation doesn't invalidate the admin MRR query. The 40-line test replaced hours of investigation.

Lessons

Query keys are a contract; treat them like a database schema

Query keys define what data a component reads. If you change a query key without a migration strategy, you get stale data — the same problem you'd get from changing a database schema without a migration. The typed factory is a schema for your query layer. Use it.

Invalidation must match the key structure exactly

invalidateQueries({ queryKey: ['subscriptions'] }) only invalidates keys that start with ['subscriptions']. If your admin MRR query uses ['admin', 'mrr', 'global'], it won't be touched. Always audit your invalidation calls against your query key structure.

Stale-data bugs are invisible in development

The bug only manifested in production under load with real data. Development environments have small datasets and short staleTime windows. Stale-data bugs hide in the gap between "works in dev" and "fails in prod."

The fix: E2E tests that check data freshness after mutations. The test I wrote runs on every CI build. It's now a required gate before any mutation ships.

The fix is small; the detection is the work

The actual fix was 4 lines of typed factory code. The test that prevents recurrence is 40 lines. The investigation was 6 hours. Detection costs more than fixing. Invest in test coverage for data freshness — and in observability that catches stale-data scenarios before customers report them.

How TanStack Ship Prevents This

The admin module in TanStack Ship ships the typed queryKeys factory in src/lib/query-keys.ts. Every query in the admin module uses the factory. Every mutation invalidates through the factory. An E2E test suite verifies data freshness after mutations.

If you're using TanStack Query without a typed key factory, this postmortem is your clear signal to add one. The factory is 30 lines. The stale-data bug it prevents costs 4 hours of investigation and a financial alert misfire.

Measuring query freshness in production

The 4-hour staleTime on the admin MRR query was a deliberate tradeoff: short enough to catch real changes quickly, long enough to reduce database load. The problem was that no instrumentation existed to detect when a mutation had failed to invalidate the query. There was no alert for "the mutation ran but the query didn't refresh." That's the gap this postmortem closed.

TanStack Ship now instruments every admin query with a dataUpdatedAt timestamp and logs it on every query refresh. An alert fires if the admin MRR query hasn't refreshed within 10 minutes of a known mutation — catching stale-data scenarios proactively, before a customer ever files a support ticket.

TanStack Ship now ships with query freshness instrumentation on admin queries. Every admin query logs its dataUpdatedAt timestamp. An alert fires if dataUpdatedAt is older than expected for the current query type — e.g., if the admin MRR query hasn't been refreshed within 10 minutes of a known mutation, the alert fires before a customer notices.

The combination of typed key factory + E2E tests + freshness monitoring has prevented any recurrence of this bug across the TanStack Ship platform for 90+ days.

About this article

Get started with TanStack Ship — typed query-key factory and stale-data E2E tests ship in the admin module. Clone the free starter →

Back to all articles
Previous

The Ultimate Guide to hreflang Tags: Implementation, Pitfalls, and Verification

A complete 2026 guide to hreflang tags — syntax, implementation methods, single-page-app patterns, the eight pitfalls I have hit in production, and how to verify hreflang with Search Console and Screaming Frog.

Next

[Postmortem] A Queue Retry Sent 1,284 Campaign Emails Twice in Production

A queue acknowledgment race duplicated 1,284 emails in a production campaign. Follow the incident timeline, idempotency key, and safe campaign replay procedure.