SaaS Observability Stack: We Debug Production Issues in 10 Minutes Now

TL;DR: Observability is the foundation of operational confidence in a SaaS application. Without it, you are debugging in the dark. This guide covers the three pillars of observability — logs, metrics, and traces — implemented for SaaS applications on Cloudflare Workers and TanStack Start, with practical patterns for production monitoring, alerting, and root cause analysis.

The Three Pillars of Observability

Logging: Structured Logging on the Edge

Log Format

Structured logging captures every request in a consistent JSON format that includes level, message, request ID, duration, and metadata. This structured approach enables filtering, alerting, and debugging across all your services:

// src/lib/logger.ts
type LogLevel = "debug" | "info" | "warn" | "error"

interface LogEntry {
  timestamp: string
  level: LogLevel
  message: string
  requestId: string
  userId?: string
  duration?: number
  error?: string
  metadata?: Record<string, unknown>
}

function createLogger(request: Request) {
  const requestId = request.headers.get("cf-ray") ?? crypto.randomUUID()

  return {
    info: (msg: string, meta?: Record<string, unknown>) =>
      log({ level: "info", message: msg, requestId, metadata: meta }),
    warn: (msg: string, meta?: Record<string, unknown>) =>
      log({ level: "warn", message: msg, requestId, metadata: meta }),
    error: (msg: string, error?: Error, meta?: Record<string, unknown>) =>
      log({
        level: "error",
        message: msg,
        requestId,
        error: error?.message,
        stack: error?.stack,
        metadata: meta,
      }),
    getRequestId: () => requestId,
  }
}

The logger attaches a unique requestId (from Cloudflare's cf-ray header) to every log entry, making it possible to correlate all logs from a single request. The createLogger factory embeds this ID so developers never need to pass it manually.

Sending Logs to Analytics Engine

Cloudflare Workers Analytics Engine provides queryable, long-term storage for your log data. Each log entry is written as a data point with blobs (strings), doubles (numeric values), and indexes for efficient filtering:

// src/lib/logger.ts — writing to Workers Analytics Engine
function log(entry: LogEntry) {
  // Write to Analytics Engine for queryable storage
  context.env.ANALYTICS.writeDataPoint({
    blobs: [
      entry.level,
      entry.message,
      entry.requestId,
      entry.userId ?? "",
      entry.error ?? "",
      JSON.stringify(entry.metadata ?? {}),
    ],
    doubles: [entry.duration ?? 0],
    indexes: [entry.requestId],
  })

  // Also output to Workers console for wrangler tail
  console.log(JSON.stringify(entry))
}

Writing to both Analytics Engine and console.log gives you the best of both worlds: persistent, queryable storage for dashboards and real-time streaming via wrangler tail for interactive debugging.

Metrics: What to Measure

SaaS Metrics Taxonomy

Collecting Performance Metrics

Request-level metrics — latency, status codes, and cache hit rates — are the foundation of any observability practice. This middleware captures timing data for every request and records it to Analytics Engine for later analysis:

// server/middleware/metrics.ts
export const metricsMiddleware = createServerFn({ method: "GET" }).handler(
  async ({ request, context }) => {
    const startTime = Date.now()

    // After the request completes, record metrics
    context.waitUntil(
      context.env.ANALYTICS.writeDataPoint({
        blobs: [
          request.url,
          request.method,
          String(context.response?.status ?? 500),
        ],
        doubles: [
          Date.now() - startTime, // duration
          request.cf?.colo ?? 0, // edge location
        ],
        indexes: [request.url.split("?")[0]], // path
      })
    )
  }
)

Using context.waitUntil ensures the metric is recorded even after the response has been sent to the client, so you never add latency to user-facing requests while still collecting comprehensive data.

Querying Metrics

Once metrics are collected, querying them requires SQL access to the Analytics Engine. This pattern extracts percentile breakdowns by API path, giving you a clear picture of which endpoints need optimization:

// Query Analytics Engine for dashboard data
export const getApiLatency = createServerFn({ method: "GET" }).handler(
  async ({}, { context }) => {
    const result = await context.env.ANALYTICS.query({
      sql: `
        SELECT
          blob3 as path,
          quantile(0.50)(double1) as p50,
          quantile(0.95)(double1) as p95,
          quantile(0.99)(double1) as p99,
          count() as requests
        FROM analytics_engine_table
        WHERE timestamp > now() - INTERVAL '1' DAY
        GROUP BY blob3
        ORDER BY p99 DESC
        LIMIT 20
      `,
    })

    return result.rows
  }
)

The quantile functions calculate percentile latencies directly in the database, avoiding the overhead of pulling raw data into your application for processing. Filtering to the last 24 hours keeps queries fast and focused on current performance.

Tracing: Distributed Request Tracing

Setting Up a Tail Worker

A Tail Worker receives events from your primary Worker after each request completes, making it ideal for sampling trace data without impacting request performance. Deploy it as a separate Worker that processes event streams:

// tail-worker.ts — deployed as a separate Worker
export default {
  async tail(events: TraceEvent[]) {
    for (const event of events) {
      if (event.duration && event.event.request) {
        await ANALYTICS.writeDataPoint({
          blobs: [
            event.event.request.method,
            event.event.request.url,
            String(event.exceptions?.[0]?.message ?? ""),
          ],
          doubles: [
            event.duration ?? 0,
            event.event.request.cf?.botManagement?.score ?? 0,
          ],
          indexes: [event.event.request.url.split("?")[0]],
        })

        // Alert on exceptions
        if (event.exceptions?.length > 0 || event.crons?.length > 0) {
          await sendIncidentAlert(event)
        }
      }
    }
  },
}

The Tail Worker captures duration, exceptions, and bot scores for every request. When exceptions are detected, it can trigger incident alerts immediately — turning your observability pipeline into an automated incident response system.

Injecting Trace Context

For requests that interact with external services (Stripe, Resend, OpenAI), propagating a trace ID lets you correlate the server function execution with downstream API calls:

// Pass trace context from server function to downstream services
export const handleWebhook = createServerFn({ method: "POST" }).handler(
  async ({ request, context }) => {
    const traceId = crypto.randomUUID()

    // Pass trace ID to downstream calls
    await fetch("https://api.stripe.com/v1/events", {
      headers: {
        "X-Trace-Id": traceId,
      },
    })

    // Log with trace context
    context.logger.info("Webhook received", { traceId, eventType: event.type })
  }
)

The X-Trace-Id header is passed to downstream services, and the same trace ID is included in log entries. This creates an end-to-end trace that connects the incoming request, your server function logic, and any external API calls.

Alerting: When to Wake Someone Up

Alert Severity Levels

Alert Configuration

Alerts should be driven by the metrics you are already collecting, not by separate monitoring infrastructure. This server function queries Analytics Engine for error rates and latency percentiles, then evaluates them against thresholds to determine if action is needed:

// server/monitoring/alerts.ts
import { createServerFn } from "@tanstack/react-start"

export const checkAlerts = createServerFn({ method: "GET" }).handler(
  async ({}, { context }) => {
    const alerts = []

    // 1. Check error rate
    const errorRate = await context.env.ANALYTICS.query({
      sql: `
        SELECT count() as total,
               countIf(blob0 = 'error') as errors
        FROM analytics_engine_table
        WHERE timestamp > now() - INTERVAL '5' MINUTE
      `,
    })

    const rate = Number(errorRate.rows[0]?.errors ?? 0) /
                 Math.max(Number(errorRate.rows[0]?.total ?? 1), 1)

    if (rate > 0.01) {
      alerts.push({
        severity: "high",
        message: `Error rate ${(rate * 100).toFixed(1)}% > 1%`,
      })
    }

    // 2. Check latency
    const latency = await context.env.ANALYTICS.query({
      sql: `
        SELECT quantile(0.99)(double1) as p99
        FROM analytics_engine_table
        WHERE timestamp > now() - INTERVAL '5' MINUTE
      `,
    })

    const p99 = Number(latency.rows[0]?.p99 ?? 0)
    if (p99 > 5000) {
      alerts.push({
        severity: "critical",
        message: `p99 latency ${p99}ms > 5000ms`,
      })
    }

    return alerts
  }
)

The alert checker runs on a cron schedule and evaluates both error rate and latency thresholds. By querying the last 5 minutes of data, it catches spikes quickly while avoiding false positives from brief transient errors.

Dashboard: Visualizing Your Observability Data

The ideal SaaS observability dashboard has three views:

Operational View (Real-Time)

┌──────────────┬──────────────┬──────────────┬──────────────┐
│ Request Rate │ Error Rate   │ p50 Latency  │ p99 Latency  │
│   1,234/min  │   0.02%      │    45ms      │    210ms     │
├──────────────┼──────────────┼──────────────┼──────────────┤
│ Active Users │ Signups (24h)│ Churn (30d)  │ MRR          │
│   847        │   23         │    3.2%      │  $12,450     │
└──────────────┴──────────────┴──────────────┴──────────────┘

Business View (Weekly)

Cohort Retention Table
Signup Funnel (Visit → Signup → Subscribe → Active)
Feature Adoption Heatmap
Top User Actions

Debug View (On-Demand)

Request Trace Explorer
Error Log Stream
Worker CPU Profile
Database Query Performance

Automating Incident Response

When alerts fire, the incident response handler logs the event, notifies the team through Slack and email (for high-severity issues), and attempts auto-remediation where possible. This server function centralizes the response logic:

// server/monitoring/incident.ts
export const handleIncident = createServerFn({ method: "POST" }).handler(
  async ({ data }: { data: { severity: string; message: string } }) => {
    // 1. Log the incident
    await logIncident(data)

    // 2. Notify (only for high+ criticality)
    if (["high", "critical"].includes(data.severity)) {
      await sendSlackNotification(
        `[${data.severity.toUpperCase()}] ${data.message}`
      )
      await sendEmail({
        to: "ops@tanstackship.com",
        subject: `[${data.severity}] ${data.message}`,
      })
    }

    // 3. Auto-remediate where possible
    if (data.message.includes("rate limit")) {
      // Scale up rate limit window
      console.log("Auto-remediation: rate limit exceeded")
    }
  }
)

The incident handler uses a severity-based notification strategy: low and medium alerts are logged silently, while high and critical alerts trigger immediate notifications. Auto-remediation actions — like adjusting rate limits — are built into the same flow so recovery happens without human intervention.

Observability Budget

Allocate your observability resources wisely:

Production Observability Checklist

• All requests produce structured JSON logs
• Logs include request ID, user ID, and duration
• Error logs include stack traces
• All critical business events are tracked as metrics
• Worker performance metrics collected (CPU, memory, subrequests)
• Alerts configured for p99 latency > 5s and error rate > 1%
• Tail worker captures trace data for sampled requests
• Dashboard covers operational, business, and debug views
• Logs have automated retention and archival policy
• Access to logs is audited and role-restricted

Conclusion

Observability is not optional for a production SaaS application. The three pillars — logs, metrics, and traces — each serve a different purpose, and a mature observability practice needs all three.

For TanStack Start on Cloudflare Workers, the native observability tooling (Workers Analytics Engine, Tail Workers, Trace Workers) provides a solid foundation without requiring third-party services. You can collect, query, and alert on operational data entirely within the Cloudflare ecosystem.

The key is to start simple: structured logs first, then metrics for the most critical business and performance indicators, then traces for debugging complex issues. Add alerting only when you have enough data to set meaningful thresholds. For a broader security perspective, see our SaaS Security Best Practices, and learn how to tie performance metrics to user experience in the Core Web Vitals Optimization Guide. The Real User Monitoring guide covers collecting field data from actual users, while the Cloudflare D1 Production Guide explains how to monitor your database performance.