Day 13/60: Distributed Tracing with Zipkin -- Production Engineering

60 Day Production Engineering Challenge

When a request crosses 15 service boundaries and times out, logs from individual services tell you almost nothing useful. Zipkin gives you a single trace view spanning every hop.

Zipkin Architecture

Four components: the collector receives spans (HTTP or Kafka transport), storage persists them, the query service exposes a retrieval API, and the web UI renders waterfall diagrams plus a dependency graph.

Deploy with Elasticsearch storage for production-grade tag searches:

version: '3.8'
services:
  zipkin:
    image: openzipkin/zipkin:3.1
    environment:
      STORAGE_TYPE: elasticsearch
      ES_HOSTS: http://elasticsearch:9200
      COLLECTOR_SAMPLE_RATE: '0.1'
    ports:
      - '9411:9411'
    depends_on:
      elasticsearch:
        condition: service_healthy

  elasticsearch:
    image: elasticsearch:8.12.0
    environment:
      discovery.type: single-node
      xpack.security.enabled: 'false'
      ES_JAVA_OPTS: -Xms1g -Xmx1g
    healthcheck:
      test: ['CMD', 'curl', '-f', 'http://localhost:9200/_cluster/health']
      interval: 10s
      retries: 10
    volumes:
      - es_data:/usr/share/elasticsearch/data

volumes:
  es_data:

B3 Propagation and Brave Instrumentation

B3 headers (X-B3-TraceId, X-B3-SpanId, X-B3-ParentSpanId, X-B3-Sampled) carry trace context between services. With Spring Boot 3 and Micrometer, add these dependencies:

// build.gradle.kts
dependencies {
    implementation("io.micrometer:micrometer-tracing-bridge-brave")
    implementation("io.zipkin.reporter2:zipkin-reporter-brave")
}

Configure in application.yml:

management:
  tracing:
    sampling:
      probability: 0.1
  zipkin:
    tracing:
      endpoint: http://zipkin:9411/api/v2/spans
spring:
  application:
    name: order-service

Brave automatically instruments RestTemplate and WebClient calls with B3 headers. Every inbound request gets a span, every outbound call propagates the trace.

Storage Backend Comparison

Zipkin vs Jaeger

Jaeger has adaptive sampling (automatic per-endpoint rate adjustment) and a Kubernetes Operator. Zipkin has a broader client library ecosystem via Brave and simpler operations. For Java/Spring Boot stacks, Zipkin. For Kubernetes-native with adaptive sampling needs, Jaeger. For new projects, use OpenTelemetry SDK with a Zipkin exporter and defer the backend choice.

Day 13/60 of the 60 Day Production Engineering Challenge