16.2 Custom Spans, Trace Analysis, and Sampling

Automatic vs. Custom Manual Instrumentation

• Automatic Instrumentation: Injects tracing middleware transparently into common web frameworks (Express, Spring Boot, Flask, Gin), HTTP client libraries, gRPC stubs, and database drivers. It automatically captures inbound routes, outbound HTTP requests, status codes, and SQL query strings without requiring code changes.
• Custom Manual Instrumentation: Necessary when developers need to measure internal algorithmic processing, business-critical transactions, or background worker threads not captured by framework wrappers.

// Creating Custom Spans with OpenTelemetry in Go
import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
)

func ProcessTransaction(ctx context.Context, orderID string, amount float64) error {
    tracer := otel.GetTracerProvider().Tracer("order-processor")
    
    // Start custom child span linked to incoming context
    ctx, span := tracer.Start(ctx, "ProcessTransaction")
    defer span.End() // ALWAYS defer End() to guarantee duration capture

    span.SetAttributes(
        attribute.String("order.id", orderID),
        attribute.Float64("order.amount", amount),
    )

    if err := executeRiskAnalysis(ctx, orderID); err != nil {
        // Record an exception event and mark span status as ERROR
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return err
    }

    span.SetStatus(codes.Ok, "Transaction processed successfully")
    return nil
}

Exam Trap: Forgetting to call span.End() (or omitting defer span.End()) causes the span to remain permanently open in memory. The span will never be exported to Cloud Trace, resulting in missing spans and an incomplete trace view.

Cloud Trace Console Analysis: Latency Distributions & Waterfall Timelines

The Cloud Trace console provides analytical views to diagnose production performance regressions:

Latency Distribution Curves (Scatter Plots)

• The Cloud Trace dashboard renders a scatter plot where the X-axis represents time of day and the Y-axis represents request latency in milliseconds or seconds. Each individual dot represents a sampled trace.
• Diagnosing Bimodal Distributions: A healthy service displays a tight, single horizontal band of dots (e.g., 20ms–40ms). If the scatter plot splits into two distinct horizontal bands (e.g., one band at 25ms and a second band at 450ms), the system is exhibiting a bimodal distribution. In microservices, this typically indicates cache hit vs. cache miss behavior, or execution paths branching between fast indexed database reads and slow unindexed full-table scans.
• Outliers: Isolated dots appearing high above the baseline represent request timeouts, garbage collection pauses, or database lock contention.

Waterfall Timelines

Clicking any point on the scatter plot opens its Waterfall Timeline:

• Visualizes hierarchical parent-child execution order chronologically.
• Serial Execution Anti-Pattern: Displays a stair-step cascade where Microservice A calls Service B (100ms), waits for completion, calls Service C (150ms), waits, and calls Service D (120ms). The total latency is 370ms. Refactoring the code to invoke independent services concurrently using asynchronous parallel calls (e.g., Go goroutines with sync.WaitGroup or JavaScript Promise.all) collapses the overall latency to the duration of the slowest single call (~150ms).
• Database Bottleneck Identification: Pinpoints slow individual SQL query spans, revealing lock contention or missing foreign key indexes.

Trace Comparison (Diff Analysis)

Cloud Trace allows developers to select two different time ranges or compare two deployment revisions (e.g., Cloud Run Canary Revision v2 vs. Baseline Revision v1) to visualize latency regressions before promoting new code to 100% of user traffic.

Trace Sampling Strategies: Head-Based vs. Tail-Based & Cost Management

Trace billing depends on the current ingestion path and pricing schedule; check the official observability pricing rather than assuming an old per-span price applies to every ingestion API. In an enterprise system serving 50,000 requests per second across 10 microservices, sampling 100% of requests would generate about 1.3 trillion spans in a 30-day month (assuming ten spans per request), resulting in massive operational costs and unnecessary storage overhead.