5.3 Service Flow, Backtrace Analysis & Horizontal/Vertical Dependency Mapping
Key Takeaways
- Service Flow provides an aggregated, top-down view of service-to-service interactions triggered by a specific service, visualizing request percentages, call counts, and response time contributions.
- Service Backtrace provides a reverse, bottom-up dependency analysis starting from a downstream bottleneck or failure to identify every upstream service, application, and user request driving load to it.
- PurePath is a granular, end-to-end distributed trace of a single discrete transaction, whereas Service Flow aggregates thousands of PurePaths to reveal structural architecture and systemic bottlenecks.
- Horizontal dependency mapping connects peer services and network flows, while vertical dependency mapping correlates service-level latency with underlying process group bottlenecks, host resource exhaustion, or hypervisor throttling.
Service Flow, Backtrace Analysis & Horizontal/Vertical Dependency Mapping
Exam Focus: Mastering the distinction between Service Flow, Service Backtrace, PurePath, and Smartscape, interpreting Response Time Contribution and call percentages, isolating performance anti-patterns (such as N+1 query loops), and bridging horizontal service dependencies down to vertical infrastructure metrics are core competencies tested on the Dynatrace Certified Associate exam.
In microservice and distributed enterprise architectures, pinpointing the root cause of performance degradation requires traversing multiple analytical dimensions. When an end user complains that a web checkout operation takes ten seconds to complete, the delay could stem from an inefficient microservice call chain, a locking relational database query, an unresponsive third-party API, an internal JVM garbage collection pause, or operating system CPU starvation on a virtual machine host.
Dynatrace provides four specialized analytical lenses to dissect distributed application performance: Smartscape (topological infrastructure graph), PurePath (granular distributed trace of a single discrete transaction), Service Flow (aggregated forward call graph across thousands of transactions), and Service Backtrace (reverse dependency tree from bottleneck to upstream entry points).
The Four Observability Lenses Compared
| Analytical Tool | Directionality | Data Granularity | Primary Diagnostic Question Answered |
|---|---|---|---|
| Smartscape | Vertical & Horizontal | Architectural Infrastructure Graph | "What is the physical and logical topology of my entire environment?" |
| PurePath | Forward (Top-Down) | Single Discrete Transaction | "What exact line of code, SQL query, or thread executed during this specific transaction?" |
| Service Flow | Forward (Top-Down) | Aggregated Multi-Transaction Graph | "When requests hit Service X, where do they go, how often, and who contributes to total latency?" |
| Service Backtrace | Reverse (Bottom-Up) | Aggregated Multi-Transaction Tree | "Who is calling this slow or failing service, and which customer applications are impacted?" |
Deconstructing the Service Flow
Service Flow is an interactive, visual representation of the service-to-service communication paths triggered by a specific service (the anchor or root service). Rather than analyzing one transaction at a time, Service Flow aggregates data across all PurePaths processed by that service during the selected timeframe (e.g., the last 2 hours or last 24 hours).
[ Frontend Web Service ] ──► (100% of requests, 150 ms total duration)
│
├───► [ Auth Service ] (100% of requests, 15 ms duration, contributes 10%)
│
└───► [ Order Processing ] (100% of requests, 120 ms duration, contributes 80%)
│
├───► [ Payment Gateway ] (25% of requests, 200 ms duration, contributes 33%)
│
└───► [ PostgreSQL DB ] (850% of requests, 2 ms duration, contributes 60%)
▲
└── [ N+1 QUERY LOOP DETECTED! ]
Key Metrics Displayed on Service Flow Nodes
Each node and connecting branch in the Service Flow displays critical quantitative metrics:
- Request Percentage (% of requests): The proportion of requests from the upstream caller that trigger invocations of this downstream service.
- A value of
100%means every single incoming request triggers exactly one call. - A value of
25%means only one out of four requests calls this downstream service (e.g., conditional business logic or caching). - A value exceeding
100%(e.g.,450%or1200%) indicates that multiple calls are made per single incoming transaction (a looping execution or batch iteration).
- A value of
- Total Call Count: The absolute number of invocations over the selected time window (e.g., 2.4 million requests).
- Average Execution Time per Call: The average latency of an individual call to that downstream service (e.g., 4 milliseconds).
- Response Time Contribution (% and ms): The most critical metric in Service Flow. It measures how much latency this downstream service adds to the total response time of the root service.
The Golden Rule of Response Time Contribution
A common pitfall in performance diagnosis—and a heavily tested concept on the certification exam—is assuming that the slowest downstream service is automatically the main bottleneck. Response Time Contribution reveals the true impact:
Diagnostic Scenario: The Fast Service that Causes Massive Latency
Consider an entry service with an average response time of 1,000 ms calling two downstream services:
- Downstream Service A (External Credit Bureau API): Average duration = 300 ms. It is called on only 5% of requests. Its contribution is $300\text{ ms} \times 0.05 = \mathbf{15\text{ ms}}$ (only 1.5% of total response time).
- Downstream Service B (Local Inventory Cache DB): Average duration = 4 ms (extremely fast). However, due to an unoptimized loop in the application code, it is called 150 times per transaction (15,000% request rate). Its contribution is $4\text{ ms} \times 150 = \mathbf{600\text{ ms}}$ (60% of total response time!).
In this scenario, optimizing the 300 ms API saves negligible time, whereas eliminating the N+1 query loop on the 4 ms database reduces overall transaction latency by 600 ms.
Advanced Filtering in Service Flow
Service Flow allows deep filtering to isolate problematic transaction subsets. Administrators can click any node or branch to filter the entire flow by:
- Response Time Thresholds: Filter for transactions taking longer than 3 seconds.
- HTTP Status Codes: Filter for transactions resulting in HTTP 5xx server errors.
- Request Attributes: Filter by captured parameters, such as customer tier (
LoyaltyStatus = Platinum) or store location. - Direct Drill-Downs: From any filtered Service Flow view, engineers can pivot directly into PurePaths (to view individual stack traces) or Multidimensional Analysis (to chart metric distributions over time).
Service Backtrace: Reverse Dependency & Blast Radius Analysis
While Service Flow traces forward from an entry service down to its dependencies, Service Backtrace provides the exact inverse: it performs a bottom-up, reverse dependency analysis.
When to Use Service Backtrace
- Triage a Bottleneck: A shared PostgreSQL database service is experiencing thread locking and 100% CPU saturation. The DBA needs to know: "Which microservices are inundating this database with queries, and in what proportions?"
- Blast Radius Assessment: A downstream authentication microservice must be taken offline for emergency patching. The SRE needs to know: "Which upstream business services and customer-facing web/mobile applications will break if this service becomes unavailable?"
- Isolating Failing Upstream Deployments: A shared service suddenly experiences an error spike. Backtrace reveals whether the errors originate from all callers or exclusively from a newly deployed version of a single upstream microservice.
[ Customer-PostgreSQL Database ] (Bottleneck Service)
▲
├─── [ Order-Processing-Service ] (Sends 75% of queries)
│ ▲
│ └─── [ Web App: Customer Portal ] (Primary Blast Radius)
│
└─── [ Analytics-Reporting-Service ] (Sends 25% of queries)
▲
└─── [ Internal Admin Dashboard ] (Secondary Blast Radius)
In the Backtrace tree, Dynatrace visualizes every intermediate service hop leading backwards to the originating applications. Clicking any upstream branch reveals the exact call counts, failure rates, and percentage contribution of that specific caller.
Horizontal vs. Vertical Dependency Correlation in Diagnostics
Effective troubleshooting requires mastering the transition between horizontal and vertical dependencies. When a service experiences degraded performance, engineers follow a systematic diagnostic methodology:
=========================================================================================
DIAGNOSTIC WORKFLOW: HORIZONTAL & VERTICAL CORRELATION
=========================================================================================
STEP 1: DETECT ANOMALY
│──► Service response time baseline violated on "Checkout-Service".
▼
STEP 2: HORIZONTAL CHECK (Service Flow)
│──► Is the latency internal to Checkout-Service, or driven by downstream dependencies?
│ ├─► Downstream DB / API is slow: Troubleshoot that downstream component.
│ └─► Downstream services are healthy: Latency is INTERNAL to Checkout-Service!
▼
STEP 3: VERTICAL CHECK - PROCESS LEVEL (Smartscape / PGI)
│──► Drill down to the underlying Process Group Instance running Checkout-Service:
│ ├─► Are JVM Garbage Collection (GC) pauses consuming 40% of runtime?
│ ├─► Are application threads locked waiting on synchronization monitors?
│ └─► Is process memory exhausted (approaching OutOfMemoryError)?
▼
STEP 4: VERTICAL CHECK - HOST & INFRASTRUCTURE LEVEL (Smartscape / Host)
│──► Drill down to the underlying Virtual Machine / Operating System:
│ ├─► Is physical host CPU saturated (CPU Steal by hypervisor noisy neighbors)?
│ ├─► Is the operating system aggressively swapping memory to disk (page faults)?
│ └─► Are network interface drivers dropping packets (TCP retransmissions)?
=========================================================================================
Diagnostic Troubleshooting Matrix: Common Scenarios
| Operational Symptom | Primary Tool | Key Visual Indicator / Metric | Immediate Actionable Step |
|---|---|---|---|
| Slow Customer Transaction | Service Flow | Downstream service shows high Response Time Contribution (e.g. >50%). | Drill down into that specific downstream service or query. |
| Suspected N+1 Query Loop | Service Flow | Downstream service/DB shows request rate >100% (e.g., 500% to 2,000%). | Refactor calling code to use batch fetching instead of loop queries. |
| Shared DB Saturation | Service Backtrace | Upstream service branch shows massive surge in query volume. | Identify rogue upstream deployment or missing cache layer. |
| Service Slowdown (No downstream calls) | Vertical PGI View | High GC suspension time or high thread wait states in JVM/CLR metrics. | Tune runtime memory heap, analyze thread dumps, or fix lock contention. |
| Intermittent Latency Spikes | Vertical Host View | High CPU Steal percentage on virtual machine host. | Move VM away from oversaturated physical hypervisor node. |
| Isolated Customer Failure | PurePath | Single transaction shows red method execution or specific HTTP 500 error. | Inspect code-level exception message, stack trace, and parameters. |
A performance engineer investigates a sudden degradation in an e-commerce checkout service. The service's average response time increased from 150 milliseconds to 1.8 seconds. Analyzing the Service Flow, the engineer notices that the checkout service calls an internal "Inventory-Validation-Service". The Inventory-Validation-Service has an individual average response time of only 8 milliseconds per request, but its Response Time Contribution to the checkout service is 1.2 seconds (representing 66% of the total checkout duration). What architectural pattern and performance problem does this Service Flow data reveal?
A mission-critical relational database service named "Customer-PostgreSQL" experiences severe CPU saturation and thread locking, causing query times to spike to over 10 seconds. The database administrator needs to immediately identify every upstream application, user touchpoint, and microservice that is sending queries to this database to assess the blast radius and determine which team's deployment triggered the surge in traffic. Which Dynatrace analytical tool is specifically engineered to provide this reverse dependency visualization?
A site reliability engineer observes a latency alert on an authentication web service. In Service Flow, the engineer confirms that none of the downstream services or database calls show elevated response times; the latency originates entirely within the authentication service itself. To isolate the root cause, the engineer transitions from horizontal dependency analysis to vertical dependency analysis. Which diagnostic sequence correctly applies vertical dependency mapping to pinpoint the infrastructure bottleneck?