11.1 Systematic Troubleshooting Methodology & Observability Concepts
Key Takeaways
Start a troubleshooting investigation by confirming the observed failure, affected users and recent changes. Test the chosen explanation before making broad changes.
An SLI measures a service outcome, an SLO sets its target, and an SLA records the provider’s commitment under stated terms.
For a request-based SLO, calculate the error budget using the same eligible requests and measurement window as the SLI.
SNMP traps are unacknowledged notifications. The USM authPriv level provides both cryptographic authentication and privacy.
Use thresholds and routes to turn signals into actionable alerts. Log aggregation and webhooks require deliberate transport, authentication and receiver behavior.
11.1 Systematic Troubleshooting Methodology & Observability Concepts
Monitoring turns measurements from Linux hosts and services into evidence for a decision. Start by asking what users need to work, which signal can measure it, and who should respond when that signal changes.
A repeatable troubleshooting workflow
- Describe the observed problem. Record the failed operation, time, affected users and last known working state. Compare that information with logs and recent changes. If a website stops responding after maintenance, inspect what changed before restoring a snapshot.
- Form a plausible explanation. Check basics such as power, interface state, service state and capacity. Compare a working peer where it shares the affected system's role and configuration.
- Test the explanation. Choose a check that distinguishes two possible causes. For example, compare a service's local response with an approved request through its normal entry point. A local success keeps the external path and configuration in scope. If the result contradicts the hypothesis, update it or escalate.
- Plan and apply a fix. Identify affected components and decide how to roll back before changing them. Use established operational controls for production changes.
- Verify the actual operation. Check a representative client workflow after recovery. Verify dependencies and test the alert that should report a recurrence.
- Record the outcome. Capture evidence, changes, verification and remaining follow-up. Preserve context that will help with the next incident.
This workflow supports the scenario work in the troubleshooting domain. The published 5.1 objectives focus on monitoring concepts, configuration and data sources.
Observability Concepts: Metrics, Logs & Traces
- Metrics aggregate numerical measurements over time. A request count and error count can show when a service crossed a threshold.
- Logs record events with context. An application log might identify the dependency and error code that appeared during a spike.
- Traces connect spans for a request across instrumented components. A trace can show which call accounted for most of the duration.
For example, a latency metric shows that a group of requests slowed at 11:12. Application logs identify which requests experienced upstream timeouts. A sample trace shows slow database calls for one of those requests. Compare the same time interval and request identifiers where available. The initial metric locates the symptom; the other signals help you test a cause.
Service Reliability: SLA, SLO, SLI & Error Budgets
| Term | Meaning | Example |
|---|---|---|
| Service-level indicator (SLI) | A measure of service behavior defined for a population and window. | Percentage of eligible HTTP requests that complete successfully within 200 ms during the past 30 days. |
| Service-level objective (SLO) | A target for the indicator. | At least 99.9% of those requests meet the success and latency criteria in each rolling 30-day window. |
| Service-level agreement (SLA) | A commitment between provider and customer, with terms that can describe consequences. | A customer agreement defines how monthly availability is measured and specifies credits where the terms apply. |
| Error budget | For a proportion-based SLO, the allowed share of events that can miss the criterion in the defined window. | A 99.9% request-based target permits 0.1% of eligible requests to miss it. |
Worked Example: a request-based budget
In one rolling 30-day window, the service handles 200,000 eligible requests. Use the same window, population and definition to measure all outcomes. At a 99.9% target, the error budget is 0.1% of 200,000, or 200 requests. If 300 requests fail or exceed 200 ms, 199,700 meet the requirement and the SLI is 99.85%. The service has used 300 of its 200 allowed misses, or 150% of the budget. A team may pause feature releases under an agreed error-budget policy. The measurement itself does not impose a deployment action.
A time-based availability target uses a different denominator. At 99.9% over exactly 30 days, the allowed downtime is 0.001 × 30 × 24 × 60 = 43.2 minutes if availability is defined by elapsed time. Do not convert a request-based budget to minutes without additional assumptions.
Monitoring Architectures: Agent-Based vs. Agentless
| Pattern | How it collects | Example and trade-off |
|---|---|---|
| Local agent or exporter | Software on the target reads metrics or logs and exposes or forwards them. | Prometheus Node Exporter exposes Linux host metrics and is commonly scraped on port 9100 by Prometheus. Access to particular metrics depends on enabled collectors, permissions and platform. Plan the agent's lifecycle. |
| External or agentless check | A monitoring system uses an existing endpoint or protocol without deploying another collector to the target. | A remote HTTP synthetic check can measure user-facing latency without another host agent. It cannot report every internal detail. An SSH-based check still depends on the target's SSH service and access controls. |
The direction of transfer is independent of where a collector runs. An exporter can run on the host and still be scraped by the central system. Some collectors can buffer outgoing events, depending on their configuration. An external probe can test the same route clients use while a local exporter reports why a node became slow.
Simple Network Management Protocol (SNMP)
An SNMP manager can poll an agent on a managed device or receive an event notification from it. A Management Information Base (MIB) module defines managed objects, identifiers and their meaning. The agent supplies values for supported object instances. For example, the scalar sysDescr.0 is represented by numeric Object Identifier (OID) 1.3.6.1.2.1.1.1.0. Translation tools can help interpret the identifiers when the relevant MIB definitions are installed.
| Operation | Behavior |
|---|---|
| Polling | The manager requests current values. The usual UDP destination for agent requests is 161. GetNext traverses objects, and GetBulk retrieves multiple results in protocol versions that support it. |
| Trap | The agent sends an unsolicited notification. A trap does not get an SNMP response. The usual UDP destination for notifications is 162. |
| Inform | The sender requests a response and can retry based on its settings. An acknowledgment confirms receipt by the peer and does not prove that an operator handled the event. |
Community-based SNMPv1 and SNMPv2c do not provide cryptographic message authentication or privacy. SNMPv2c adds protocol features such as bulk retrieval and can convey 64-bit counters where the MIB supplies them. In SNMPv3's User-based Security Model (USM), choose the security level for the required protection:
noAuthNoPrivprovides neither cryptographic message authentication nor privacy.authNoPrivauthenticates and checks message integrity but leaves the scoped payload unencrypted.authPrivadds privacy to authenticated messages. Exact algorithms depend on what both peers support. The standards specify options including AES-128 privacy and HMAC-SHA-2 authentication.
For a managed device using Net-SNMP, configure a limited monitoring identity and its authentication and privacy settings through the approved client configuration. Avoid passing passphrases in command arguments or pasting them into shell history. With the relevant MIB files installed, snmptranslate -On SNMPv2-MIB::sysDescr.0 is a local lookup of the numeric identifier.
Health Checks, Thresholds, Events & Notifications
A health check should match what you need to know. A process can be alive while requests fail. For example on Kubernetes, a failed readiness probe makes a Pod unready for normal Service traffic, but the container can keep running. A failed liveness probe reaching its configured threshold can cause the kubelet to restart the container according to its policy. Use readiness for the ability to serve and liveness for conditions a restart can help recover.
An event is an observed occurrence. An alert is created when a configured rule evaluates to an actionable condition. A notification delivers that alert to a recipient or system. For example, a disk measurement may be logged every minute. An alert might fire only after low available space persists for 10 minutes. The receiver can send a nonurgent ticket for gradual saturation and page for an incident that is already affecting critical requests.
A duration can suppress short spikes. Different trigger and recovery thresholds can reduce repeated transitions near a boundary when the rule engine supports that design. Group alerts for one incident and route them to an owner who can act. Verify that alerting detects failures in the monitoring path itself, such as a stopped collector or missing heartbeats.
Webhooks for event delivery
A webhook can deliver structured alert data over HTTP to another service. For example, an Alertmanager receiver can post a batch of alerts to a ticketing adapter. The adapter can create or update a ticket based on configured logic. The payload does not grant authority to run administrative actions. Configure secure transport, authenticate the peer as appropriate, validate incoming data, and handle duplicate deliveries before relying on automatic actions.
Centralized Log Aggregation
A log pipeline connects local producers, a collector or forwarder, and central storage. Add host, service, and timestamp information so an operator can correlate records across machines. For example, collect system logs with the journal and forward selected entries through a configured collector. Check clock synchronization, retention and how the pipeline handles backpressure or an offline receiver.
A search backend can index selected fields and text. Loki indexes stream labels and stores compressed log chunks, so use stable labels suited to the queries you need. Protect sensitive log data and control who can read or retain it. TCP port 6514 is the assigned port for the RFC 5425 syslog-over-TLS transport mapping. Other authenticated transports exist. Confirm the protocol, certificate validation, buffering and delivery behavior in your deployment.
An internal web application became inaccessible overnight. Users have reported the symptom but the scope is unknown. Which step gives the best starting evidence?
Collect the exact errors, affected users and times, then compare recent changes and logs
Assume an Apache syntax error and replace its configuration
Roll back every package updated overnight
Treat the gateway as the confirmed cause before checking what failed
SNMP polling must provide cryptographic message authentication and payload privacy. Both peers support the stated algorithms. Which configuration meets those requirements?
SNMPv2c with a read-only community
SNMPv3 USM noAuthNoPriv
SNMPv3 USM authPriv with HMAC-SHA-2 authentication and AES-128 privacy
SNMPv3 USM authNoPriv with HMAC-SHA-2
At least 99.9% of eligible requests must succeed within 200 ms in a rolling 30-day window. In one such window, 300 of 200,000 requests miss the criterion. An agreed policy pauses feature rollouts when the allowed misses are exhausted. How should the team interpret the result?
99.9% is an SLA and the 300 misses are within the allowed budget
99.9% is the SLO and the 300 misses exceed the 200-request error budget
99.9% is the observed SLI and the budget allows 300 misses
99.9% is the budget and the team has the whole budget left
Sections you finish are checked off in the contents.