Hardware, Metadata, and Config Files

Hardware Metrics: What Host Dashboards Really Show

Before you customize dashboards, Datadog collects host-level hardware metrics through the Agent's system checks. These timeseries reflect physical or virtual resource usage:

Sustained saturation on CPU, memory, or disk correlates with incident scenarios: slow databases, Agent check timeouts, or missing logs because disk writes fail. The exam may ask which metric you inspect first for a given symptom—match the metric to the resource bottleneck, not whichever graph is prettiest.

Virtualization adds nuance: CPU steal time on oversubscribed hypervisors can throttle guests even when guest CPU graphs look moderate. You do not need deep hypervisor expertise, but recognize that host metrics contextualize application slowness.

Metadata vs Metrics: Two Different Languages

Timeseries metrics are numeric measurements indexed by time—system.cpu.user, system.mem.used, system.disk.in_use. They support aggregation: averages, maxes, rates.

Metadata describes identity and context without being a plotted value itself. In Datadog, metadata often appears as tags attached to hosts, metrics, logs, or traces:

• env:staging — deployment environment
• role:database — functional role
• team:payments — ownership
• os:linux — operating system family

An exam favorite: which item is metadata? role:database is a tag (metadata); system.cpu.user is a metric. Tags filter and group; metrics measure.

Cardinality caution

Tags should come from bounded sets. Using user_id:984302 or request_id:abc creates unbounded cardinality—too many unique tag values. Datadog may struggle to aggregate, dashboards slow down, and billing for custom metrics can spike. The Fundamentals exam tests whether you recognize good tags (env, service, role) vs bad tags (unique per request).

Configuration Files: Layout and Purpose

On a standard Linux install, the Agent's main configuration file is:

/etc/datadog-agent/datadog.yaml

This file holds global settings: api_key, site, hostname, tags, proxy configuration, and feature toggles. Integration-specific configs live under:

/etc/datadog-agent/conf.d/<integration>.d/conf.yaml

Windows paths differ (%ProgramData%\Datadog\...) but the same conceptual split applies: one main file plus per-integration snippets.

When a stem references "main Agent configuration file on Linux," /etc/datadog-agent/datadog.yaml is the expected answer—not log paths, not random JSON under /opt.

Editing YAML Safely

Agent configuration uses YAML, a human-readable format built on indentation instead of braces. Rules that matter on exams and in production:

1. Indentation defines nesting—usually two spaces per level
2. Tabs are dangerous—many parsers reject tab-indented YAML
3. Comments start with # and are ignored
4. Strings with special characters may need quotes
5. Duplicate keys—last wins or parse fails depending on parser settings; treat as errors

Example fragment:

api_key: "<YOUR_API_KEY>"
site: datadoghq.com
tags:
  - env:prod
  - team:checkout
logs_enabled: true

If an engineer pastes a tab before tags:, the Agent may fail to start entirely—a config parse error, not a network issue. Always run agent configcheck or agent status after edits (Chapter 5 goes deeper).

JSON in Datadog Workflows

JSON appears heavily in API automation—creating monitors, dashboards, or querying metrics programmatically. Unlike YAML config files, JSON uses explicit { } brackets and "key": "value" pairs. Datadog API responses are JSON documents you parse in Python, curl, or Terraform providers.

Do not confuse the formats: YAML for Agent files on disk, JSON for HTTP API bodies. An exam distractor might show JSON syntax inside datadog.yaml—invalid unless properly embedded as a string.

Environment Variables and File Permissions

Secrets such as the API key can live in datadog.yaml or arrive via DD_API_KEY. Regardless of storage, apply least privilege file permissions—typically root or the dd-agent service account can read datadog.yaml, not world-readable chmod 644 on shared servers with untrusted users.

World-writable config (chmod 777) is never correct—any local user could hijack the key or break settings.

Hardware Metadata from the Cloud

Cloud providers expose metadata endpoints (AWS instance ID, Azure VM name, GCP zone). Datadog can enrich hosts with these attributes as tags. That is still metadata, not a replacement for system.cpu metrics. Questions may ask how to group all database hosts—role:database beats instance-id:i-0abc123 for human dashboards, though instance IDs are fine as supplementary tags.

Worked Scenario

Problem: After adding logs_enabled: true to datadog.yaml, the Agent fails to start.

Investigation path:

1. Check YAML indentation under the new key
2. Validate there is no tab character introduced by the editor
3. Confirm logs section syntax matches documented structure
4. Read Agent startup logs for parse errors

Lesson: Config-file modification questions often test syntax and placement, not whether logs are a good idea.

Takeaway Table