Agent Logs and Debug

Quick Answer: Agent logs on Linux are in /var/log/datadog/. Enable debug by setting log_level: DEBUG in datadog.yaml and restarting the Agent, or use DD_LOG_LEVEL=debug in containers. On systemd hosts, journalctl -u datadog-agent.service shows service startup errors.

After datadog-agent status points to a failing component, Agent logs provide the narrative: parse errors, permission denials, proxy failures, and intake rejections. The Fundamentals exam tests log locations, how to raise verbosity, and when OS-level journals beat tailing agent.log.

Why Logs Come After Status

status tells you what is wrong at a summary level; logs explain why. Exam scenarios often say an integration "looks configured" but metrics never arrive. The correct next step is frequently review Agent logs and enable debug mode if needed — not deleting API keys or rebuilding dashboards.

Agent Log Locations

The primary file is typically agent.log. Datadog rotates logs when a file reaches 10 MB by default, keeping one backup (agent.log.1). You can tune rotation with:

• log_file_max_size — max bytes per file (default 10,485,760)
• log_file_max_rolls — number of backup files to retain (default 1)

Integration-specific troubleshooting may also reference check logs under paths like /var/log/datadog/<CHECK_NAME>.log on Linux, but the Fundamentals exam emphasizes the main Agent log directory first.

Reading Logs on Linux

Common commands:

sudo tail -f /var/log/datadog/agent.log
sudo grep -i error /var/log/datadog/agent.log

Look for lines mentioning the failing integration, ERROR/WARN levels, forwarder retries, or authentication failures against Datadog intake endpoints.

systemd Journal vs Agent Log Files

On systemd-based Linux, the Agent may fail to start before agent.log receives useful entries. If systemctl status datadog-agent shows only a generic failure, run:

sudo journalctl -u datadog-agent.service

The journal often exposes YAML syntax errors, missing directories, or permission problems during boot — exactly the scenario tested when systemctl status "is not detailed enough."

Enabling Debug Mode

Debug mode increases log detail for Support and advanced self-troubleshooting.

Bare-metal / VM Agent

1. Edit the main config file (/etc/datadog-agent/datadog.yaml on Linux).
2. Set log_level: DEBUG (uncomment and change from the default INFO).
3. Restart the Agent so the new level takes effect.
4. Reproduce the issue, capture logs, then return to INFO when finished.

On the Exam: Editing datadog.yaml alone does not reload settings. Many scenarios hinge on forgetting to restart the Agent after a config change.

Containerized Agent

Set the environment variable DD_LOG_LEVEL=debug when starting the container (or equivalent in Helm/DaemonSet manifests). This mirrors the log_level setting without editing a file inside the image.

Runtime log level (Agent v6.19+ / v7.19+)

For a temporary change without editing YAML:

agent config set log_level debug

Remote flares initiated from Fleet Automation can also enable debug briefly; Datadog resets the log level after the flare completes.

Debug Mode Risks and Cleanup

Debug logging is powerful but not free:

• Log volume grows quickly on busy hosts.
• Disk pressure can follow if rotation settings are tight.
• CPU overhead increases slightly while verbose tracing is active.

Best practice: enable DEBUG only for the reproduction window, collect evidence, then revert to INFO (or WARN in very chatty environments). Leaving DEBUG on permanently is a common production mistake — rarely the exam's correct "next step."

Connecting Logs to Other Troubleshooting Tools

Worked Scenario: Integration Configured but Silent

A candidate adds conf.d/mysql.d/conf.yaml, restarts MySQL, and still sees no metrics. status lists mysql with an obscure WARNING. The exam asks for the strongest next step after status.

1. datadog-agent check mysql — immediate integration error output.
2. Tail /var/log/datadog/agent.log — persistent errors between scheduled runs.
3. Enable DEBUG + restart — if the WARNING text is too vague at INFO level.

Deleting the API key or disabling TLS are distractors. The exam rewards log-driven diagnosis aligned with Datadog documentation.

Log Collection Pitfalls (Agent as Log Shipper)

When troubleshooting log collection (distinct from Agent debug logs), remember:

• If two YAML files define the same log source, the Agent uses the first file alphabetically and ignores duplicates — a source of "I edited the wrong file" confusion.
• Applications that write without newline terminators can cause the tailer to wait indefinitely for a complete line.

These behaviors appear in Fundamentals troubleshooting items even though they involve conf.d log configs rather than agent.log itself.