7.4 Cloud Telemetry Sources, Collection Architecture & Retention

7.4 Cloud Telemetry Sources, Collection Architecture & Retention

Quick Answer: Effective cloud monitoring combines signals from the identity and management planes, provider services, virtual networks, workloads, applications, data stores, and posture tools. A resilient collection architecture preserves an original copy, routes events through account or regional layers, normalizes data for analysis, and monitors the health of the pipeline itself. Retention is risk based: an event must remain available long enough for detection, investigation, audit, and legal needs without violating privacy or creating uncontrolled cost.

Build a telemetry inventory before choosing a SIEM

Cloud environments produce several evidence classes. Each answers a different investigative question:

Management-plane trails are essential, but they do not show every data read or application decision. Network flow records show connections, but often not payload semantics. Application logs add business context, but an attacker may influence their contents. Correlation across sources produces stronger conclusions than treating any single log as complete.

Source configuration and responsibility

Telemetry is subject to shared responsibility. A provider may make an audit service available, while the customer must enable the right event categories, select destinations, protect keys, and pay data or retention charges. Managed SaaS may expose only an administrative activity feed and export API. IaaS customers can also collect guest and application data. The monitoring plan must state which events the provider produces, which the customer configures, and which are unavailable.

For every source, record the owner, regions and accounts covered, event classes enabled, latency, expected volume, schema, destination, encryption, retention, and known blind spots. Revisit the inventory when a new service or region is approved.

Cascading collection architecture

A large organization rarely sends every producer directly to one SIEM endpoint. A typical cascade is:

1. Native source: the provider service, workload agent, application, or posture tool emits an event.
2. Local routing layer: an account, subscription, project, cluster, or SaaS connector forwards events and buffers temporary failures.
3. Regional aggregation: high-volume records are compressed, filtered under an approved rule, or routed to a regional destination to respect latency and residency.
4. Protected archive: an immutable or tightly controlled copy of original events is stored outside ordinary workload administration.
5. Processing and enrichment: parsers normalize time, identity, asset, and network fields; enrichment adds owner, sensitivity, vulnerability, and threat context.
6. Detection and analysis: SIEM, security analytics, and response systems evaluate rules, behavior, and correlated attack paths.

The archive and analysis platform serve different purposes. The SIEM may retain a fast searchable subset, while the archive preserves original evidence for longer investigation or audit periods. If a parser fails or a rule changes, analysts can reprocess the original data.

Integrity, ordering, and time

Security evidence should be protected in transit and at rest. Limit writers to append or ingestion actions; separate retention and deletion authority; record access to the evidence store; and use object immutability or equivalent controls where required. Hashes and provider integrity features can support evidence validation, but chain-of-custody documentation still records acquisition, transfer, transformation, and analyst access.

Distributed systems do not guarantee perfect arrival order. Clocks may drift, services may batch events, and cross-region links may delay delivery. Preserve the event's source timestamp, ingestion timestamp, and a stable identifier. Use reliable time synchronization where the customer controls the host. Detection logic should tolerate defined delay rather than assuming every timestamp is exact.

Collection-pipeline availability

The telemetry pipeline is itself a production security service. Monitor it for:

• a source that suddenly emits zero events;
• forwarding permission or encryption-key failures;
• queue depth, rejected events, and throttling;
• parser error rates and unexpected schema changes;
• archive write failures or retention changes;
• unusual deletion, exclusion, or filter changes; and
• cost spikes that could trigger a dangerous attempt to disable collection.

Use buffering and retry with bounded backoff, route failures to a dead-letter destination, and alert on both loss and excessive delay. Avoid a circular design in which the only alert that the SIEM is unavailable must be generated by that same SIEM.

Retention and privacy

Longer is not automatically better. Determine retention from detection windows, incident investigation needs, regulatory or contract requirements, litigation holds, and recovery objectives. Balance those needs against cost, data minimization, employee privacy, and jurisdictional restrictions. Restrict sensitive payload capture; tokenize or mask fields when full content is unnecessary; and separate high-risk logs from broad analyst access.

Document tiering: how long data remains in high-speed analytics, when it moves to lower-cost archive, and when it is defensibly deleted. Test retrieval from the archive. A retention statement has little value if investigators cannot restore and search the evidence within the incident timeline.