Data Frequency, Flow, and Acquisition
Key Takeaways
- Frequency and flow—batch, streaming, and event-driven—must match the decision’s latency needs and the organization’s real pipeline capability
- Acquisition methods include APIs, scheduled extracts, warehouse queries, files, and primary/manual collection—each with access, SLA, and validation implications
- Lineage basics (origin, movement, transformation, storage) support trust, debugging, and auditability during and after sourcing
- Owners and stewards authorize use and explain meaning; without them, access and definition risks stay unmanaged
- When quality or access fails, adjust the acquisition approach; when personal data is involved, privacy and consent gates apply before collection and use
Data Frequency, Flow, and Acquisition
Quick Answer: Identifying sources is incomplete until you plan how often data moves, how it flows, and how you will obtain it under real access, ownership, lineage, quality, and privacy constraints. Domain 2 expects CBDA practitioners to choose acquisition approaches that fit the research question’s time sensitivity—and to change course when access or quality blocks the original plan.
A perfect source on a diagram is useless if you cannot extract it before the decision meeting, if streaming is promised but only monthly files exist, or if personal data is pulled without a lawful basis. This section connects frequency, flow, acquisition mechanics, stewardship, and privacy gates.
Frequency and flow: batch, streaming, event
Batch
Batch movement processes data in scheduled chunks—hourly, nightly, weekly. Most warehouse loads, CSV drops, and “run the monthly close extract” patterns are batch.
Fit when: decisions are periodic; full reconciliations matter; volumes are large; systems cannot tolerate continuous query load.
Risks: staleness between runs; long failure recovery; “as-of” confusion if business date ≠ load date.
Streaming
Streaming continuously ingests records or micro-batches with low latency—clickstream pipes, fraud scores, IoT feeds.
Fit when: decisions or monitoring truly need near-real-time signals.
Risks: higher engineering cost; harder exactly-once semantics; quality issues arrive continuously; overkill for monthly strategy choices.
Event-driven flows
Event-driven architectures emit business events (OrderPlaced, ClaimDenied, SubscriptionCancelled) that downstream consumers process. Related to streaming but centered on domain events rather than endless telemetry alone.
Fit when: process milestones must trigger analytics features or alerts.
Risks: incomplete event contracts; missing late/corrected events; consumers misunderstanding event meaning.
Matching flow to the decision (CBDA lens)
| Decision need | Usually appropriate flow | Poor fit |
|---|---|---|
| Monthly retention offer design | Batch warehouse snapshots | Mandatory real-time streaming build in two weeks |
| Intraday fraud block | Streaming / event scores | Weekly CSV only |
| Post-campaign diagnostic | Batch campaign + sales merge | Continuous stream with no identity resolution |
| Sensor predictive maintenance | Streaming or high-frequency batch | Annual survey |
Exam trap: equating “modern analytics” with streaming for every question. CBDA rewards latency fit, not fashion.
How and when to obtain data
Acquisition is the practical path from “source identified” to “dataset in the analytics environment.”
Common acquisition methods
| Method | How it works | Watch-outs |
|---|---|---|
| Warehouse / lake SQL access | Query governed tables in place or CTAS to project sandbox | Role permissions; row-level security; accidental full scans |
| Scheduled extracts / ETL jobs | Engineering pipeline lands files or tables on a cadence | Job failures; silent schema drift; late SLAs |
| APIs | Pull pages/resources programmatically | Auth tokens, rate limits, pagination, partial fields, versioning |
| File drops (SFTP, object storage) | Partner or system deposits files | Naming chaos, delimiter issues, incomplete files |
| Application exports | UI or admin export from CRM/ERP | Manual, not scalable, audit gaps |
| Primary / manual collection | Surveys, observations, SME-coded samples, time studies | Bias, consent, small samples, training of collectors |
| Vendor secure transfer | Third-party delivers under contract | Legal terms, encryption, retention, redistribution |
“When” is part of the plan
Acquisition timing should align with:
- Decision deadline (evidence ready date).
- Source refresh (do not extract mid-load if incomplete).
- Access windows (production freeze, maintenance).
- Privacy reviews (cannot start until approved).
- SME availability for definition confirmation on first pull.
A sourcing plan that ignores calendar reality becomes an assumption failure from Domain 1 dressed as a Domain 2 surprise.
Access is not a footnote
Before acquisition, confirm:
- Who can approve access (data owner, security, privacy).
- What environment is allowed (prod read replica, analytics sandbox, masked tier).
- What fields are in scope (column-level restrictions).
- How credentials are stored (no tokens in personal notes).
- How long data may be retained for the project.
On scenarios, “download production PII to a laptop to go faster” is almost never the correct option.
Lineage basics
Data lineage is the documented history of a data element’s origin, movement, transformation, and storage across its lifecycle. For CBDA sourcing, lineage answers:
- Where did this field come from (system, table, form, vendor)?
- What joins, filters, and calculations changed it?
- When was it last refreshed?
- Which dashboard or model consumes it?
You do not need an enterprise graph tool for every study, but you do need enough lineage notes that another analyst can reproduce the pull and that stakeholders can trust (or challenge) intermediate steps.
Lightweight lineage for a project extract:
- Source system and object names.
- Filter predicates (population, dates).
- Transformations (currency conversion, status mapping).
- Join keys and grain after join.
- Destination location and refresh stamp.
- Known exclusions.
Lineage supports debugging (“why is Region B empty?”), auditability, and ethical transparency when results influence decisions.
Ownership and stewards
| Role | Typical responsibility in acquisition |
|---|---|
| Data owner | Accountable for data domain; approves use and risk acceptance |
| Data steward | Day-to-day definition quality, reference values, issue triage |
| System owner / platform team | Technical access paths, pipeline reliability, performance |
| Privacy / security | Lawful basis, masking, access patterns, retention |
| BA / analytics lead | Ensures acquired data still maps to research question; documents limits |
Without owners and stewards, teams acquire orphaned data: files nobody will explain when anomalies appear. CBDA scenarios often make the best next step engage the steward/owner before inventing field meanings.
Adjust acquisition when quality or access fails
Domain 2 is iterative. Planned acquisition fails in predictable ways:
| Failure | Adjustment options |
|---|---|
| Access denied / delayed | Narrow field list; use certified aggregates; escalate with business sponsor; re-sequence work; consider approved proxy |
| Extract incomplete | Re-pull with validation counts; quarantine bad files; partial population analysis with stated limits |
| API unstable | Fall back to warehouse replica; reduce frequency; cache snapshots |
| Quality below threshold | Change source; strengthen filters; sample investigation; refine question grain; postpone claims that need that field |
| Vendor feed late | Use last good external snapshot with as-of labeling; shift decision date; drop external features temporarily |
| Join keys fail | Steward workshop; fuzzy match with confidence thresholds; analyze unlinked segments separately |
Principle: Adjust the acquisition approach (or the question’s data ambition) deliberately and visibly. Do not silently ship on a failed pull, and do not treat the first blocked API as proof the business question was worthless.
Validation after acquisition (tie-in)
Even before deep Domain 2 quality chapters, plan basic post-pull checks:
- Row counts vs source control totals.
- Date range coverage.
- Null rates on must-have fields.
- Referential integrity on keys.
- Freshness timestamp present.
- Unexpected category values in reference fields.
These checks decide whether to proceed, re-pull, or re-plan.
Privacy and consent gates for personal data
When acquiring data about people (customers, patients, employees, citizens), privacy is a sourcing gate, not a final-report footnote.
Gates to pass before acquisition
- Purpose limitation: Is use consistent with stated purposes and policies?
- Lawful basis / consent where required: Especially marketing analytics, special category data, cross-border transfers.
- Minimum necessary: Pull only fields needed for the research question.
- Access environment: Masked, anonymized, or secure enclave as required.
- Retention and deletion: How long project copies live; who deletes them.
- Third-party terms: Vendor data may forbid certain model uses or combining with other sets.
- Transparency to stakeholders: Decision makers know what personal data underpins recommendations.
Techniques often paired with gated acquisition
- Anonymization / aggregation when individual identity is unnecessary.
- Pseudonymization (replace identifiers with tokens) when linkage is needed inside a controlled boundary.
- Suppression of small cells that re-identify.
- Contractual controls with processors and commercial data providers.
Exam theme: If a stem offers a fast path that copies identifiable personal data into uncontrolled tools, the better option is almost always satisfy privacy/consent requirements or redesign the dataset—not “speed over policy.”
Putting frequency, flow, and acquisition together
Example plan snippet — claims denial diagnostic
- Decision: Prioritize denial-prevention interventions next quarter (batch OK).
- Flow: Nightly warehouse load of adjudicated claims + weekly operational extract of pending claims for trend early-warning (hybrid batch).
- Acquisition: SQL to analytics sandbox for 18 months adjudicated history; API not required.
- Lineage: Document claim_id grain, denial_reason mapping table version, exclusion of test providers.
- Stewards: Claims data steward confirms reason-code dictionary; privacy approves de-identified member tokens.
- Fallback: If pending-claims API access slips, proceed with adjudicated-only diagnostic and label latency limit.
- Privacy gate: No free-text clinical notes in v1 without additional review.
This plan shows CBDA thinking: latency matched to decision, explicit acquisition path, ownership, lineage, fallback, and privacy.
CBDA exam habits for frequency, flow, and acquisition
- Prefer latency matched to the decision, not streaming-by-default.
- Prefer acquisition options that include access approval and validation.
- Prefer lineage and steward engagement when definitions or trust are unclear.
- Prefer adjust acquisition / use fallbacks when pulls fail—before abandoning the research intent.
- Prefer privacy and consent gates whenever personal data is in play.
- Reject uncontrolled PII copies, silent use of failed extracts, and “collect everything continuously” without purpose.
Leadership needs a monthly decision on which retention offers to expand. Data engineering can deliver a reliable month-end warehouse snapshot but not a production streaming pipeline this quarter. What is the BEST CBDA-aligned choice?
Which description BEST matches data lineage in a sourcing context?
An API extract of customer emails and phone numbers is proposed for a churn study sandbox without privacy review, because “the model will be more accurate.” What should the BA do?