Gather the Required Data

Why Schedules Slip Here

Task 5 of Domain III is where the plan meets the source systems. You have defined the data you need, named the data subject matter experts (SMEs), located the sources, and readied the workspace. Now the data has to physically move. This is the point at which AI project schedules slip, because every step depends on someone else's production system, someone else's change window, and someone else's approval.

Extraction Planning

Full versus incremental. A full extract copies everything. It is the simplest thing to reconcile and the heaviest load on the source. An incremental extract pulls only records changed since the last run — far cheaper, but it depends on a reliable change marker such as a modified-date column or change-data-capture log, and silently misses records when that marker is unreliable. The usual pattern is one full baseline extract followed by incrementals.

Impact on source production systems. The systems holding your data are running the business. A full extract of a claims database during business hours can degrade the application adjusters are using. Negotiate an extraction window with the system owner — off-peak, time-boxed, with an agreed abort condition — and put it on the schedule as a dependency. Assuming you can pull whenever you like is the fastest way to have your access revoked.

Point-in-time correctness. Data extracted while transactions are still posting is internally inconsistent: an order table read at 02:00 and a payments table read at 04:00 will not reconcile, and the mismatch looks exactly like a data quality problem. Ask for an as-of snapshot — a consistent read at a single logical point in time — and record that timestamp, both for validation now and for reproducibility later.

Transfer and Migration to the Development Environment

• Volume against bandwidth. Do the arithmetic before committing to a date. Multi-terabyte transfers over a constrained link take days, not hours, and may need staged transfers or a physical transfer appliance. That calculation is a schedule input, not a technical footnote.
• Secure transfer. Encrypted, approved channels only. Email attachments, personal cloud drives, and unencrypted removable media are the classic finding in a post-incident review.
• Proof that nothing was lost. Compare record counts and checksums on both sides, and record the comparison. "It looks about right" is not evidence, and truncated or partially transferred files rarely announce themselves.
• Sensitive fields in transit. Fields the model does not need — full identifiers, payment details, free-text notes containing personal data — should be masked, tokenized, or dropped at the source rather than moved and cleaned later. Every copy you create is a copy you must protect, log, and eventually delete.

Ongoing Feeds, Not One-Off Pulls

A one-off extract is enough to explore data. It is not enough to run a solution. A model in production needs the same data, in the same shape, on the same schedule, indefinitely. If the training data arrived because an analyst ran a query each Friday and emailed a spreadsheet, you do not have a data source — you have a hidden manual dependency that breaks shortly after go-live, when that analyst is on leave or changes role.

Treat the recurring feed as a deliverable of this phase: an automated, scheduled pipeline with a documented frequency, a named operational owner, and a support arrangement. Establishing it now costs a fraction of retrofitting it during Model Operationalization.

Validating Completeness and Accuracy During Collection

Validate as the data arrives, not after modeling starts. Five checks earn their place:

1. Reconcile record counts and control totals against the source. Row counts plus summed key numeric fields — total claim value, total units shipped — captured at the source and compared after landing.
2. Confirm the date range landed as expected. A request for 36 months that returns 27 usually means a filter, a retention purge, or a partition nobody mentioned.
3. Look for silent truncation. Text cut at a column width, numeric overflow, decimals rounded away by a type mismatch. Nothing fails; the values are simply and quietly wrong.
4. Check character encoding. A mismatch between source and target encoding corrupts names, addresses, and free text — and damages records for customers with accented or non-Latin names far more than others, making it a fairness problem as well as a quality one.
5. Confirm referential integrity across joined sources. Orphaned keys and unexpected join fan-out, where one row becomes five, change your record counts and distort the model's view of reality.

Refresh and Update Procedures

Every feed needs a documented refresh procedure covering frequency and timing, the named owner, monitoring, and alerting on failure — including the silent failures: a file that arrives empty, arrives late, or arrives containing yesterday's contents. Define who is notified, how quickly, and what the fallback is. Feeds that fail loudly get fixed; feeds that fail silently poison a model for weeks before anyone notices.

Collection and Validation Reference

Reconcile Before You Proceed

When an extract arrives with fewer records than expected, reconcile against the source with the data SME before proceeding. Do not re-plan the model around the smaller file, do not assume the gap is immaterial because the remaining volume "looks sufficient," and do not quietly continue with partial data. The gap has a cause — a filter, a retention policy, a permission boundary, an aborted transfer — and until you know which, you cannot say whether the data still represents the business problem. Proceeding quietly is the most heavily punished behavior in this task: the defect stays invisible until Model Evaluation, where it is expensive to unwind.