4.6 Data Engineering Code, IaC & CI/CD

4.6 Data Engineering Code, IaC & CI/CD

Data engineering code includes SQL transformations, Python orchestration, Scala or Java Spark jobs, shell automation, and infrastructure definitions. DEA-C01 does not reward memorizing syntax from every language; it tests whether the solution is repeatable, observable, testable, and safe to change.

Language and framework choice

Use SQL for set-based filtering, aggregation, windows, and joins that the engine can optimize. Use Python for orchestration, Glue scripts, Lambda handlers, and broad library support. Scala or Java can fit Spark or streaming teams that need JVM integration. R serves statistical workflows. Bash and PowerShell are useful for small deployment or operational steps but become fragile when they accumulate complex state.

Move large computation to the distributed engine. A Python loop that retrieves every row and transforms it locally discards Redshift, Athena, or Spark parallelism. Parameterize queries, project only needed columns, and avoid driver-side collect operations.

Software engineering baseline

Store code in version control with small reviewed changes. A useful repository separates source, tests, infrastructure, schemas, and environment configuration. Never commit passwords or access keys. Review data migrations independently from application code: a reversible code deployment can still launch an irreversible backfill, so require an explicit migration plan, bounded blast radius, reconciliation checks, and a recovery decision before promotion.

Test at several levels:

• Unit tests exercise pure transforms with small deterministic inputs, including nulls and boundary cases.
• Contract tests check producer and consumer schema compatibility.
• Data-quality tests validate uniqueness, completeness, referential integrity, and business rules.
• Integration tests use an isolated AWS environment to verify IAM, networking, service APIs, and actual formats.
• Reconciliation tests compare source and target counts and control totals.

Structured logs should include run ID, dataset, partition or watermark, source version, record counts, and error category. Do not log secrets or raw PII. Emit metrics for freshness, backlog age, rejected rows, and duration so an operator can detect a logically successful but incomplete run.

Infrastructure as Code

AWS CloudFormation declares resources and dependencies in JSON or YAML and manages them as stacks. It detects stack-level changes and can roll back a failed update, but data migrations and destructive replacements still need explicit review.

AWS Cloud Development Kit (AWS CDK) defines constructs in supported programming languages and synthesizes CloudFormation templates. CDK is not a separate deployment control plane; inspect the synthesized template and asset changes.

AWS Serverless Application Model (AWS SAM) extends CloudFormation with concise serverless resource types and tooling for packaging, local testing, and deployment. It is well suited to a Lambda, API Gateway, Step Functions, and DynamoDB pipeline. Use parameters, mappings, and separate stacks or stages rather than copying templates and hand-editing each environment.

Infrastructure as Code should define IAM roles, KMS keys, buckets, event mappings, log retention, alarms, and failure destinations together. A pipeline is incomplete if only its happy-path compute is declared.

CI/CD stages

A robust delivery path looks like this:

1. A pull request runs formatting, linting, unit tests, and secret scanning.
2. Schema compatibility and sample-data tests run.
3. CDK synth or SAM/CloudFormation validation produces a reviewable template.
4. Security checks identify wildcard IAM, public networking, and unencrypted resources.
5. The pipeline builds an immutable artifact identified by commit and digest.
6. A deployment role updates a development stack; integration and quality tests run.
7. The same artifact is promoted to later environments with approval gates.
8. Post-deployment canaries and freshness alarms verify the data path.

AWS CodeBuild can execute builds, CodePipeline can orchestrate stages, and CodeDeploy can manage supported deployment strategies. Equivalent third-party tools are valid when they provide the same controls.

Data-aware release safety

Rolling back code does not automatically roll back data. A bad transform may already have overwritten a partition. Prefer write-audit-publish: write to a run-specific staging location, validate it, then atomically update a catalog pointer, view, alias, or manifest. Preserve the prior version long enough to recover.

For schema changes, use expand-and-contract. Add a compatible field, deploy readers that tolerate both forms, migrate producers, backfill, then remove the old field only after dependency evidence permits it.

Jobs should be idempotent by partition and run ID. Record the code version, input watermark, schema version, and output location. When a deployment fails halfway, rerunning the same artifact against the same inputs should converge on one correct result.

Exam traps

Manual console creation is not repeatable deployment. A CDK construct without tests is not automatically safe. A green application deployment does not prove data completeness. And CI/CD credentials should come from short-lived assumed roles, not static keys stored in build variables.