Free Databricks Data Engineer Associate Exam Flashcards

A Lakehouse keeps data in open cloud storage while adding warehouse-style reliability, governance, and performance. In Databricks, Delta Lake, Unity Catalog, and compute engines work together to support BI, ETL, streaming, and ML on the same data foundation.

The control plane hosts workspace services such as the web application, APIs, notebooks, and job management. The data plane is where clusters and SQL warehouses process customer data in the configured cloud environment.

A traditional architecture often copies data from a lake into a separate warehouse. A Lakehouse lets curated tables, analytics, and pipelines operate directly on governed lake storage, reducing duplicate storage and extra ETL movement.

Bronze tables preserve raw ingested data, silver tables clean and conform it, and gold tables provide business-ready aggregates or serving models. Each layer has a different purpose, so transformations should become stricter as data moves forward.

A SQL warehouse is compute designed for SQL analytics, dashboards, and BI workloads. It is separate from general-purpose notebook clusters and is often paired with Photon for fast SQL execution.

The Delta transaction log records table changes as ordered commits. It lets readers see a consistent table snapshot while writers add, update, or delete data atomically.

ACID guarantees mean a write either commits completely or not at all, committed data remains consistent, concurrent operations are isolated, and successful commits are durable. This prevents partial or conflicting table states.

Time travel queries a previous Delta table version or timestamp. It is useful for audits, rollback analysis, and comparing results before and after a pipeline change.

MERGE INTO applies conditional updates, deletes, and inserts in one atomic operation. It is the standard Delta Lake pattern for upserts, change-data processing, and maintaining slowly changing dimensions.

Schema enforcement rejects writes that do not match the table definition. Schema evolution intentionally allows compatible changes, such as adding new columns, when the pipeline is configured to accept them.

WHERE filters input rows before grouping. HAVING filters grouped results after aggregation, so it can reference aggregate values such as counts or sums.

A CTE names an intermediate query with WITH, making complex SQL easier to read and reuse inside a single statement. It is useful for staging transformations without creating a permanent table.

A window function computes a value across related rows without collapsing them into one row. Examples include ranking, running totals, lagged values, and deduplication with row_number.

explode turns each element of an array into a separate output row. It is a common step when normalizing semi-structured source data before joining or aggregating it.

Avoid Python UDFs when a built-in Spark SQL or DataFrame function can do the same work. Built-ins are easier for Spark to optimize and usually perform better across distributed data.

Auto Loader incrementally discovers and ingests new files from cloud storage. It is designed for scalable file ingestion with schema tracking and checkpointed progress.

COPY INTO loads new files from a storage location into a Delta table using SQL. It is a simple pattern for incremental file ingestion when you do not need a continuously running stream.

A rescued data column stores fields that could not be parsed into the expected schema. It helps preserve unexpected source data while allowing the pipeline to continue and alert on schema drift.

Bronze ingestion should preserve source records with minimal transformation. Add operational metadata such as file path, load time, or source system so later layers can trace and repair data issues.

When reading from external systems, separate extraction from transformation concerns. Land source data reliably first, then apply cleansing and business logic in governed Delta tables.

A checkpoint stores streaming progress and state so a query can restart without reprocessing everything from the beginning. It is required for reliable recovery in stateful or incremental pipelines.

A streaming table is maintained from continuously or incrementally arriving data. It fits workloads where the table should update as new source records arrive instead of being rebuilt only by batch jobs.

An available-now trigger processes all currently available input and then stops. It is useful for scheduled incremental pipelines that need streaming semantics without a permanently running job.

A watermark tells Spark how long to keep state for late-arriving event-time data. It bounds state size while still allowing delayed records within the accepted lateness window.

Append mode writes only newly finalized rows. Complete mode rewrites the full result table for aggregations where the whole result may change. Choose the mode that matches the query's update pattern.

Silver transformations standardize types, remove duplicates, apply valid ranges, resolve keys, and conform data across sources. The goal is trusted, reusable detail-level data.

Gold tables are designed for consumption, such as dashboards, reports, feature tables, or application queries. They usually contain curated joins, aggregates, or business metrics from silver data.

An idempotent pipeline can be rerun for the same input without creating duplicate or inconsistent output. Use deterministic keys, MERGE logic, checkpoints, and overwrite boundaries carefully.

Use row_number over a partition key with an ordering rule to select one preferred record from duplicates. The ordering should be deterministic, such as newest event time plus a tie-breaker.

Type 1 overwrites old attribute values and keeps only the latest state. Type 2 preserves history by expiring old records and inserting new current records with effective dates or status flags.

Lakeflow Declarative Pipelines let engineers define desired tables, dependencies, and quality rules while Databricks manages execution planning, incremental processing, and pipeline monitoring.

An expectation defines a data quality rule inside a pipeline. Depending on configuration, invalid records can be counted, dropped, or cause the pipeline update to fail.

LIVE table references express dependencies between pipeline tables. They let the pipeline engine understand ordering instead of relying on manual task sequencing inside notebooks.

Declarative pipelines focus on what tables should exist and how they are derived. The platform handles dependency orchestration and incremental updates, which reduces manual control-flow code.

Use a fail action when bad data should block downstream publication, such as broken primary keys or invalid reference data. Use warn or drop for issues that should be measured or isolated without stopping all processing.

A job cluster is created for a specific job run and terminated after the run completes. It improves isolation and cost control for scheduled production workloads compared with leaving interactive compute running.

A multi-task job coordinates notebooks, SQL tasks, pipeline updates, or other steps with dependencies. It is used when production work needs ordered execution, retries, alerts, and monitoring in one workflow.

A task dependency states that one task must finish before another starts. It prevents downstream transformations from reading incomplete upstream outputs.

A Databricks Asset Bundle defines jobs, pipelines, notebooks, permissions, and related resources as deployable project configuration. It supports repeatable promotion across environments.

A service principal is a non-human identity for automation. Using one for production jobs avoids tying scheduled workloads to an individual user's account or permissions lifecycle.

OPTIMIZE compacts many small Delta files into fewer larger files. This reduces file-listing and scan overhead, especially for tables that receive frequent small writes.

VACUUM removes data files that are no longer referenced by the active Delta table and are outside the retention window. It reduces storage use but can limit time travel to older versions.

Data skipping avoids reading files whose stored statistics show they cannot contain matching rows. It works best when frequently filtered columns are well organized in the table layout.

Liquid clustering organizes Delta table data around chosen columns without requiring a fixed partition layout. It is useful when query patterns evolve or traditional partitioning would create too many small partitions.

Partitioning can improve pruning when queries filter by the partition column, but high-cardinality or rarely filtered partitions can create many small files. Choose partitions based on access patterns, not just column names.

Unity Catalog organizes data with a three-level namespace: catalog, schema, and object. This structure supports centralized governance across tables, views, functions, models, and volumes.

A managed table is stored in a governed location managed by the platform. An external table points to data at an existing external location, so storage lifecycle and access design need extra care.

Grant only the permissions needed, prefer groups over individual users, and grant at the narrowest practical scope. Review grants regularly so access does not accumulate over time.

A column mask transforms values in a sensitive column, such as hiding part of an identifier. A row filter limits which rows a user can see, such as restricting records by region or tenant.

Delta Sharing provides governed data sharing without requiring consumers to receive copied files. Providers control what is shared and can update or revoke access through the sharing configuration.