Databricks Data Engineer Associate Exam Flashcards

Q: What topics do these Databricks Data Engineer Associate flashcards cover?

They cover Lakehouse fundamentals, Delta Lake, Spark SQL, ingestion, streaming, ETL transformations, Lakeflow pipelines, Databricks jobs, performance tuning, Unity Catalog governance, and security.

Q: Are these flashcards copied from exam questions?

No. The cards are original study prompts written to reinforce concepts and workflows, not to reproduce live exam or practice question wording.

Q: How should I use these flashcards with practice questions?

Use the flashcards for active recall of concepts, then use practice questions to apply those concepts in scenarios. Revisit any topic where you miss questions or cannot explain the tradeoff clearly.

Question 1

Lakehouse architecture

Accepted Answer

A Lakehouse keeps data in open cloud storage while adding warehouse-style reliability, governance, and performance. In Databricks, Delta Lake, Unity Catalog, and compute engines work together to support BI, ETL, streaming, and ML on the same data foundation.

Question 2

Control plane vs. data plane

Accepted Answer

The control plane hosts workspace services such as the web application, APIs, notebooks, and job management. The data plane is where clusters and SQL warehouses process customer data in the configured cloud environment.

Question 3

Why the Lakehouse reduces data duplication

Accepted Answer

A traditional architecture often copies data from a lake into a separate warehouse. A Lakehouse lets curated tables, analytics, and pipelines operate directly on governed lake storage, reducing duplicate storage and extra ETL movement.

Question 4

Medallion architecture

Accepted Answer

Bronze tables preserve raw ingested data, silver tables clean and conform it, and gold tables provide business-ready aggregates or serving models. Each layer has a different purpose, so transformations should become stricter as data moves forward.

Question 5

Databricks SQL warehouse

Accepted Answer

A SQL warehouse is compute designed for SQL analytics, dashboards, and BI workloads. It is separate from general-purpose notebook clusters and is often paired with Photon for fast SQL execution.

Question 6

Delta Lake transaction log

Accepted Answer

The Delta transaction log records table changes as ordered commits. It lets readers see a consistent table snapshot while writers add, update, or delete data atomically.

Question 7

ACID transactions in Delta Lake

Accepted Answer

ACID guarantees mean a write either commits completely or not at all, committed data remains consistent, concurrent operations are isolated, and successful commits are durable. This prevents partial or conflicting table states.

Question 8

Time travel

Accepted Answer

Time travel queries a previous Delta table version or timestamp. It is useful for audits, rollback analysis, and comparing results before and after a pipeline change.

Question 9

MERGE INTO

Accepted Answer

MERGE INTO applies conditional updates, deletes, and inserts in one atomic operation. It is the standard Delta Lake pattern for upserts, change-data processing, and maintaining slowly changing dimensions.

Question 10

Schema enforcement vs. schema evolution

Accepted Answer

Schema enforcement rejects writes that do not match the table definition. Schema evolution intentionally allows compatible changes, such as adding new columns, when the pipeline is configured to accept them.

Question 11

WHERE vs. HAVING

Accepted Answer

WHERE filters input rows before grouping. HAVING filters grouped results after aggregation, so it can reference aggregate values such as counts or sums.

Question 12

Common Table Expression (CTE)

Accepted Answer

A CTE names an intermediate query with WITH, making complex SQL easier to read and reuse inside a single statement. It is useful for staging transformations without creating a permanent table.

Question 13

Window function

Accepted Answer

A window function computes a value across related rows without collapsing them into one row. Examples include ranking, running totals, lagged values, and deduplication with row_number.

Question 14

explode for nested data

Accepted Answer

explode turns each element of an array into a separate output row. It is a common step when normalizing semi-structured source data before joining or aggregating it.

Question 15

When to avoid Python UDFs

Accepted Answer

Avoid Python UDFs when a built-in Spark SQL or DataFrame function can do the same work. Built-ins are easier for Spark to optimize and usually perform better across distributed data.

Question 16

Auto Loader

Accepted Answer

Auto Loader incrementally discovers and ingests new files from cloud storage. It is designed for scalable file ingestion with schema tracking and checkpointed progress.

Question 17

COPY INTO

Accepted Answer

COPY INTO loads new files from a storage location into a Delta table using SQL. It is a simple pattern for incremental file ingestion when you do not need a continuously running stream.

Question 18

Rescued data column

Accepted Answer

A rescued data column stores fields that could not be parsed into the expected schema. It helps preserve unexpected source data while allowing the pipeline to continue and alert on schema drift.

Question 19

Bronze ingestion principle

Accepted Answer

Bronze ingestion should preserve source records with minimal transformation. Add operational metadata such as file path, load time, or source system so later layers can trace and repair data issues.

Question 20

External source extraction

Accepted Answer

When reading from external systems, separate extraction from transformation concerns. Land source data reliably first, then apply cleansing and business logic in governed Delta tables.

Question 21

Structured Streaming checkpoint

Accepted Answer

A checkpoint stores streaming progress and state so a query can restart without reprocessing everything from the beginning. It is required for reliable recovery in stateful or incremental pipelines.

Question 22

Streaming table

Accepted Answer

A streaming table is maintained from continuously or incrementally arriving data. It fits workloads where the table should update as new source records arrive instead of being rebuilt only by batch jobs.

Question 23

Trigger available now

Accepted Answer

An available-now trigger processes all currently available input and then stops. It is useful for scheduled incremental pipelines that need streaming semantics without a permanently running job.

Question 24

Watermark

Accepted Answer

A watermark tells Spark how long to keep state for late-arriving event-time data. It bounds state size while still allowing delayed records within the accepted lateness window.

Question 25

Append vs. complete streaming output

Accepted Answer

Append mode writes only newly finalized rows. Complete mode rewrites the full result table for aggregations where the whole result may change. Choose the mode that matches the query's update pattern.

Question 26

Silver-layer cleansing

Accepted Answer

Silver transformations standardize types, remove duplicates, apply valid ranges, resolve keys, and conform data across sources. The goal is trusted, reusable detail-level data.

Question 27

Gold-layer modeling

Accepted Answer

Gold tables are designed for consumption, such as dashboards, reports, feature tables, or application queries. They usually contain curated joins, aggregates, or business metrics from silver data.

Question 28

Idempotent pipeline

Accepted Answer

An idempotent pipeline can be rerun for the same input without creating duplicate or inconsistent output. Use deterministic keys, MERGE logic, checkpoints, and overwrite boundaries carefully.

Question 29

Deduplication with row_number

Accepted Answer

Use row_number over a partition key with an ordering rule to select one preferred record from duplicates. The ordering should be deterministic, such as newest event time plus a tie-breaker.

Question 30

SCD Type 1 vs. Type 2

Accepted Answer

Type 1 overwrites old attribute values and keeps only the latest state. Type 2 preserves history by expiring old records and inserting new current records with effective dates or status flags.

Question 31

Lakeflow Declarative Pipelines

Accepted Answer

Lakeflow Declarative Pipelines let engineers define desired tables, dependencies, and quality rules while Databricks manages execution planning, incremental processing, and pipeline monitoring.

Question 32

Pipeline expectation

Accepted Answer

An expectation defines a data quality rule inside a pipeline. Depending on configuration, invalid records can be counted, dropped, or cause the pipeline update to fail.

Question 33

LIVE references

Accepted Answer

LIVE table references express dependencies between pipeline tables. They let the pipeline engine understand ordering instead of relying on manual task sequencing inside notebooks.

Question 34

Declarative pipeline mindset

Accepted Answer

Declarative pipelines focus on what tables should exist and how they are derived. The platform handles dependency orchestration and incremental updates, which reduces manual control-flow code.

Question 35

Fail-fast data quality rule

Accepted Answer

Use a fail action when bad data should block downstream publication, such as broken primary keys or invalid reference data. Use warn or drop for issues that should be measured or isolated without stopping all processing.

Question 36

Job cluster

Accepted Answer

A job cluster is created for a specific job run and terminated after the run completes. It improves isolation and cost control for scheduled production workloads compared with leaving interactive compute running.

Question 37

Multi-task job

Accepted Answer

A multi-task job coordinates notebooks, SQL tasks, pipeline updates, or other steps with dependencies. It is used when production work needs ordered execution, retries, alerts, and monitoring in one workflow.

Question 38

Task dependency

Accepted Answer

A task dependency states that one task must finish before another starts. It prevents downstream transformations from reading incomplete upstream outputs.

Question 39

Databricks Asset Bundle

Accepted Answer

A Databricks Asset Bundle defines jobs, pipelines, notebooks, permissions, and related resources as deployable project configuration. It supports repeatable promotion across environments.

Question 40

Service principal for production jobs

Accepted Answer

A service principal is a non-human identity for automation. Using one for production jobs avoids tying scheduled workloads to an individual user's account or permissions lifecycle.

Question 41

OPTIMIZE

Accepted Answer

OPTIMIZE compacts many small Delta files into fewer larger files. This reduces file-listing and scan overhead, especially for tables that receive frequent small writes.

Question 42

VACUUM

Accepted Answer

VACUUM removes data files that are no longer referenced by the active Delta table and are outside the retention window. It reduces storage use but can limit time travel to older versions.

Question 43

Data skipping

Accepted Answer

Data skipping avoids reading files whose stored statistics show they cannot contain matching rows. It works best when frequently filtered columns are well organized in the table layout.

Question 44

Liquid clustering

Accepted Answer

Liquid clustering organizes Delta table data around chosen columns without requiring a fixed partition layout. It is useful when query patterns evolve or traditional partitioning would create too many small partitions.

Question 45

Partitioning tradeoff

Accepted Answer

Partitioning can improve pruning when queries filter by the partition column, but high-cardinality or rarely filtered partitions can create many small files. Choose partitions based on access patterns, not just column names.

Question 46

Unity Catalog namespace

Accepted Answer

Unity Catalog organizes data with a three-level namespace: catalog, schema, and object. This structure supports centralized governance across tables, views, functions, models, and volumes.

Question 47

Managed table vs. external table

Accepted Answer

A managed table is stored in a governed location managed by the platform. An external table points to data at an existing external location, so storage lifecycle and access design need extra care.

Question 48

Least privilege in Unity Catalog

Accepted Answer

Grant only the permissions needed, prefer groups over individual users, and grant at the narrowest practical scope. Review grants regularly so access does not accumulate over time.

Question 49

Column mask vs. row filter

Accepted Answer

A column mask transforms values in a sensitive column, such as hiding part of an identifier. A row filter limits which rows a user can see, such as restricting records by region or tenant.

Question 50

Delta Sharing

Accepted Answer

Delta Sharing provides governed data sharing without requiring consumers to receive copied files. Providers control what is shared and can update or revoke access through the sharing configuration.

Free Databricks Data Engineer Associate Exam Flashcards

Filter by Topic

Jump to Card

About These Databricks Data Engineer Associate Flashcards

Topics Covered

Frequently Asked Questions

Databricks Certified Data Engineer Associate

Explore More Databricks Certifications

Databricks Certified Data Analyst Associate

Databricks Certified Data Engineer Professional

Databricks Certified Machine Learning Associate

Databricks Generative AI Engineer Associate

Databricks Certified Associate Developer for Apache Spark

Databricks Certified Machine Learning Professional

More From This Family