A company needs to design a data pipeline that can handle unpredictable traffic spikes up to 10x normal volume during promotional events. The solution must automatically scale without manual intervention. Which GCP architecture pattern should be used?
Gcp De Designing Systems
Question · Easy
A financial services firm is designing a data lake architecture that must support both structured and unstructured data with different access patterns. They need the ability to query data in-place without moving it. Which storage and query combination meets these requirements?
Question · Medium
Introduction & Exam Overview
0.1 About the Google Cloud Professional Data Engineer Exam
Study guide
An e-commerce company needs to design a real-time recommendation system that processes user behavior events and updates recommendations within seconds. The system must handle millions of events per second with low latency. Which architecture is most appropriate?
Tech Exam Prep Podcast
Listen on Spotify
Podcast
A healthcare organization is designing a data pipeline for processing sensitive patient data. They must ensure HIPAA compliance while allowing data scientists to analyze de-identified data. Which approach should they implement?
A gaming company needs to design a leaderboard system that can handle millions of players with real-time score updates and millisecond query latency. Which GCP services should they use?
Question · Hard
A manufacturing company wants to design a hybrid data architecture that keeps sensitive on-premises data but processes analytics in the cloud. They need to join on-premises data with cloud data without full migration. What is the best approach?
0.2 Study Strategy & Cloud Data Engineering Methodology
A retail company is designing a data warehouse solution that must support thousands of concurrent analysts running complex queries without performance degradation. Which BigQuery feature is most critical for this requirement?
A media streaming company needs to design a content recommendation pipeline that processes viewing history, updates user profiles, and generates personalized recommendations. The system must handle both batch historical processing and real-time updates. Which pattern should they implement?
A financial institution needs to ensure that sensitive credit card data is encrypted at all times, including during processing. Which encryption approach meets this requirement in GCP?
An organization needs to implement data governance that automatically classifies sensitive data across multiple GCP projects and enforces access policies based on classification. Which solution should they implement?
Chapter 1: Designing for Security and Compliance
1.1 IAM Roles, Service Accounts, and Least Privilege for Data
A company operates in multiple countries and must ensure EU customer data never leaves EU regions while US data stays in US regions. How should they design their data architecture?
A healthcare provider needs to implement audit logging for all data access to patient records to meet regulatory requirements. Which GCP features should they enable?
A company needs to implement row-level security in BigQuery so analysts can only see data for regions they are authorized to access. Which approach should they use?
An organization needs to securely share sensitive datasets with external partners while maintaining control over who can access what data and for how long. Which GCP solution should they use?
1.2 Data Encryption, Key Management, and Privacy with Cloud DLP
A company is migrating a 500TB on-premises data warehouse to BigQuery. They need to minimize downtime and ensure data consistency. Which migration strategy should they use?
A retail company needs to ensure their data pipeline continues operating during a regional outage. They have an RPO of 1 hour and RTO of 4 hours. Which architecture meets these requirements?
A streaming data pipeline experiences occasional failures due to malformed JSON messages. The pipeline must continue processing valid messages and handle errors gracefully. Which Dataflow feature should be implemented?
A company is migrating from an on-premises Apache Kafka cluster to GCP. They need a managed service with minimal code changes. Which migration approach is most appropriate?
1.3 Compliance, Data Sovereignty, and Geographic Placement
A data engineer needs to design a pipeline that can recover from infrastructure failures without losing data or reprocessing already-processed records. Which combination of features ensures exactly-once processing semantics?
A SaaS company needs to migrate customer data from a multi-tenant on-premises database to Cloud Spanner while maintaining data isolation. Which approach ensures secure migration?
An organization needs to design a disaster recovery strategy for their BigQuery datasets with a 1-hour RPO. What is the most cost-effective solution?
A data pipeline processes critical financial transactions and must continue operating even during partial network outages. Which design pattern should be implemented?
1.4 Multi-Environment Architecture: Development, Staging, and Production
A data engineer needs to process 10TB of historical log files stored in Cloud Storage and load the results into BigQuery. The processing involves complex transformations using Python libraries. Which service is most appropriate?
Gcp De Ingesting Processing
A company needs to run Spark jobs on a 5TB dataset for machine learning preprocessing. They want a fully managed service without cluster management overhead. Which GCP service should they use?
A data engineer needs to join a 500GB BigQuery table with a 1GB lookup table. The lookup table changes daily. What is the most efficient approach?
An organization needs to process daily batch files from an SFTP server and load them into BigQuery with validation and transformation. Which solution is most appropriate?
Chapter 2: Designing for Reliability and Resilience
2.1 High Availability, Disaster Recovery, and Failover Patterns
A data pipeline needs to process Parquet files from Cloud Storage and apply complex windowing aggregations based on event time. Which Apache Beam feature should be used?
A company needs to incrementally process only new records added to a large BigQuery table since the last run. Which pattern provides the best performance?
A data engineer needs to denormalize nested JSON data from API responses before loading to BigQuery. The JSON has multiple levels of nesting with arrays. Which approach handles this best?
A streaming pipeline needs to aggregate clickstream events into 5-minute windows but events may arrive up to 30 minutes late. Which Apache Beam windowing strategy should be used?
2.2 Fault Tolerance and ACID Guarantees in Distributed Systems
A real-time analytics pipeline processes events from Cloud Pub/Sub and writes aggregated results to BigQuery. The pipeline must ensure no duplicate records in BigQuery even if the pipeline restarts. Which delivery guarantee and sink combination is appropriate?
A streaming application needs to enrich events with reference data that changes hourly. The enrichment must have low latency and not impact throughput. Which pattern should be implemented?
A company needs to detect anomalies in IoT sensor data in real-time. The detection algorithm requires the last 1 hour of data for each sensor. Which Dataflow state management approach is most efficient?
A streaming pipeline uses Cloud Pub/Sub and experiences occasional message processing failures due to temporary external service outages. How should these failures be handled?
2.3 Data Validation, Cleansing, and Quality Assurance with Dataform
A data engineer needs to calculate running totals per user from a clickstream with events arriving out of order. Which Apache Beam feature ensures correct totals?
A streaming pipeline must write to BigQuery with exactly-once semantics and handle schema evolution automatically. Which configuration achieves this?
A company needs to join a high-throughput clickstream (Pub/Sub) with slowly-changing user profile data (BigQuery) in real-time. Which pattern minimizes latency?
A streaming pipeline must process events in the order they were generated for each user session. Events may arrive out of order. Which windowing and trigger strategy ensures ordered processing?
Loading more…