Professional Data Engineer Free Practice Questions — Page 8
Question 70
Your car factory is pushing machine measurements as messages into a Pub/Sub topic in your Google Cloud project. A Dataflow streaming job, that you wrote with the Apache Beam SDK, reads these messages, sends acknowledgment to Pub/Sub, applies some custom business logic in a DoFn instance, and writes the result to BigQuery. You want to ensure that if your business logic fails on a message, the message will be sent to a Pub/Sub topic that you want to monitor for alerting purposes. What should you do?
A. Enable retaining of acknowledged messages in your Pub/Sub pull subscription. Use Cloud Monitoring to monitor the subscription/num_retained_acked_messages metric on this subscription.
B. Use an exception handling block in your Dataflow’s DoFn code to push the messages that failed to be transformed through a side output and to a new Pub/Sub topic. Use Cloud Monitoring to monitor the topic/num_unacked_messages_by_region metric on this new topic.
C. Enable dead lettering in your Pub/Sub pull subscription, and specify a new Pub/Sub topic as the dead letter topic. Use Cloud Monitoring to monitor the subscription/dead_letter_message_count metric on your pull subscription.
D. Create a snapshot of your Pub/Sub pull subscription. Use Cloud Monitoring to monitor the snapshot/num_messages metric on this snapshot.
Show Answer
Correct Answer: B
Explanation: Dataflow acknowledges Pub/Sub messages independently of your downstream business logic. If a DoFn throws after the message has been acknowledged, Pub/Sub dead-lettering will not capture it because dead-letter topics only apply to messages that are repeatedly not acknowledged. The recommended pattern is to catch exceptions in the DoFn, emit failed records via a side output, write them to a dedicated Pub/Sub topic, and monitor that topic for alerting.
Question 71
You are designing a real-time system for a ride hailing app that identifies areas with high demand for rides to effectively reroute available drivers to meet the demand. The system ingests data from multiple sources to Pub/Sub, processes the data, and stores the results for visualization and analysis in real-time dashboards. The data sources include driver location updates every 5 seconds and app-based booking events from riders. The data processing involves real-time aggregation of supply and demand data for the last 30 seconds, every 2 seconds, and storing the results in a low-latency system for visualization. What should you do?
A. Group the data by using a tumbling window in a Dataflow pipeline, and write the aggregated data to Memorystore.
B. Group the data by using a hopping window in a Dataflow pipeline, and write the aggregated data to Memorystore.
C. Group the data by using a session window in a Dataflow pipeline, and write the aggregated data to BigQuery.
D. Group the data by using a hopping window in a Dataflow pipeline, and write the aggregated data to BigQuery.
Show Answer
Correct Answer: B
Explanation: The requirement is to compute aggregates over the last 30 seconds every 2 seconds, which requires overlapping windows (a hopping/sliding window), not tumbling or session windows. For storing results used by real-time dashboards with low-latency access, Memorystore is more appropriate than BigQuery, which is an analytical data warehouse rather than a low-latency serving layer.
Question 72
You are designing a Dataflow pipeline for a batch processing job. You want to mitigate multiple zonal failures at job submission time. What should you do?
A. Submit duplicate pipelines in two different zones by using the --zone flag.
B. Set the pipeline staging location as a regional Cloud Storage bucket.
C. Specify a worker region by using the --region flag.
D. Create an Eventarc trigger to resubmit the job in case of zonal failure when submitting the job.
Show Answer
Correct Answer: C
Explanation: To mitigate multiple zonal failures at job submission time, submit the Dataflow job using a worker region rather than pinning it to a single zone. Using the --region flag allows Dataflow to place workers across available zones within the region and improves resilience to zonal failures. Using --zone ties execution to one zone, a regional Cloud Storage bucket only affects staging storage, and Eventarc resubmission is not the recommended mitigation for submission-time zonal failures.
Question 73
You have a BigQuery dataset named “customers”. All tables will be tagged by using a Data Catalog tag template named “gdpr”. The template contains one mandatory field, “has_sensitive_data”, with a boolean value. All employees must be able to do a simple search and find tables in the dataset that have either true or false in the “has_sensitive_data’ field. However, only the Human Resources (HR) group should be able to see the data inside the tables for which “has_sensitive data” is true. You give the all employees group the bigquery.metadataViewer and bigquery.connectionUser roles on the dataset. You want to minimize configuration overhead. What should you do next?
A. Create the “gdpr” tag template with private visibility. Assign the bigquery.dataViewer role to the HR group on the tables that contain sensitive data.
B. Create the “gdpr” tag template with private visibility. Assign the datacatalog.tagTemplateViewer role on this tag to the all employees group, and assign the bigquery.dataViewer role to the HR group on the tables that contain sensitive data.
C. Create the “gdpr” tag template with public visibility. Assign the bigquery.dataViewer role to the HR group on the tables that contain sensitive data.
D. Create the “gdpr” tag template with public visibility. Assign the datacatalog.tagTemplateViewer role on this tag to the all employees group, and assign the bigquery.dataViewer role to the HR group on the tables that contain sensitive data.
Show Answer
Correct Answer: C
Explanation: Use a public tag template so all users with metadata access can discover and search tagged BigQuery tables by the has_sensitive_data field. The existing bigquery.metadataViewer role allows employees to view table metadata, while only granting bigquery.dataViewer on sensitive tables to the HR group restricts access to table contents. Adding tagTemplateViewer is unnecessary for this requirement and would increase configuration overhead.
Question 74
You need to create a SQL pipeline. The pipeline runs an aggregate SQL transformation on a BigQuery table every two hours and appends the result to another existing BigQuery table. You need to configure the pipeline to retry if errors occur. You want the pipeline to send an email notification after three consecutive failures. What should you do?
A. Use the BigQueryUpsertTableOperator in Cloud Composer, set the retry parameter to three, and set the email_on_failure parameter to true.
B. Use the BigQueryInsertJobOperator in Cloud Composer, set the retry parameter to three, and set the email_on_failure parameter to true.
C. Create a BigQuery scheduled query to run the SQL transformation with schedule options that repeats every two hours, and enable email notifications.
D. Create a BigQuery scheduled query to run the SQL transformation with schedule options that repeats every two hours, and enable notification to Pub/Sub topic. Use Pub/Sub and Cloud Functions to send an email after three failed executions.
Show Answer
Correct Answer: D
Explanation: The key requirement is not merely retrying a single task, but sending an email only after three consecutive failed scheduled executions. Cloud Composer's retries and email_on_failure operate at the task-instance level, with email sent after the task exhausts its retries, not after three separate scheduled run failures. BigQuery scheduled queries provide the 2-hour schedule, and Pub/Sub plus Cloud Functions can track consecutive failed executions and send an email only after the third failure. Although the option does not explicitly describe retry behavior, it is the only one that satisfies the 'after three consecutive failures' notification requirement.
Question 75
You created a new version of a Dataflow streaming data ingestion pipeline that reads from Pub/Sub and writes to BigQuery. The previous version of the pipeline that runs in production uses a 5-minute window for processing. You need to deploy the new version of the pipeline without losing any data, creating inconsistencies, or increasing the processing latency by more than 10 minutes. What should you do?
A. Update the old pipeline with the new pipeline code.
B. Snapshot the old pipeline, stop the old pipeline, and then start the new pipeline from the snapshot.
C. Drain the old pipeline, then start the new pipeline.
D. Cancel the old pipeline, then start the new pipeline.
Show Answer
Correct Answer: A
Explanation: For deploying a new version of a Dataflow streaming pipeline without data loss, inconsistencies, or significant added latency, the preferred approach is an in-place update of the running streaming job when the changes are compatible. This preserves pipeline state and avoids draining (which can prematurely close windows) or stop-and-replace workflows that introduce transition complexity. Snapshots are primarily for stop-and-replace or recovery scenarios, not the first choice when an update is supported.
Sources:
https://cloud.google.com/blog/products/data-analytics/apache-kafka-for-gcp-users-connectors-for-pubsub-dataflow-and-bigquery
Question 76
You are creating a data model in BigQuery that will hold retail transaction data. Your two largest tables, sales_transaction_header and sales_transaction_line, have a tightly coupled immutable relationship. These tables are rarely modified after load and are frequently joined when queried. You need to model the sales_transaction_header and sales_transaction_line tables to improve the performance of data analytics queries. What should you do?
A. Create a sales_transaction table that holds the sales_transaction_header information as rows and the sales_transaction_line rows as nested and repeated fields.
B. Create a sales_transaction table that holds the sales_transaction_header and sales_transaction_line information as rows, duplicating the sales_transaction_header data for each line.
C. Create a sales_transaction table that stores the sales_transaction_header and sales_transaction_line data as a JSON data type.
D. Create separate sales_transaction_header and sales_transaction_line tables and, when querying, specify the sales_transaction_line first in the WHERE clause.
Show Answer
Correct Answer: A
Explanation: BigQuery best practices recommend using nested and repeated fields to model hierarchical one-to-many relationships that are tightly coupled, immutable, and frequently queried together. Storing transaction lines as a repeated nested field under each transaction header avoids expensive joins and improves query performance. Duplicating header data increases storage and maintenance overhead, JSON reduces query optimization benefits, and WHERE clause order does not affect join performance in BigQuery's optimizer.
Question 77
You are building a streaming Dataflow pipeline that ingests noise level data from hundreds of sensors placed near construction sites across a city. The sensors measure noise level every ten seconds, and send that data to the pipeline when levels reach above 70 dBA. You need to detect the average noise level from a sensor when data is received for a duration of more than 30 minutes, but the window ends when no data has been received for 15 minutes. What should you do?
A. Use session windows with a 15-minute gap duration.
B. Use session windows with a 30-minute gap duration.
C. Use hopping windows with a 15-minute window, and a thirty-minute period.
D. Use tumbling windows with a 15-minute window and a fifteen-minute .withAllowedLateness operator.
Show Answer
Correct Answer: A
Explanation: Session windows are designed for bursts of activity separated by inactivity. A 15-minute session gap means the window remains open as long as events continue arriving with less than 15 minutes between them, and it closes after 15 minutes of silence. The requirement to detect averages for sessions lasting more than 30 minutes is handled by evaluating/filtering session duration after windowing; it does not require a 30-minute session gap. Hopping and tumbling windows are fixed-time windows and do not end based on inactivity, and allowed lateness is unrelated to inactivity-based window closure.
Question 78
You maintain ETL pipelines. You notice that a streaming pipeline running on Dataflow is taking a long time to process incoming data, which causes output delays. You also noticed that the pipeline graph was automatically optimized by Dataflow and merged into one step. You want to identify where the potential bottleneck is occurring. What should you do?
A. Insert a Reshuffle operation after each processing step, and monitor the execution details in the Dataflow console.
B. Insert output sinks after each key processing step, and observe the writing throughput of each block.
C. Log debug information in each ParDo function, and analyze the logs at execution time.
D. Verify that the Dataflow service accounts have appropriate permissions to write the processed data to the output sinks.
Show Answer
Correct Answer: A
Explanation: Dataflow may fuse adjacent transforms into a single execution stage, making it difficult to identify which transform is the bottleneck. Inserting a Reshuffle breaks fusion, creating separate stages that can be inspected in the Dataflow execution graph and metrics. Monitoring these stages helps isolate where processing is slowing down. The other options either add unnecessary side effects, provide limited diagnostic value for fused stages, or are unrelated to performance bottlenecks.
Question 79
You are on the data governance team and are implementing security requirements. You need to encrypt all your data in BigQuery by using an encryption key managed by your team. You must implement a mechanism to generate and store encryption material only on your on-premises hardware security module (HSM). You want to rely on Google managed solutions. What should you do?
A. Create the encryption key in the on-premises HSM, and import it into a Cloud Key Management Service (Cloud KMS) key. Associate the created Cloud KMS key while creating the BigQuery resources.
B. Create the encryption key in the on-premises HSM and link it to a Cloud External Key Manager (Cloud EKM) key. Associate the created Cloud KMS key while creating the BigQuery resources.
C. Create the encryption key in the on-premises HSM, and import it into Cloud Key Management Service (Cloud HSM) key. Associate the created Cloud HSM key while creating the BigQuery resources.
D. Create the encryption key in the on-premises HSM. Create BigQuery resources and encrypt data while ingesting them into BigQuery.
Show Answer
Correct Answer: B
Explanation: BigQuery supports customer-managed encryption keys through Cloud KMS, including Cloud EKM. The requirement is that encryption material be generated and stored only on an on-premises HSM while relying on Google-managed integration. Cloud EKM is specifically designed to let Google Cloud services use externally managed keys that remain in your external key manager or on-premises HSM. Options A and C require importing key material into Google-managed KMS/HSM, which violates the requirement that the key material remain only on-premises. Option D is not an integrated CMEK solution for BigQuery.
$19
Get all 332 questions with detailed answers and explanations
Instant download HTML + PDF delivered the moment payment clears.
Secure Stripe checkout we never see or store your card details.
7-day refund if files are defective see our refund policy.