Professional Data Engineer Free Practice Questions — Page 12
Question 110
Your infrastructure team has set up an interconnect link between Google Cloud and the on-premises network. You are designing a high-throughput streaming pipeline to ingest data in streaming from an Apache Kafka cluster hosted on- premises. You want to store the data in BigQuery, with as minimal latency as possible. What should you do?
A. Setup a Kafka Connect bridge between Kafka and Pub/Sub. Use a Google-provided Dataflow template to read the data from Pub/Sub, and write the data to BigQuery.
B. Use a proxy host in the VPC in Google Cloud connecting to Kafka. Write a Dataflow pipeline, read data from the proxy host, and write the data to BigQuery.
C. Use Dataflow, write a pipeline that reads the data from Kafka, and writes the data to BigQuery.
D. Setup a Kafka Connect bridge between Kafka and Pub/Sub. Write a Dataflow pipeline, read the data from Pub/Sub, and write the data to BigQuery.
Show Answer
Correct Answer: C
Explanation: A direct Dataflow streaming pipeline that reads from the on-premises Kafka cluster over the existing Cloud Interconnect and writes to BigQuery minimizes latency by avoiding an extra Pub/Sub hop. Option B introduces an unnecessary proxy. Options A and D add Kafka Connect and Pub/Sub, increasing architecture complexity and adding an extra ingestion stage. Google Dataflow natively supports reading from Kafka and writing to BigQuery for high-throughput streaming use cases.
Question 111
You want to migrate your existing Teradata data warehouse to BigQuery. You want to move the historical data to BigQuery by using the most efficient method that requires the least amount of programming, but local storage space on your existing data warehouse is limited. What should you do?
A. Use BigQuery Data Transfer Service by using the Java Database Connectivity (JDBC) driver with FastExport connection.
B. Create a Teradata Parallel Transporter (TPT) export script to export the historical data, and import to BigQuery by using the bq command-line tool.
C. Use BigQuery Data Transfer Service with the Teradata Parallel Transporter (TPT) tbuild utility.
D. Create a script to export the historical data, and upload in batches to Cloud Storage. Set up a BigQuery Data Transfer Service instance from Cloud Storage to BigQuery.
Show Answer
Correct Answer: A
Explanation: The key constraints are minimal programming effort and limited local storage on the Teradata system. Google recommends using extraction via a JDBC driver with a FastExport connection when local storage for extracted files is constrained or when TPT cannot be used. This streams data rather than relying on staging large export files locally, making it the most appropriate option. Options B and D require custom scripting and staging files, while C is not the documented recommendation for this specific storage-constrained scenario.
Question 112
Different teams in your organization store customer and performance data in BigQuery. Each team needs to keep full control of their collected data, be able to query data within their projects, and be able to exchange their data with other teams. You need to implement an organization-wide solution, while minimizing operational tasks and costs. What should you do?
A. Ask each team to create authorized views of their data. Grant the biquery.jobUser role to each team.
B. Create a BigQuery scheduled query to replicate all customer data into team projects.
C. Ask each team to publish their data in Analytics Hub. Direct the other teams to subscribe to them.
D. Enable each team to create materialized views of the data they need to access in their projects.
Show Answer
Correct Answer: C
Explanation: Analytics Hub is designed for governed, organization-wide data sharing in BigQuery. Each team retains ownership and control of its datasets while publishing data products that other teams can subscribe to and query without duplicating data. This minimizes operational overhead and storage costs compared with replication, and is more scalable than managing many authorized views or materialized views.
Question 113
You are developing a model to identify the factors that lead to sales conversions for your customers. You have completed processing your data. You want to continue through the model development lifecycle. What should you do next?
A. Use your model to run predictions on fresh customer input data.
B. Monitor your model performance, and make any adjustments needed.
C. Delineate what data will be used for testing and what will be used for training the model.
D. Test and evaluate your model on your curated data to determine how well the model performs.
Show Answer
Correct Answer: C
Explanation: After data processing/preparation, the next step in the model development workflow is to split the curated dataset into training and testing (and often validation) sets. Only then do you train the model, evaluate it, deploy it for predictions, and later monitor its performance. Therefore, delineating the data for training and testing comes before evaluation, prediction, and monitoring.
Question 114
You currently use a SQL-based tool to visualize your data stored in BigQuery. The data visualizations require the use of outer joins and analytic functions. Visualizations must be based on data that is no less than 4 hours old. Business users are complaining that the visualizations are too slow to generate. You want to improve the performance of the visualization queries while minimizing the maintenance overhead of the data preparation pipeline. What should you do?
A. Create materialized views with the allow_non_incremental_definition option set to true for the visualization queries. Specify the max_staleness parameter to 4 hours and the enable_refresh parameter to true. Reference the materialized views in the data visualization tool.
B. Create views for the visualization queries. Reference the views in the data visualization tool.
C. Create a Cloud Function instance to export the visualization query results as parquet files to a Cloud Storage bucket. Use Cloud Scheduler to trigger the Cloud Function every 4 hours. Reference the parquet files in the data visualization tool.
D. Create materialized views for the visualization queries. Use the incremental updates capability of BigQuery materialized views to handle changed data automatically. Reference the materialized views in the data visualization tool.
Show Answer
Correct Answer: A
Explanation: The queries require outer joins and analytic functions, which are supported by non-incremental materialized views rather than standard incremental materialized views. Because the business accepts data up to 4 hours stale, configure a non-incremental materialized view with allow_non_incremental_definition=true, max_staleness=4 hours, and automatic refresh enabled. This provides precomputed results for much faster visualization queries with minimal operational overhead. Standard views do not improve performance, exporting to Cloud Storage adds maintenance, and incremental materialized views do not support these query patterns.
Question 115
You have a streaming pipeline that ingests data from Pub/Sub in production. You need to update this streaming pipeline with improved business logic. You need to ensure that the updated pipeline reprocesses the previous two days of delivered Pub/Sub messages. What should you do? (Choose two.)
A. Use the Pub/Sub subscription clear-retry-policy flag
B. Use Pub/Sub Snapshot capture two days before the deployment.
C. Create a new Pub/Sub subscription two days before the deployment.
D. Use the Pub/Sub subscription retain-acked-messages flag.
E. Use Pub/Sub Seek with a timestamp.
Show Answer
Correct Answer: D, E
Explanation: To replay the previous two days of delivered (acknowledged) Pub/Sub messages, the subscription must retain acknowledged messages, and then you use Seek to a timestamp from two days ago. A snapshot captures acknowledgment state at a point in time and by itself does not replay all messages published over the following two days; the required seek-to-snapshot operation is not offered as an option.
Question 116
You stream order data by using a Dataflow pipeline, and write the aggregated result to Memorystore. You provisioned a Memorystore for Redis instance with Basic Tier, 4 GB capacity, which is used by 40 clients for read-only access. You are expecting the number of read-only clients to increase significantly to a few hundred and you need to be able to support the demand. You want to ensure that read and write access availability is not impacted, and any changes you make can be deployed quickly. What should you do?
A. Create a new Memorystore for Redis instance with Standard Tier. Set capacity to 4 GB and read replica to No read replicas (high availability only). Delete the old instance.
B. Create a new Memorystore for Redis instance with Standard Tier. Set capacity to 5 GB and create multiple read replicas. Delete the old instance.
C. Create a new Memorystore for Memcached instance. Set a minimum of three nodes, and memory per node to 4 GB. Modify the Dataflow pipeline and all clients to use the Memcached instance. Delete the old instance.
D. Create multiple new Memorystore for Redis instances with Basic Tier (4 GB capacity). Modify the Dataflow pipeline and new clients to use all instances.
Show Answer
Correct Answer: B
Explanation: To support a large increase in read-only clients without affecting write performance, use Memorystore for Redis Standard Tier with read replicas. Read replicas distribute read traffic while the primary continues handling writes, and Standard Tier provides high availability. Basic Tier does not support read replicas, Memcached would require significant application changes and lacks Redis semantics, and sharding across multiple Basic instances increases operational complexity and code changes. The increased capacity option reflects the requirement for read replicas in the provided choices.
Question 117
Your organization has two Google Cloud projects, project A and project B. In project A, you have a Pub/Sub topic that receives data from confidential sources. Only the resources in project A should be able to access the data in that topic. You want to ensure that project B and any future project cannot access data in the project A topic. What should you do?
A. Add firewall rules in project A so only traffic from the VPC in project A is permitted.
B. Configure VPC Service Controls in the organization with a perimeter around project A.
C. Use Identity and Access Management conditions to ensure that only users and service accounts in project A. can access resources in project A.
D. Configure VPC Service Controls in the organization with a perimeter around the VPC of project A.
Show Answer
Correct Answer: B
Explanation: VPC Service Controls are designed to protect supported Google Cloud services such as Pub/Sub from data exfiltration by creating a security perimeter around projects. A perimeter around project A ensures that requests from project B or future projects outside the perimeter cannot access the protected Pub/Sub data, even if IAM permissions are mistakenly granted. Firewall rules do not apply to the managed Pub/Sub API, and VPC Service Controls are configured around projects, not VPC networks. IAM alone controls identity-based access but does not provide the same organization-wide data exfiltration protection or future-proof isolation described in the question.
Question 118
Your organization stores customer data in an on-premises Apache Hadoop cluster in Apache Parquet format. Data is processed on a daily basis by Apache Spark jobs that run on the cluster. You are migrating the Spark jobs and Parquet data to Google Cloud. BigQuery will be used on future transformation pipelines so you need to ensure that your data is available in BigQuery. You want to use managed services, while minimizing ETL data processing changes and overhead costs. What should you do?
A. Migrate your data to Cloud Storage and migrate the metadata to Dataproc Metastore (DPMS). Refactor Spark pipelines to write and read data on Cloud Storage, and run them on Dataproc Serverless.
B. Migrate your data to Cloud Storage and register the bucket as a Dataplex asset. Refactor Spark pipelines to write and read data on Cloud Storage, and run them on Dataproc Serverless.
C. Migrate your data to BigQuery. Refactor Spark pipelines to write and read data on BigQuery, and run them on Dataproc Serverless.
D. Migrate your data to BigLake. Refactor Spark pipelines to write and read data on Cloud Storage, and run them on Dataproc on Compute Engine.
Show Answer
Correct Answer: B
Explanation: Keeping Parquet data in Cloud Storage minimizes Spark ETL changes because Spark continues to read/write Parquet. Running on Dataproc Serverless satisfies the managed-service requirement. Registering the Cloud Storage bucket as a Dataplex asset makes the data discoverable and available to BigQuery through integrated metadata management and external table support, aligning with the requirement that the data be available in BigQuery without migrating all data into BigQuery storage. Dataproc Metastore is for Hive-compatible metadata and does not by itself make the data available in BigQuery. Migrating all data into BigQuery would require greater pipeline refactoring and potentially higher storage/processing costs, while Dataproc on Compute Engine is less managed than Serverless.
Sources:
https://cloud.google.com/bigquery/docs/migration/pipelines
Question 119
A web server sends click events to a Pub/Sub topic as messages. The web server includes an eventTimestamp attribute in the messages, which is the time when the click occurred. You have a Dataflow streaming job that reads from this Pub/Sub topic through a subscription, applies some transformations, and writes the result to another Pub/Sub topic for use by the advertising department. The advertising department needs to receive each message within 30 seconds of the corresponding click occurrence, but they report receiving the messages late. Your Dataflow job's system lag is about 5 seconds, and the data freshness is about 40 seconds. Inspecting a few messages show no more than 1 second lag between their eventTimestamp and publishTime. What is the problem and what should you do?
A. The advertising department is causing delays when consuming the messages. Work with the advertising department to fix this.
B. Messages in your Dataflow job are taking more than 30 seconds to process. Optimize your job or increase the number of workers to fix this.
G. Messages in your Dataflow job are processed in less than 30 seconds, but your job cannot keep up with the backlog in the Pub/Sub subscription. Optimize your job or increase the number of workers to fix this.
D. The web server is not pushing messages fast enough to Pub/Sub. Work with the web server team to fix this.
Show Answer
Correct Answer: G
Explanation: The key metrics distinguish processing time from backlog. System lag of about 5 seconds indicates the pipeline processes elements quickly once they are being handled. Data freshness of about 40 seconds means the output reflects data that is around 40 seconds old, indicating the pipeline is falling behind due to input backlog. The eventTimestamp-to-publishTime delay is only about 1 second, so the web server is publishing promptly. The delay is not attributable to downstream consumers because the messages are already late when produced by the Dataflow pipeline. The appropriate action is to optimize the pipeline or increase worker capacity so it can keep up with the incoming message rate.
$19
Get all 332 questions with detailed answers and explanations
Instant download HTML + PDF delivered the moment payment clears.
Secure Stripe checkout we never see or store your card details.
7-day refund if files are defective see our refund policy.