A company runs an Apache Spark application every night in an Amazon EMR cluster. The company uses Amazon EC2 instances to supply compute capacity for the EMR cluster. The company deployed the Spark application in cluster mode.
An error occurs in the Spark application. A log for the error is stored in the application's Spark driver standard error logs. A data engineer needs to investigate the error.
Where can the data engineer find this error log?
A. The engineer can connect to the web UI on the live cluster to see the YARN ResourceManager logs.
B. The engineer can connect to the persistent application UI to see the first YARN container log in the Spark UI.
C. The engineer can connect to the Amazon EMR console to see the Amazon EMR step logs that are archived in Amazon S3.
D. The engineer can connect to the primary node of the cluster by using SSH to see the Spark history server logs.
Show Answer
Correct Answer: B
Explanation: In Spark cluster mode on Amazon EMR, the Spark driver runs inside the first YARN container (the ApplicationMaster), so the driver's stdout/stderr are part of the YARN container logs. The persistent application UI exposes these YARN container logs, including the first container log where the driver stderr is stored.
Question 72
A data engineer is configuring an AWS Glue Apache Spark extract, transform, and load (ETL) job. The job contains a sort-merge join of two large and equally sized DataFrames.
The job is failing with the following error: No space left on device.
Which solution will resolve the error?
A. Use the AWS Glue Spark shuffle manager.
B. Deploy are Amazon Elastic Block Store (Amazon EBS) volume for the job to use.
C. Convert the sort-merge join in the job to be a broadcast join.
D. Convert the DataFrames to DynamicFrames, and perform a DynamicFrame join in the job.
Show Answer
Correct Answer: A
Explanation: A sort-merge join between two large, similarly sized DataFrames generates substantial shuffle data. In AWS Glue, using the AWS Glue Spark shuffle manager (S3-backed shuffle) offloads shuffle data from local disk, preventing local storage exhaustion that causes the 'No space left on device' error. Broadcast joins are inappropriate because neither DataFrame is small, EBS volumes cannot be attached to AWS Glue jobs for this purpose, and DynamicFrame joins still rely on Spark execution and do not eliminate shuffle disk usage.
Question 73
A company has a data pipeline that uses an Amazon RDS instance, AWS Glue jobs, and an Amazon S3 bucket. The RDS instance and AWS Glue jobs run in a private subnet of a VPC and in the same security group.
A user made a change to the security group that prevents the AWS Glue jobs from connecting to the RDS instance. After the change, the security group contains a single rule that allows inbound SSH traffic from a specific IP address.
The company must resolve the connectivity issue.
Which solution will meet this requirement?
A. Add an inbound rule that allows all TCP traffic on all TCP ports. Set the security group as the source.
B. Add an inbound rule that allows all TCP traffic on all UDP ports. Set the private IP address of the RDS instance as the source.
C. Add an inbound rule that allows all TCP traffic on all TCP ports. Set the DNS name of the RDS instance as the source.
D. Replace the source of the existing SSH rule with the private IP address of the RDS instance. Create an outbound rule with the same source, destination, and protocol as the inbound SSH rule.
Show Answer
Correct Answer: A
Explanation: When resources that share the same security group need to communicate, the security group must include an inbound rule that references itself as the source. AWS Glue jobs connecting to an Amazon RDS instance require the RDS security group to allow inbound traffic from the Glue jobs' security group. Since both use the same security group, a self-referencing inbound rule allowing TCP traffic restores connectivity. Security groups cannot use a DNS name as a source, and using the RDS private IP as the source is incorrect because the source of connections is the Glue job ENIs, not the database.
Question 74
A data engineer notices slow query performance on a highly partitioned table that is in Amazon Athena. The table contains daily data for the previous 5 years, partitioned by date.
The data engineer wants to improve query performance and to automate partition management.
Which solution will meet these requirements?
A. Use an AWS Lambda function that runs daily. Configure the function to manually create new partitions in AWS Glue for each day’s data.
B. Use partition projection in Athena. Configure the table properties by using a date range from 5 years ago to the present.
C. Reduce the number of partitions by changing the partitioning schema from daily to monthly granularity.
D. Increase the processing capacity of Athena queries by allocating more compute resources.
Show Answer
Correct Answer: B
Explanation: Partition projection in Amazon Athena eliminates the need to maintain partition metadata in the AWS Glue Data Catalog by calculating partitions dynamically from table properties. This reduces partition planning overhead for highly partitioned tables and automates partition management. A requires ongoing operational management, C reduces partition count but changes data layout and granularity without addressing automation as effectively, and D is not applicable because Athena is serverless and does not support user-allocated compute resources.
Question 75
A company uses AWS Glue Apache Spark jobs to handle extract, transform, and load (ETL) workloads. The company has enabled logging and monitoring for all AWS Glue jobs.
One of the AWS Glue jobs begins to fail. A data engineer investigates the error and wants to examine metrics for all individual stages within the job.
How can the data engineer access the stage metrics?
A. Examine the AWS Glue job and stage details in the Spark UI.
B. Examine the AWS Glue job and stage metrics in Amazon CloudWatch.
C. Examine the AWS Glue job and stage logs in AWS CloudTrail logs.
D. Examine the AWS Glue job and stage details by using the run insights feature on the job.
Show Answer
Correct Answer: A
Explanation: AWS Glue Spark jobs expose the Spark UI, which provides detailed stage-level metrics such as stages, tasks, execution time, shuffle activity, and executor information. CloudWatch provides job-level monitoring rather than detailed Spark stage metrics, CloudTrail is for API auditing, and Run Insights summarizes job execution and issues but does not replace the Spark UI for inspecting all individual stage metrics.
Question 76
A company uses AWS Glue jobs to implement several data pipelines. The pipelines are critical to the company.
The company needs to implement a monitoring mechanism that will alert stakeholders if the pipelines fail.
Which solution will meet these requirements with the LEAST operational overhead?
A. Create an Amazon EventBridge rule to match AWS Glue job failure events. Configure the rule to target an AWS Lambda function to process events. Configure the function to send notifications to an Amazon Simple Notification Service (Amazon SNS) topic.
B. Configure an Amazon CloudWatch Logs log group for the AWS Glue jobs. Create an Amazon EventBridge rule to match new log creation events in the log group. Configure the rule to target an AWS Lambda function that reads the logs and sends notifications to an Amazon Simple Notification Service (Amazon SNS) topic if AWS Glue job failure logs are present.
C. Create an Amazon EventBridge rule to match AWS Glue job failure events. Define an Amazon CloudWatch metric based on the EventBridge rule. Set up a CloudWatch alarm based on the metric to send notifications to an Amazon Simple Notification Service (Amazon SNS) topic.
D. Configure an Amazon CloudWatch Logs log group for the AWS Glue jobs. Create an Amazon EventBridge rule to match new log creation events in the log group. Configure the rule to send notifications to an Amazon Simple Notification Service (Amazon SNS) topic.
Show Answer
Correct Answer: C
Explanation: AWS Glue emits job state change events to Amazon EventBridge, including failure events. Using an EventBridge rule to create a CloudWatch metric and a CloudWatch alarm that notifies an SNS topic is a fully managed, serverless approach with less operational overhead than maintaining a Lambda function or parsing CloudWatch Logs.
Question 77
A data engineer is troubleshooting an AWS Glue workflow that occasionally fails. The engineer determines that the failures are a result of data quality issues. A business reporting team needs to receive an email notification any time the workflow fails in the future.
Which solution will meet this requirement?
A. Create an Amazon Simple Notification Service (Amazon SNS) FIFO topic. Subscribe the team’s email account to the SNS topic. Create an AWS Lambda function that initiates when the AWS Glue job state changes to FAILED. Set the SNS topic as the target.
B. Create an Amazon Simple Notification Service (Amazon SNS) standard topic. Subscribe the team’s email account to the SNS topic. Create an Amazon EventBridge rule that triggers when the AWS Glue job state changes to FAILED. Set the SNS topic as the target.
C. Create an Amazon Simple Queue Service (Amazon SQS) FIFO queue. Subscribe the team’s email account to the SQS queue. Create an AWS Config rule that triggers when the AWS Glue job state changes to FAILED. Set the SQS queue as the target.
D. Create an Amazon Simple Queue Service (Amazon SQS) standard queue. Subscribe the team’s email account to the SQS queue. Create an Amazon EventBridge rule that triggers when the AWS Glue job state changes to FAILESet the SQS queue as the target.
Show Answer
Correct Answer: B
Explanation: Amazon EventBridge natively receives AWS Glue job state change events, including FAILED. An EventBridge rule can target an Amazon SNS standard topic, and the reporting team's email can subscribe to that topic for notifications. FIFO ordering is unnecessary for failure alerts, Lambda is not required because EventBridge can directly invoke SNS, AWS Config does not monitor Glue job state changes, and Amazon SQS does not support email subscriptions.
Question 78
A company uses Amazon S3 and AWS Glue Data Catalog to manage a data lake that contains contact information for customers. The company uses PySpark and AWS Glue jobs with a DynamicFrame to run a workflow that processes data within the data lake.
A data engineer notices that the workflow is generating errors as a result of how customer postal codes are stored in the data lake. Some postal codes include unnecessary numbers or invalid characters.
The data engineer needs a solution to address the errors and correct the postal codes in the data lake.
A. Create a schema definition for PySpark that matches the format the processing workflow requires for postal codes. Pass the schema to the DynamicFrame during processing.
B. Use AWS Glue workflow properties to allow job state sharing. Configure the AWS Glue jobs to read values from the postal code column by using the properties from a previously successful run of the jobs.
C. Configure the column.push_down_predicate setting and the catalogPartitionPredicate settings for the postal code column in the DynamicFrame.
D. Set the DynamicFrame additional_options parameter ‘useS3ListImplementation’ to True.
Show Answer
Correct Answer: A
Explanation: Among the provided options, defining and applying an explicit schema is the only choice related to how AWS Glue/PySpark interprets the postal code field during processing. The other options concern workflow state sharing, partition pruning, or S3 listing behavior and do not address data format issues in postal codes. While a schema alone does not remove invalid characters, it is the closest and only relevant option presented.
Question 79
A data engineer needs to run a data transformation job whenever a user adds a file to an Amazon S3 bucket. The job will run for less than 1 minute. The job must send the output through an email message to the data engineer. The data engineer expects users to add one file every hour of the day.
Which solution will meet these requirements in the MOST operationally efficient way?
A. Create a small Amazon EC2 instance that polls the S3 bucket for new files. Run transformation code on a schedule to generate the output. Use operating system commands to send email messages.
B. Run an Amazon Elastic Container Service (Amazon ECS) task to poll the S3 bucket for new files. Run transformation code on a schedule to generate the output. Use operating system commands to send email messages.
C. Create an AWS Lambda function to transform the data. Use Amazon S3 Event Notifications to invoke the Lambda function when a new object is created. Publish the output to an Amazon Simple Notification Service (Amazon SNS) topic. Subscribe the data engineer’s email account to the topic.
D. Deploy an Amazon EMR cluster. Use EMR File System (EMRFS) to access the files in the S3 bucket. Run transformation code on a schedule to generate the output to a second S3 bucket. Create an Amazon Simple Notification Service (Amazon SNS) topic. Configure Amazon S3 Event Notifications to notify the topic when a new object is created.
Show Answer
Correct Answer: C
Explanation: AWS Lambda triggered by Amazon S3 Event Notifications is the most operationally efficient event-driven solution for a short job (<1 minute). After transforming the data, the Lambda function can publish the result to an Amazon SNS topic, which delivers the output by email to the subscribed data engineer. This avoids polling, servers, containers, or EMR clusters and scales automatically.
Question 80
A data engineer is building a data pipeline. A large data file is uploaded to an Amazon S3 bucket once each day at unpredictable times. An AWS Glue workflow uses hundreds of workers to process the file and load the data into Amazon Redshift. The company wants to process the file as quickly as possible.
Which solution will meet these requirements?
A. Create an on-demand AWS Glue trigger to start the workflow. Create an AWS Lambda function that runs every 15 minutes to check the S3 bucket for the daily file. Configure the function to start the AWS Glue workflow if the file is present.
B. Create an event-based AWS Glue trigger to start the workflow. Configure Amazon S3 to log events to AWS CloudTrail. Create a rule in Amazon EventBridge to forward PutObject events to the AWS Glue trigger.
C. Create a scheduled AWS Glue trigger to start the workflow. Create a cron job that runs the AWS Glue job every 15 minutes. Set up the AWS Glue job to check the S3 bucket for the daily file. Configure the job to stop if the file is not present.
D. Create an on-demand AWS Glue trigger to start the workflow. Create an AWS Database Migration Service (AWS DMS) migration task. Set the DMS source as the S3 bucket. Set the target endpoint as the AWS Glue workflow.
Show Answer
Correct Answer: B
Explanation: An event-driven architecture is the fastest way to start processing when the S3 object arrives at an unpredictable time. Configure S3 object creation events (via CloudTrail and EventBridge as described in the option) to invoke an event-based AWS Glue trigger immediately, avoiding polling delays and unnecessary checks. Options A and C rely on 15-minute polling/scheduling, introducing latency, and D misuses AWS DMS, which cannot target a Glue workflow.
$19
Get all 310 questions with detailed answers and explanations
Instant download HTML + PDF delivered the moment payment clears.
Secure Stripe checkout we never see or store your card details.
7-day refund if files are defective see our refund policy.