Databricks-Certified-Professional-Data-Engineer Databricks Certified Data Engineer Professional Exam Practice Test

Answer : D

When a Databricks job runs multiple tasks with dependencies, the tasks are executed in a dependency graph. If a task fails, the downstream tasks that depend on it are skipped and marked as Upstream failed. However, the failed task may have already committed some changes to the Lakehouse before the failure occurred, and those changes are not rolled back automatically. Therefore, the job run may result in a partial update of the Lakehouse. To avoid this, you can use thetransactional writesfeature of Delta Lake to ensure that the changes are only committed when the entire job run succeeds. Alternatively, you can use theRun ifcondition to configure tasks to run even when some or all of their dependencies have failed, allowing your job to recover from failures and continue running.Reference:

transactional writes: https://docs.databricks.com/delta/delta-intro.html#transactional-writes

Run if: https://docs.databricks.com/en/workflows/jobs/conditional-tasks.html

Answer : C

To deduplicate data against previously processed records as it is inserted into a Delta table, you can use the merge operation with an insert-only clause. This allows you to insert new records that do not match any existing records based on a unique key, while ignoring duplicate records that match existing records. For example, you can use the following syntax:

MERGE INTO target_table USING source_table ON target_table.unique_key = source_table.unique_key WHEN NOT MATCHED THEN INSERT *

This will insert only the records from the source table that have a unique key that is not present in the target table, and skip the records that have a matching key. This way, you can avoid inserting duplicate records into the Delta table.

https://docs.databricks.com/delta/delta-update.html#upsert-into-a-table-using-merge

https://docs.databricks.com/delta/delta-update.html#insert-only-merge

Answer : E

https://www.databricks.com/blog/2020/08/31/introducing-the-databricks-web-terminal.html

The code is using %sh to execute shell code on the driver node. This means that the code is not taking advantage of the worker nodes or Databricks optimized Spark. This is why the code is taking longer to execute. A better approach would be to use Databricks libraries and APIs to read and write data from Git and DBFS, and to leverage the parallelism and performance of Spark. For example, you can use theDatabricks Connectfeature to run your Python code on a remote Databricks cluster, or you can use theSpark Git Connectorto read data from Git repositories as Spark DataFrames.

Answer : D

To create an external or unmanaged Delta Lake table, you need to use the EXTERNAL keyword in the CREATE TABLE statement. This indicates that the table is not managed by the catalog and the data files are not deleted when the table is dropped. You also need to provide a LOCATION clause to specify the path where the data files are stored. For example:

CREATE EXTERNAL TABLE events ( date DATE, eventId STRING, eventType STRING, data STRING) USING DELTA LOCATION '/mnt/delta/events';

This creates an external Delta Lake table named events that references the data files in the '/mnt/delta/events' path. If you drop this table, the data files will remain intact and you can recreate the table with the same statement.

https://docs.databricks.com/delta/delta-batch.html#create-a-table

https://docs.databricks.com/delta/delta-batch.html#drop-a-table

Answer : E

Partitioning the data by the topic field allows the company to apply different access control policies and retention policies for different topics. For example, the company can use theTable Access Controlfeature to grant or revoke permissions to the registration topic based on user roles or groups. The company can also use theDELETEcommand to remove records from the registration topic that are older than 14 days, while keeping the records from other topics indefinitely. Partitioning by the topic field also improves the performance of queries that filter by the topic field, as they can skip reading irrelevant partitions.Reference:

Table Access Control: https://docs.databricks.com/security/access-control/table-acls/index.html

DELETE: https://docs.databricks.com/delta/delta-update.html#delete-from-a-table

Answer : A

Regex, or regular expressions, are a powerful way of matching patterns in text. They can be used to identify key areas of text when parsing Spark Driver log4j output, such as the log level, the timestamp, the thread name, the class name, the method name, and the message. Regex can be applied in various languages and frameworks, such as Scala, Python, Java, Spark SQL, and Databricks notebooks.Reference:

https://docs.databricks.com/notebooks/notebooks-use.html#use-regular-expressions

https://docs.databricks.com/spark/latest/spark-sql/udf-scala.html#using-regular-expressions-in-udfs

https://docs.databricks.com/spark/latest/sparkr/functions/regexp_extract.html

https://docs.databricks.com/spark/latest/sparkr/functions/regexp_replace.html

Answer : C

When a table is dropped in Delta Lake, the table is removed from the catalog and the data is deleted. This is because Delta Lake is a transactional storage layer that provides ACID guarantees. When a table is dropped, the transaction log is updated to reflect the deletion of the table and the data is deleted from the underlying storage.Reference:

https://docs.databricks.com/delta/quick-start.html#drop-a-table

https://docs.databricks.com/delta/delta-batch.html#drop-table