Databricks-Certified-Professional-Data-Engineer Exam Practice Test Instant Access

Question 1

A junior data engineer is working to implement logic for a Lakehouse table named silver_device_recordings. The source data contains 100 unique fields in a highly nested JSON structure.

The silver_device_recordings table will be used downstream for highly selective joins on a number of fields, and will also be leveraged by the machine learning team to filter on a handful of relevant fields, in total, 15 fields have been identified that will often be used for filter and join logic.

The data engineer is trying to determine the best approach for dealing with these nested fields before declaring the table schema.

Which of the following accurately presents information about Delta Lake and Databricks that may Impact their decision-making process?

ABecause Delta Lake uses Parquet for data storage, Dremel encoding information for nesting can be directly referenced by the Delta transaction log.

BTungsten encoding used by Databricks is optimized for storing string data: newly-added native support for querying JSON strings means that string types are always most efficient.

CSchema inference and evolution on Databricks ensure that inferred types will always accurately match the data types used by downstream systems.

DBy default Delta Lake collects statistics on the first 32 columns in a table; these statistics are leveraged for data skipping when executing selective queries.

Answer : D

Delta Lake, built on top of Parquet, enhances query performance through data skipping, which is based on the statistics collected for each file in a table. For tables with a large number of columns, Delta Lake by default collects and stores statistics only for the first 32 columns. These statistics include min/max values and null counts, which are used to optimize query execution by skipping irrelevant data files. When dealing with highly nested JSON structures, understanding this behavior is crucial for schema design, especially when determining which fields should be flattened or prioritized in the table structure to leverage data skipping efficiently for performance optimization. Reference: Databricks documentation on Delta Lake optimization techniques, including data skipping and statistics collection (https://docs.databricks.com/delta/optimizations/index.html).

Question 2

A Data engineer wants to run unit's tests using common Python testing frameworks on python functions defined across several Databricks notebooks currently used in production.

How can the data engineer run unit tests against function that work with data in production?

ARun unit tests against non-production data that closely mirrors production

BDefine and unit test functions using Files in Repos

CDefine units test and functions within the same notebook

DDefine and import unit test functions from a separate Databricks notebook

Answer : A

The best practice for running unit tests on functions that interact with data is to use a dataset that closely mirrors the production data. This approach allows data engineers to validate the logic of their functions without the risk of affecting the actual production data. It's important to have a representative sample of production data to catch edge cases and ensure the functions will work correctly when used in a production environment.

Databricks Documentation on Testing: Testing and Validation of Data and Notebooks

Question 3

Which is a key benefit of an end-to-end test?

AIt closely simulates real world usage of your application.

BIt pinpoint errors in the building blocks of your application.

CIt provides testing coverage for all code paths and branches.

DIt makes it easier to automate your test suite

Answer : A

End-to-end testing is a methodology used to test whether the flow of an application, from start to finish, behaves as expected. The key benefit of an end-to-end test is that it closely simulates real-world, user behavior, ensuring that the system as a whole operates correctly.

Software Testing: End-to-End Testing

Question 4

A data engineer is testing a collection of mathematical functions, one of which calculates the area under a curve as described by another function.

Which kind of the test does the above line exemplify?

AIntegration

BUnit

CManual

Dfunctional

Answer : B

A unit test is designed to verify the correctness of a small, isolated piece of code, typically a single function. Testing a mathematical function that calculates the area under a curve is an example of a unit test because it is testing a specific, individual function to ensure it operates as expected.

Software Testing Fundamentals: Unit Testing

Question 5

What is a method of installing a Python package scoped at the notebook level to all nodes in the currently active cluster?

AUse &Pip install in a notebook cell

BRun source env/bin/activate in a notebook setup script

CInstall libraries from PyPi using the cluster UI

DUse &sh install in a notebook cell

Answer : C

Installing a Python package scoped at the notebook level to all nodes in the currently active cluster in Databricks can be achieved by using the Libraries tab in the cluster UI. This interface allows you to install libraries across all nodes in the cluster. While the %pip command in a notebook cell would only affect the driver node, using the cluster UI ensures that the package is installed on all nodes.

Databricks Documentation on Libraries: Libraries

Question 6

A Databricks job has been configured with 3 tasks, each of which is a Databricks notebook. Task A does not depend on other tasks. Tasks B and C run in parallel, with each having a serial dependency on Task A.

If task A fails during a scheduled run, which statement describes the results of this run?

ABecause all tasks are managed as a dependency graph, no changes will be committed to the Lakehouse until all tasks have successfully been completed.

BTasks B and C will attempt to run as configured; any changes made in task A will be rolled back due to task failure.

CUnless all tasks complete successfully, no changes will be committed to the Lakehouse; because task A failed, all commits will be rolled back automatically.

DTasks B and C will be skipped; some logic expressed in task A may have been committed before task failure.

ETasks B and C will be skipped; task A will not commit any changes because of stage failure.

Answer : D

When a Databricks job runs multiple tasks with dependencies, the tasks are executed in a dependency graph. If a task fails, the downstream tasks that depend on it are skipped and marked as Upstream failed. However, the failed task may have already committed some changes to the Lakehouse before the failure occurred, and those changes are not rolled back automatically. Therefore, the job run may result in a partial update of the Lakehouse. To avoid this, you can use thetransactional writesfeature of Delta Lake to ensure that the changes are only committed when the entire job run succeeds. Alternatively, you can use theRun ifcondition to configure tasks to run even when some or all of their dependencies have failed, allowing your job to recover from failures and continue running.Reference:

transactional writes: https://docs.databricks.com/delta/delta-intro.html#transactional-writes

Run if: https://docs.databricks.com/en/workflows/jobs/conditional-tasks.html

Question 7

A data engineer is configuring a pipeline that will potentially see late-arriving, duplicate records.

In addition to de-duplicating records within the batch, which of the following approaches allows the data engineer to deduplicate data against previously processed records as it is inserted into a Delta table?

ASet the configuration delta.deduplicate = true.

BVACUUM the Delta table after each batch completes.

CPerform an insert-only merge with a matching condition on a unique key.

DPerform a full outer join on a unique key and overwrite existing data.

ERely on Delta Lake schema enforcement to prevent duplicate records.

Answer : C

To deduplicate data against previously processed records as it is inserted into a Delta table, you can use the merge operation with an insert-only clause. This allows you to insert new records that do not match any existing records based on a unique key, while ignoring duplicate records that match existing records. For example, you can use the following syntax:

MERGE INTO target_table USING source_table ON target_table.unique_key = source_table.unique_key WHEN NOT MATCHED THEN INSERT *

This will insert only the records from the source table that have a unique key that is not present in the target table, and skip the records that have a matching key. This way, you can avoid inserting duplicate records into the Delta table.

https://docs.databricks.com/delta/delta-update.html#upsert-into-a-table-using-merge

https://docs.databricks.com/delta/delta-update.html#insert-only-merge

Databricks-Certified-Professional-Data-Engineer Databricks Certified Data Engineer Professional Exam Practice Test