Databricks-Machine-Learning-Associate Exam Practice Test Instant Access

Question 1

Which statement describes a Spark ML transformer?

AA transformer is an algorithm which can transform one DataFrame into another DataFrame

BA transformer is a hyperparameter grid that can be used to train a model

CA transformer chains multiple algorithms together to transform an ML workflow

DA transformer is a learning algorithm that can use a DataFrame to train a model

Answer : A

In Spark ML, a transformer is an algorithm that can transform one DataFrame into another DataFrame. It takes a DataFrame as input and produces a new DataFrame as output. This transformation can involve adding new columns, modifying existing ones, or applying feature transformations. Examples of transformers in Spark MLlib include feature transformers like StringIndexer, VectorAssembler, and StandardScaler.

Databricks documentation on transformers: Transformers in Spark ML

Question 2

A machine learning engineer is using the following code block to scale the inference of a single-node model on a Spark DataFrame with one million records:

Assuming the default Spark configuration is in place, which of the following is a benefit of using an Iterator?

AThe data will be limited to a single executor preventing the model from being loaded multiple times

BThe model will be limited to a single executor preventing the data from being distributed

CThe model only needs to be loaded once per executor rather than once per batch during the inference process

DThe data will be distributed across multiple executors during the inference process

Answer : C

Using an iterator in the pandas_udf ensures that the model only needs to be loaded once per executor rather than once per batch. This approach reduces the overhead associated with repeatedly loading the model during the inference process, leading to more efficient and faster predictions. The data will be distributed across multiple executors, but each executor will load the model only once, optimizing the inference process.

Databricks documentation on pandas UDFs: Pandas UDFs

Question 3

Which of the following tools can be used to distribute large-scale feature engineering without the use of a UDF or pandas Function API for machine learning pipelines?

AKeras

BScikit-learn

CPyTorch

DSpark ML

Answer : D

Spark MLlib is a machine learning library within Apache Spark that provides scalable and distributed machine learning algorithms. It is designed to work with Spark DataFrames and leverages Spark's distributed computing capabilities to perform large-scale feature engineering and model training without the need for user-defined functions (UDFs) or the pandas Function API. Spark MLlib provides built-in transformations and algorithms that can be applied directly to large datasets.

Databricks documentation on Spark MLlib: Spark MLlib

Question 4

A data scientist has developed a linear regression model using Spark ML and computed the predictions in a Spark DataFrame preds_df with the following schema:

prediction DOUBLE

actual DOUBLE

Which of the following code blocks can be used to compute the root mean-squared-error of the model according to the data in preds_df and assign it to the rmse variable?

AOption A

BOption B

COption C

DOption D

Answer : C

To compute the root mean-squared-error (RMSE) of a linear regression model using Spark ML, the RegressionEvaluator class is used. The RegressionEvaluator is specifically designed for regression tasks and can calculate various metrics, including RMSE, based on the columns containing predictions and actual values.

The correct code block to compute RMSE from the preds_df DataFrame is:

regression_evaluator = RegressionEvaluator( predictionCol='prediction', labelCol='actual', metricName='rmse' ) rmse = regression_evaluator.evaluate(preds_df)

This code creates an instance of RegressionEvaluator, specifying the prediction and label columns, as well as the metric to be computed ('rmse'). It then evaluates the predictions in preds_df and assigns the resulting RMSE value to the rmse variable.

Options A and B incorrectly use BinaryClassificationEvaluator, which is not suitable for regression tasks. Option D also incorrectly uses BinaryClassificationEvaluator.

PySpark ML Documentation

Question 5

A data scientist has written a feature engineering notebook that utilizes the pandas library. As the size of the data processed by the notebook increases, the notebook's runtime is drastically increasing, but it is processing slowly as the size of the data included in the process increases.

Which of the following tools can the data scientist use to spend the least amount of time refactoring their notebook to scale with big data?

APySpark DataFrame API

Bpandas API on Spark

CSpark SQL

DFeature Store

Answer : B

The pandas API on Spark provides a way to scale pandas operations to big data while minimizing the need for refactoring existing pandas code. It allows users to run pandas operations on Spark DataFrames, leveraging Spark's distributed computing capabilities to handle large datasets more efficiently. This approach requires minimal changes to the existing code, making it a convenient option for scaling pandas-based feature engineering notebooks.

Databricks documentation on pandas API on Spark: pandas API on Spark

Question 6

Which of the following statements describes a Spark ML estimator?

AAn estimator is a hyperparameter arid that can be used to train a model

BAn estimator chains multiple alqorithms toqether to specify an ML workflow

CAn estimator is a trained ML model which turns a DataFrame with features into a DataFrame with predictions

DAn estimator is an alqorithm which can be fit on a DataFrame to produce a Transformer

EAn estimator is an evaluation tool to assess to the quality of a model

Answer : D

In the context of Spark MLlib, an estimator refers to an algorithm which can be 'fit' on a DataFrame to produce a model (referred to as a Transformer), which can then be used to transform one DataFrame into another, typically adding predictions or model scores. This is a fundamental concept in machine learning pipelines in Spark, where the workflow includes fitting estimators to data to produce transformers.

Reference

Spark MLlib Documentation: https://spark.apache.org/docs/latest/ml-pipeline.html#estimators

Question 7

A machine learning engineer wants to parallelize the training of group-specific models using the Pandas Function API. They have developed the train_model function, and they want to apply it to each group of DataFrame df.

They have written the following incomplete code block:

Which of the following pieces of code can be used to fill in the above blank to complete the task?

AapplyInPandas

BmapInPandas

Cpredict

Dtrain_model

EgroupedApplyIn

Answer : B

The function mapInPandas in the PySpark DataFrame API allows for applying a function to each partition of the DataFrame. When working with grouped data, groupby followed by applyInPandas is the correct approach to apply a function to each group as a separate Pandas DataFrame. However, if the function should apply across each partition of the grouped data rather than on each individual group, mapInPandas would be utilized. Since the code snippet indicates the use of groupby, the intent seems to be to apply train_model on each group specifically, which aligns with applyInPandas. Thus, applyInPandas is a better fit to ensure that each group generated by groupby is processed through the train_model function, preserving the partitioning and grouping integrity.

Reference

PySpark Documentation on applying functions to grouped data: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.GroupedData.applyInPandas.html

Databricks Certified Machine Learning Associate Exam Practice Test