Associate-Developer-Apache-Spark Dumps Updated Jun 27, 2022 Practice Test and 179 unique questions [Q35-Q53] : Actual Test Materials : http://blog.actualtests4sure.com

NEW QUESTION 35
Which of the following describes the conversion of a computational query into an execution plan in Spark?

Spark uses the catalog to resolve the optimized logical plan.

The catalog assigns specific resources to the optimized memory plan.

The executed physical plan depends on a cost optimization from a previous stage.

Depending on whether DataFrame API or SQL API are used, the physical plan may differ.

The catalog assigns specific resources to the physical plan.

NEW QUESTION 36
The code block shown below should return a DataFrame with all columns of DataFrame transactionsDf, but only maximum 2 rows in which column productId has at least the value 2. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__(__2__).__3__

1. where
2. “productId” > 2
3. max(2)

1. where
2. transactionsDf[productId] >= 2
3. limit(2)

1. filter
2. productId > 2
3. max(2)

1. filter
2. col(“productId”) >= 2
3. limit(2)

1. where
2. productId >= 2
3. limit(2)

NEW QUESTION 37
Which of the following statements about the differences between actions and transformations is correct?

Actions are evaluated lazily, while transformations are not evaluated lazily.

Actions generate RDDs, while transformations do not.

Actions do not send results to the driver, while transformations do.

Actions can be queued for delayed execution, while transformations can only be processed immediately.

Actions can trigger Adaptive Query Execution, while transformation cannot.

NEW QUESTION 38
Which of the following code blocks reads in the parquet file stored at location filePath, given that all columns in the parquet file contain only whole numbers and are stored in the most appropriate format for this kind of data?

1.spark.read.schema(
2. StructType(
3. StructField(“transactionId”, IntegerType(), True),
4. StructField(“predError”, IntegerType(), True)
5. )).load(filePath)

1.spark.read.schema([
2. StructField(“transactionId”, NumberType(), True),
3. StructField(“predError”, IntegerType(), True)
4. ]).load(filePath)

1.spark.read.schema(
2. StructType([
3. StructField(“transactionId”, StringType(), True),
4. StructField(“predError”, IntegerType(), True)]
5. )).parquet(filePath)

1.spark.read.schema(
2. StructType([
3. StructField(“transactionId”, IntegerType(), True),
4. StructField(“predError”, IntegerType(), True)]
5. )).format(“parquet”).load(filePath)

1.spark.read.schema([
2. StructField(“transactionId”, IntegerType(), True),
3. StructField(“predError”, IntegerType(), True)
4. ]).load(filePath, format=”parquet”)

NEW QUESTION 39
Which of the following code blocks uses a schema fileSchema to read a parquet file at location filePath into a DataFrame?

spark.read.schema(fileSchema).format(“parquet”).load(filePath)

spark.read.schema(“fileSchema”).format(“parquet”).load(filePath)

spark.read().schema(fileSchema).parquet(filePath)

spark.read().schema(fileSchema).format(parquet).load(filePath)

spark.read.schema(fileSchema).open(filePath)

NEW QUESTION 40
Which of the following code blocks immediately removes the previously cached DataFrame transactionsDf from memory and disk?

array_remove(transactionsDf, “*”)

transactionsDf.unpersist()
(Correct)

del transactionsDf

transactionsDf.clearCache()

transactionsDf.persist()

NEW QUESTION 41
Which of the following code blocks creates a new DataFrame with 3 columns, productId, highest, and lowest, that shows the biggest and smallest values of column value per value in column productId from DataFrame transactionsDf?
Sample of DataFrame transactionsDf:
1.+————-+———+—–+——-+———+—-+
2.|transactionId|predError|value|storeId|productId| f|
3.+————-+———+—–+——-+———+—-+
4.| 1| 3| 4| 25| 1|null|
5.| 2| 6| 7| 2| 2|null|
6.| 3| 3| null| 25| 3|null|
7.| 4| null| null| 3| 2|null|
8.| 5| null| null| null| 2|null|
9.| 6| 3| 2| 25| 2|null|
10.+————-+———+—–+——-+———+—-+

transactionsDf.max(‘value’).min(‘value’)

transactionsDf.agg(max(‘value’).alias(‘highest’), min(‘value’).alias(‘lowest’))

transactionsDf.groupby(col(productId)).agg(max(col(value)).alias(“highest”), min(col(value)).alias(“lowest”))

transactionsDf.groupby(‘productId’).agg(max(‘value’).alias(‘highest’), min(‘value’).alias(‘lowest’))

transactionsDf.groupby(“productId”).agg({“highest”: max(“value”), “lowest”: min(“value”)})

NEW QUESTION 42
Which of the following code blocks returns a copy of DataFrame itemsDf where the column supplier has been renamed to manufacturer?

itemsDf.withColumn([“supplier”, “manufacturer”])

itemsDf.withColumn(“supplier”).alias(“manufacturer”)

itemsDf.withColumnRenamed(“supplier”, “manufacturer”)

itemsDf.withColumnRenamed(col(“manufacturer”), col(“supplier”))

itemsDf.withColumnsRenamed(“supplier”, “manufacturer”)

Explanation
itemsDf.withColumnRenamed(“supplier”, “manufacturer”)
Correct! This uses the relatively trivial DataFrame method withColumnRenamed for renaming column supplier to column manufacturer.
Note that the question asks for “a copy of DataFrame itemsDf”. This may be confusing if you are not familiar with Spark yet. RDDs (Resilient Distributed Datasets) are the foundation of Spark DataFrames and are immutable. As such, DataFrames are immutable, too. Any command that changes anything in the DataFrame therefore necessarily returns a copy, or a new version, of it that has the changes applied.
itemsDf.withColumnsRenamed(“supplier”, “manufacturer”)
Incorrect. Spark’s DataFrame API does not have a withColumnsRenamed() method.
itemsDf.withColumnRenamed(col(“manufacturer”), col(“supplier”))
No. Watch out – although the col() method works for many methods of the DataFrame API, withColumnRenamed is not one of them. As outlined in the documentation linked below, withColumnRenamed expects strings.
itemsDf.withColumn([“supplier”, “manufacturer”])
Wrong. While DataFrame.withColumn() exists in Spark, it has a different purpose than renaming columns.
withColumn is typically used to add columns to DataFrames, taking the name of the new column as a first, and a Column as a second argument. Learn more via the documentation that is linked below.
itemsDf.withColumn(“supplier”).alias(“manufacturer”)
No. While DataFrame.withColumn() exists, it requires 2 arguments. Furthermore, the alias() method on DataFrames would not help the cause of renaming a column much. DataFrame.alias() can be useful in addressing the input of join statements. However, this is far outside of the scope of this question. If you are curious nevertheless, check out the link below.
More info: pyspark.sql.DataFrame.withColumnRenamed – PySpark 3.1.1 documentation, pyspark.sql.DataFrame.withColumn – PySpark 3.1.1 documentation, and pyspark.sql.DataFrame.alias – PySpark 3.1.2 documentation (https://bit.ly/3aSB5tm , https://bit.ly/2Tv4rbE , https://bit.ly/2RbhBd2) Static notebook | Dynamic notebook: See test 1 (https://flrs.github.io/spark_practice_tests_code/#1/31.html ,
https://bit.ly/sparkpracticeexams_import_instructions)

NEW QUESTION 43
The code block displayed below contains an error. The code block should produce a DataFrame with color as the only column and three rows with color values of red, blue, and green, respectively.
Find the error.
Code block:
1.spark.createDataFrame([(“red”,), (“blue”,), (“green”,)], “color”)
Instead of calling spark.createDataFrame, just DataFrame should be called.

The commas in the tuples with the colors should be eliminated.

The colors red, blue, and green should be expressed as a simple Python list, and not a list of tuples.

Instead of color, a data type should be specified.

The “color” expression needs to be wrapped in brackets, so it reads [“color”].

1. filter
2. col(“supplier”).isin(“Sports”)
3. “itemName”
4. explode(col(“attributes”))

1. where
2. col(“supplier”).contains(“Sports”)
3. “itemName”
4. “attributes”

1. where
2. col(supplier).contains(“Sports”)
3. explode(attributes)
4. itemName

1. where
2. “Sports”.isin(col(“Supplier”))
3. “itemName”
4. array_explode(“attributes”)

1. filter
2. col(“supplier”).contains(“Sports”)
3. “itemName”
4. explode(“attributes”)

Explanation
Output of correct code block:
+———————————-+——+
|itemName |col |
+———————————-+——+
|Thick Coat for Walking in the Snow|blue |
|Thick Coat for Walking in the Snow|winter|
|Thick Coat for Walking in the Snow|cozy |
|Outdoors Backpack |green |
|Outdoors Backpack |summer|
|Outdoors Backpack |travel|
+———————————-+——+
The key to solving this question is knowing about Spark’s explode operator. Using this operator, you can extract values from arrays into single rows. The following guidance steps through the answers systematically from the first to the last gap. Note that there are many ways to solving the gap questions and filtering out wrong answers, you do not always have to start filtering out from the first gap, but can also exclude some answers based on obvious problems you see with them.
The answers to the first gap present you with two options: filter and where. These two are actually synonyms in PySpark, so using either of those is fine. The answer options to this gap therefore do not help us in selecting the right answer.
The second gap is more interesting. One answer option includes “Sports”.isin(col(“Supplier”)). This construct does not work, since Python’s string does not have an isin method. Another option contains col(supplier). Here, Python will try to interpret supplier as a variable. We have not set this variable, so this is not a viable answer. Then, you are left with answers options that include col (“supplier”).contains(“Sports”) and col(“supplier”).isin(“Sports”). The question states that we are looking for suppliers whose name includes Sports, so we have to go for the contains operator here.
We would use the isin operator if we wanted to filter out for supplier names that match any entries in a list of supplier names.
Finally, we are left with two answers that fill the third gap both with “itemName” and the fourth gap either with explode(“attributes”) or “attributes”. While both are correct Spark syntax, only explode (“attributes”) will help us achieve our goal. Specifically, the question asks for one attribute from column attributes per row – this is what the explode() operator does.
One answer option also includes array_explode() which is not a valid operator in PySpark.
More info: pyspark.sql.functions.explode – PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3

NEW QUESTION 45
Which of the following statements about Spark’s DataFrames is incorrect?

Spark’s DataFrames are immutable.

Spark’s DataFrames are equal to Python’s DataFrames.

Data in DataFrames is organized into named columns.

RDDs are at the core of DataFrames.

The data in DataFrames may be split into multiple chunks.

1. select
2. count
3. col(“itemNameElements”)
4. >3

1. filter
2. count
3. itemNameElements
4. >=3

1. select
2. count
3. “itemNameElements”
4. >3

1. filter
2. size
3. “itemNameElements”
4. >=3
(Correct)

1. select
2. size
3. “itemNameElements”
4. >3

NEW QUESTION 47
The code block displayed below contains an error. The code block should merge the rows of DataFrames transactionsDfMonday and transactionsDfTuesday into a new DataFrame, matching column names and inserting null values where column names do not appear in both DataFrames. Find the error.
Sample of DataFrame transactionsDfMonday:
1.+————-+———+—–+——-+———+—-+
2.|transactionId|predError|value|storeId|productId| f|
3.+————-+———+—–+——-+———+—-+
4.| 5| null| null| null| 2|null|
5.| 6| 3| 2| 25| 2|null|
6.+————-+———+—–+——-+———+—-+
Sample of DataFrame transactionsDfTuesday:
1.+——-+————-+———+—–+
2.|storeId|transactionId|productId|value|
3.+——-+————-+———+—–+
4.| 25| 1| 1| 4|
5.| 2| 2| 2| 7|
6.| 3| 4| 2| null|
7.| null| 5| 2| null|
8.+——-+————-+———+—–+
Code block:
sc.union([transactionsDfMonday, transactionsDfTuesday])

The DataFrames’ RDDs need to be passed into the sc.union method instead of the DataFrame variable names.

Instead of union, the concat method should be used, making sure to not use its default arguments.

Instead of the Spark context, transactionDfMonday should be called with the join method instead of the union method, making sure to use its default arguments.

Instead of the Spark context, transactionDfMonday should be called with the union method.

Instead of the Spark context, transactionDfMonday should be called with the unionByName method instead of the union method, making sure to not use its default arguments.

NEW QUESTION 48
Which of the following describes the role of tasks in the Spark execution hierarchy?

Tasks are the smallest element in the execution hierarchy.

Within one task, the slots are the unit of work done for each partition of the data.

Tasks are the second-smallest element in the execution hierarchy.

Stages with narrow dependencies can be grouped into one task.

Tasks with wide dependencies can be grouped into one stage.

NEW QUESTION 49
The code block shown below should write DataFrame transactionsDf to disk at path csvPath as a single CSV file, using tabs (t characters) as separators between columns, expressing missing values as string n/a, and omitting a header row with column names. Choose the answer that correctly fills the blanks in the code block to accomplish this.
transactionsDf.__1__.write.__2__(__3__, ” “).__4__.__5__(csvPath)

1. coalesce(1)
2. option
3. “sep”
4. option(“header”, True)
5. path

1. coalesce(1)
2. option
3. “colsep”
4. option(“nullValue”, “n/a”)
5. path

1. repartition(1)
2. option
3. “sep”
4. option(“nullValue”, “n/a”)
5. csv
(Correct)

1. csv
2. option
3. “sep”
4. option(“emptyValue”, “n/a”)
5. path
* 1. repartition(1)
2. mode
3. “sep”
4. mode(“nullValue”, “n/a”)
5. csv

NEW QUESTION 50
Which of the following describes a shuffle?

A shuffle is a process that is executed during a broadcast hash join.

A shuffle is a process that compares data across executors.

A shuffle is a process that compares data across partitions.

A shuffle is a Spark operation that results from DataFrame.coalesce().

A shuffle is a process that allocates partitions to executors.

NEW QUESTION 51
The code block displayed below contains an error. The code block is intended to join DataFrame itemsDf with the larger DataFrame transactionsDf on column itemId. Find the error.
Code block:
transactionsDf.join(itemsDf, “itemId”, how=”broadcast”)

The syntax is wrong, how= should be removed from the code block.

The join method should be replaced by the broadcast method.

Spark will only perform the broadcast operation if this behavior has been enabled on the Spark cluster.

The larger DataFrame transactionsDf is being broadcasted, rather than the smaller DataFrame itemsDf.

broadcast is not a valid join type.

NEW QUESTION 52
The code block displayed below contains an error. When the code block below has executed, it should have divided DataFrame transactionsDf into 14 parts, based on columns storeId and transactionDate (in this order). Find the error.
Code block:
transactionsDf.coalesce(14, (“storeId”, “transactionDate”))

The parentheses around the column names need to be removed and .select() needs to be appended to the code block.

Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .count() needs to be appended to the code block.
(Correct)

Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .select() needs to be appended to the code block.

Operator coalesce needs to be replaced by repartition and the parentheses around the column names need to be replaced by square brackets.

Operator coalesce needs to be replaced by repartition.

NEW QUESTION 53
Which of the following options describes the responsibility of the executors in Spark?

The executors accept jobs from the driver, analyze those jobs, and return results to the driver.

The executors accept tasks from the driver, execute those tasks, and return results to the cluster manager.

The executors accept tasks from the driver, execute those tasks, and return results to the driver.

The executors accept tasks from the cluster manager, execute those tasks, and return results to the driver.

The executors accept jobs from the driver, plan those jobs, and return results to the cluster manager.