Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam Dumps

To read all files recursively within a nested directory structure, Spark requires the recursiveFileLookup option to be explicitly enabled. According to Databricks official documentation, when dealing with deeply nested Parquet files in a directory tree (as shown in this example), you should set:

df = spark.read.option('recursiveFileLookup', 'true').parquet('/path/events/data/')

This ensures that Spark searches through all subdirectories under /path/events/data/ and reads any Parquet files it finds, regardless of the folder depth.

Option A is incorrect because while it includes an option, inferSchema is irrelevant here and does not enable recursive file reading.

Option C is incorrect because wildcards may not reliably match deep nested structures beyond one directory level.

Option D is incorrect because it will only read files directly within /path/events/data/ and not subdirectories like /2023/01/01.

Databricks documentation reference:

'To read files recursively from nested folders, set the recursiveFileLookup option to true. This is useful when data is organized in hierarchical folder structures' --- Databricks documentation on Parquet files ingestion and options.

To define a micro-batch interval, the correct syntax is:

query = df.writeStream \

.outputMode('append') \

.trigger(processingTime='5 seconds') \

.start()

This schedules the query to execute every 5 seconds.

Continuous mode (used in Option A) is experimental and has limited sink support.

Option D is incorrect because processingTime must be a string (not an integer).

Option B triggers as fast as possible without interval control.

The default broadcast join threshold in Spark is:

spark.sql.autoBroadcastJoinThreshold = 10MB

Since df2 is only 8 MB (less than 10 MB), Spark will automatically apply a broadcast join without requiring explicit hints.

From the Spark documentation:

''If one side of the join is smaller than the broadcast threshold, Spark will automatically broadcast it to all executors.''

A is incorrect because Spark does support auto broadcast even with static plans.

B is correct: Spark will automatically broadcast df2.

C and D are incorrect because Spark's default logic handles this optimization.

Final Answer: B

By default, the spark.read.text() method reads a text file one line per record. This means that each line in a text file becomes one row in the resulting DataFrame.

To read each file as a single record, Apache Spark provides the option wholetext, which, when set to True, causes Spark to treat the entire file contents as one single string per row.

Correct usage:

df = spark.read.option('wholetext', True).text(txt_path)

This way, each record in the DataFrame will contain the full content of one file instead of one line per record.

To also include the file path, the function input_file_name() can be used to create an additional column that stores the complete path of the file being read:

from pyspark.sql.functions import input_file_name

df = spark.read.option('wholetext', True).text(txt_path) \

.withColumn('file_path', input_file_name())

This approach satisfies both requirements from the question:

Each record holds the entire contents of a file.

Each record also contains the file path from which the text was read.

Why the other options are incorrect:

B or D (lineSep) -- The lineSep option only defines the delimiter between lines. It does not combine the entire file content into a single record.

C (wholetext=False) -- This is the default behavior, which still reads one record per line rather than per file.

Reference (Databricks Apache Spark 3.5 -- Python / Study Guide):

PySpark API Reference: DataFrameReader.text --- describes the wholetext option.

PySpark Functions: input_file_name() --- adds a column with the source file path.

Databricks Certified Associate Developer for Apache Spark Exam Guide (June 2025): Section ''Using Spark DataFrame APIs'' --- covers reading files and handling DataFrames.

Underutilization with timeout warnings often indicates insufficient parallelism --- meaning there aren't enough executors to process all tasks concurrently.

Solution:

Increase the number of executors to allow more parallel task execution and better resource utilization.

Example configuration:

--conf spark.executor.instances=8

This distributes the workload more effectively across cluster nodes and reduces idle time for pending tasks.

Why the other options are incorrect:

A: Extending timeouts hides the symptom, not the root cause (lack of executors).

B: More memory per executor won't fix scheduling bottlenecks.

C: Reducing partition size may increase overhead and does not fix resource imbalance.

Databricks Exam Guide (June 2025): Section ''Troubleshooting and Tuning Apache Spark DataFrame API Applications'' --- tuning executors and cluster utilization.

Spark Configuration --- executor instances and resource scaling.

===========