Troubleshooting Spark Issues

When any Spark job or application fails, you should identify the errors and exceptions that cause the failure. You can access the Spark logs to identify errors and exceptions.

This topic provides information about the errors and exceptions that you might encounter when running Spark jobs or applications. You can resolve these errors and exceptions by following the respective workarounds.

• Out of Memory Exceptions
  • Driver Memory Exceptions
    • Exception due to Spark driver running out of memory
    • Job failure because the Application Master that launches the driver exceeds memory limits
  • Executor Memory Exceptions
    • Exception because executor runs out of memory
    • FetchFailedException due to executor running out of memory
    • Executor container killed by YARN for exceeding memory limits
• Spark job repeatedly fails
• Spark Shell Command failure
• Error when the total size of results is greater than the Spark Driver Max Result Size value
• Too Large Frame error
• Spark jobs fail because of compilation failures

You might want to use the Sparklens experimental open service tool that is available on http://sparklens.qubole.net to identify the potential opportunities for optimizations with respect to driver side computations, lack of parallelism, skew, etc. For more information about Sparklens, see the Sparklens blog.

Out of Memory Exceptions

Spark jobs might fail due to out of memory exceptions at the driver or executor end. When troubleshooting the out of memory exceptions, you should understand how much memory and cores the application requires, and these are the essential parameters for optimizing the Spark appication. Based on the resource requirements, you can modify the Spark application parameters to resolve the out-of-memory exceptions.

For more information about resource allocation, Spark application parameters, and determining resource requirements, see An Introduction to Apache Spark Optimization in Qubole.

Driver Memory Exceptions

Exception due to Spark driver running out of memory

• Description: When the Spark driver runs out of memory, exceptions similar to the following exception occur.

  Exception in thread "broadcast-exchange-0" java.lang.OutOfMemoryError: Not enough memory to build and broadcast the table
  to all worker nodes. As a workaround, you can either disable broadcast by setting spark.sql.autoBroadcastJoinThreshold to -1
  or increase the spark driver memory by setting spark.driver.memory to a higher value
  

• Resolution: Set a higher value for the driver memory, using one of the following commands in Spark Submit Command Line Options on the Workbench page:

  • --conf spark.driver.memory= <XX>g

    OR

  • --driver-memory <XX>G

Job failure because the Application Master that launches the driver exceeds memory limits

• Description: A Spark job may fail when the Application Master (AM) that launches the driver exceeds the memory limit and is eventually terminated by YARN. The following error occurs:

  Diagnostics: Container [pid=<XXXXX>,containerID=container_<XXXXXXXXXX>_<XXXX>_<XX>_<XXXXXX>] is running beyond physical memory limits.
  Current usage: <XX> GB of <XX> GB physical memory used; <XX> GB of <XX> GB virtual memory used. Killing container
  

• Resolution: Set a higher value for the driver memory, using one of the following commands in Spark Submit Command Line Options on the Workbench page:

  • --conf spark.driver.memory= <XX>g

    OR

  • --driver-memory <XX>G

  As a result, a higher value is set for the AM memory limit.

Executor Memory Exceptions

Exception because executor runs out of memory

• Description: When the executor runs out of memory, the following exception might occur.

  Executor task launch worker for task XXXXXX ERROR Executor: Exception in task XX.X in stage X.X (TID XXXXXX)
  java.lang.OutOfMemoryError: GC overhead limit exceeded
  

• Resolution: Set a higher value for the executor memory, using one of the following commands in Spark Submit Command Line Options on the Workbench page:

  • --conf spark.executor.memory= <XX>g

    OR

  • --executor-memory <XX>G

FetchFailedException due to executor running out of memory

• Description: When the executor runs out of memory, the following exception may occur.

  ShuffleMapStage XX (sql at SqlWrapper.scala:XX) failed in X.XXX s due to org.apache.spark.shuffle.FetchFailedException:
  failed to allocate XXXXX byte(s) of direct memory (used: XXXXX, max: XXXXX)
  

• Resolution: From the Workbench page, perform the following steps in Spark Submit Command Line Options:

  1. Set a higher value for the executor memory, using one of the following commands:

     • --conf spark.executor.memory= <XX>g

       OR

     • --executor-memory <XX>G

  2. Increase the number of shuffle partitions, using the following command: --spark.sql.shuffle.partitions

Executor container killed by YARN for exceeding memory limits

• Description: When the container hosting the executor needs more memory for overhead tasks or executor tasks, the following error occurs.

  org.apache.spark.SparkException: Job aborted due to stage failure: Task X in stage X.X failed X times,
  most recent failure: Lost task X.X in stage X.X (TID XX, XX.XX.X.XXX, executor X): ExecutorLostFailure
  (executor X exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. XX.X GB
  of XX.X GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead
  

• Resolution: Set a higher value for spark.yarn.executor.memoryOverhead based on the requirements of the job. The executor memory overhead value increases with the executor size (approximately by 6-10%). As a best practice, modify the executor memory value accordingly.

  To set a higher value for executor memory overhead, enter the following command in Spark Submit Command Line Options on the Workbench page: --conf spark.yarn.executor.memoryOverhead=XXXX

  Note

  For Spark 2.3 and later versions, use the new parameter spark.executor.memoryOverhead instead of spark.yarn.executor.memoryOverhead.

  If increasing the executor memory overhead value or executor memory value does not resolve the issue, you can either use a larger instance, or reduce the number of cores.

  To reduce the njmber of cores, enter the following in the Spark Submit Command Line Options on the Workbench page: --executor-cores=XX. Reducing the number of cores can waste memory, but the job will run.

Spark job repeatedly fails

• Description: When the cluster is fully scaled and the cluster is not able to manage the job size, the Spark job may fail repeatedly.
• Resolution: Run the Sparklens tool to analyze the job execution and optimize the configuration accordingly.

For more information about Sparklens, see the Sparklens blog.

Spark Shell Command failure

• Description: When a spark application is submitted through a shell command in QDS, it may fail with the following error.

  Qubole > Shell Command failed, exit code unknown
  2018-08-02 12:43:18,031 WARNING shellcli.py:265 - run - Application failed or got killed
  

  In this case, the actual reason that kills the application is hidden and you might not able to find the reason in the logs directly.

• Resolution:

  1. Navigate to the Workbench page
  2. Click on the Resources tab to analyze the errors and perform the appropriate action.
  3. Run the job again using the Spark Submit Command Line Options on the Workbench page.

Error when the total size of results is greater than the Spark Driver Max Result Size value

• Description: When the total size of results is greater than the Spark Driver Max Result Size value, the following error occurs.

  org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of x tasks (y MB) is bigger
  than spark.driver.maxResultSize (z MB)
  

• Resolution: Increase the Spark Drive Max Result Size value by modifying the value of --conf spark.driver.maxResultSize

  in the Spark Submit Command Line Options on the Workbench page.

Too Large Frame error

• Description: When the size of the shuffle data blocks exceeds the limit of 2 GB, which spark can handle, the following error occurs.

  org.apache.spark.shuffle.FetchFailedException: Too large frame: XXXXXXXXXX
  at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:513)
  at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:444)
  
  
  
  Caused by: java.lang.IllegalArgumentException: Too large frame: XXXXXXXXXX
  at org.spark_project.guava.base.Preconditions.checkArgument(Preconditions.java:119)
  at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:133)
  

• Resolution: Perform one of the following steps to resolve this error:

  Solution 1:

  1. Run the job on Spark 2.2 or higher version because Spark 2.2 or higher handles this issue in a better way when compared to other lower versions of Spark. For information, see https://issues.apache.org/jira/browse/SPARK-19659.

  2. Use the following Spark configuration:
     1. Modify the value of spark.sql.shuffle.partitions from the default 200 to a value greater than 2001.
     2. Set the value of spark.default.parallelism to the same value as spark.sql.shuffle.partitions.

  Solution 2:

  1. Identify the DataFrame that is causing the issue.
     1. Add a Spark action(for instance, df.count()) after creating a new DataFrame.
     2. Print anything to check the DataFrame.
     3. If the print statement is not executed for a DataFrame, then the issue is with that DataFrame.
  2. After the DataFrame is identified, repartition the DataFrame by using df.repartition() and then cache it by using df.cache().
  3. If there is skewness in the data and you are using Spark version earlier than 2.2, then modify the code.

Spark jobs fail because of compilation failures

When you run a Spark program or application from the Workbench page, the code is compiled and then submitted for execution.

If there are syntax errors, or the JARs or classes are missing, jobs may fail during compilation or at runtime.

• Description: If there are any errors in the syntax, the job may fail even before the job is submitted.

  The following figure shows a syntax error in a Spark program written in Scala.

  

• Resolution: Check the code for any syntax errors and rectify the syntax. Rerun the program.

  The following figure shows a Spark job that ran successfully and displayed results.

  

• Description: class/JAR-not-found errors occur when you run a Spark program that uses functionality in a JAR that is not available in the Spark program’s classpath; the error occurs either during compilation, or, if the program is compiled locally and then submitted for execution, at runtime.

  The following figure shows an example of a class-not-found error.

  

• Resolution: Add the dependent classes and jars and rerun the program. See Specifying Dependent Jars for Spark Jobs.