Viewing Spark Application Status

You can view the status of a Spark Application that is created for the notebook in the status widget on the notebook panel. The widget also displays links to the Spark UI, Driver Logs, and Kernel Log. Additionally, you can view the progress of the Spark job when you run the code.

When you create a Jupyter notebook, the Spark application is not created. When you run any Spark bound command, the Spark application is created and started.

Viewing Spark Application Status

Click on the down arrow next to the kernel on the right corner.

The widget displays the status of the Spark application as shown below.

When the Spark Application Status is ready, you can access these logs to identify the errors for debbuging and troubleshooting issues. These links open as separate tabs in the JupyterLab interface.

Viewing Spark Job Progress

When you run a code in the Jupyter notebook, you can see the progress of the Spark job at each cell.

You can view the details of that Spark job by clicking on the View Details hyperlink.

The following animated GIF shows a sample Spark Job progress.