Spark

This section explains how to configure and use Spark on a Qubole cluster. It covers the following topics:

Spark Basics

Spark Versions and Guidelines

Data Science

Data Engineering

• Introduction
• Supported Interfaces for Spark
• Understanding Spark Cluster Worker Node Memory and Defaults
• Running a Simple Spark Application
• Composing Spark Commands in the Workbench Page
• Spark Integration with BigQuery
• Accessing Data Stores through Spark Clusters

• For details about supported Spark versions and deprecated Spark versions, see QDS Components: Supported Versions and Cloud Platforms.
• For more information about the features in older supported versions of Spark, see Apache Spark Documentation.

• Spark Versions Supportability Matrix
• Spark Best Practices
• Introduction to Sparklens

• Configuring a Spark Notebook
• Understanding Spark Notebooks and Interpreters
• Understanding Spark Interpreter Status
• Using the User Interpreter Mode for Spark Notebooks

• Creating a Spark Schedule
• Running Spark Notebooks in a Schedule
• Spark Streaming
• Spark Structured Streaming