Spark Versions Supportability MatrixΒΆ
Spark versions support different versions of components related to Spark.
The following table lists the supported components and versions for the Spark 3 and Spark 2.x versions.
| Components | Supported Version for Spark 3 | Supported Version for Spark 2.x |
|---|---|---|
| Scala | 2.12 (default) | 2.11 (default) |
| Java | 8 | 8 |
| Hive Metastore | >= 2.3 (default) | 1.2.1, 2.3, 2.1, 3.1 |
| Python | 3.x, 2.x | 2.x, 3.x |
| Hadoop | 2.x | 2.x |
Note
- For Spark 3.0, if you are using a self-managed Hive metastore and have an older metastore version (Hive 1.2), few metastore operations from Spark applications might fail. Therefore, you should upgrade metastores to Hive 2.3 or later version. QDS-managed metastore is upgraded by default.
- Python 2.x will be deprecated soon for Spark 3.x versions.