QDS Components: Supported Versions and Cloud Platforms¶
Supported Versions¶
The following table shows the currently supported open source versions of QDS components and the Cloud platforms on which they run.
Important
Any other software available on the cluster is subject to change or removal. In general, if you want to install any required software version, then Qubole strongly recommends to install it on a cluster through the node bootstrap.
| QDS Component | Currently Supported Versions | Supported on (Cloud Platforms) |
|---|---|---|
| Airflow | 1.10.0, 1.10.2QDS, 1.10.9QDS (beta) | AWS, Azure, Oracle OCI, GCP |
| Cascading | All | AWS, Azure, Oracle OCI |
| Hadoop 2 | 2.6.0 | AWS, Azure, Oracle OCI, GCP |
| Hive | 1.2 (deprecated), 2.1.1, 2.3, 3.1.1 (beta) | AWS |
| 1.2 (deprecated), 2.1.1, 2.3 | Azure, Oracle OCI | |
| 2.1.1, 2.3 | GCP | |
| Java |
|
AWS, Azure, Oracle OCI |
|
AWS, Azure | |
|
GCP | |
| MapReduce | 0.20.1 | AWS |
| 2.6.0 | AWS, Azure, Oracle OCI, GCP | |
| Pig | 0.11, 0.15, 0.17 (beta) | AWS |
| 0.11 | Azure, Oracle OCI | |
| Presto | 0.193 (deprecated), 0.208, and 317 | AWS |
| 0.193 (deprecated), 0.208, and 317 | Azure | |
| 0.208, 317 | GCP | |
| Python | 2.6, 2.7, and 3.5 Airflow supports only 2.7 and later |
AWS See Can I use Python 2.7 for Hadoop tasks? for more information |
| 2.7 and 3.5 | Azure, Oracle OCI | |
| 3.6.x and 3.7 | GCP | |
| R | 3.3.3 | AWS, Oracle OCI |
| 3.3.2 | Azure | |
| 3.5.2 | GCP | |
| RubiX | 0.2.11 | AWS, Azure |
| Scala | 2.10 for Spark versions older than 2.0.0 | AWS |
| 2.11 for Spark 2.0.0 and later | AWS, Azure, Oracle OCI, GCP | |
| Spark (See also Spark Version Support) | 1.6.2, 2.0.2, 2.1.1, 2.2.0, 2.2.1, 2.3.1, 2.4.0, 2.4.3 | AWS |
| 2.0 (deprecated), 2.1.0 (deprecated), 2.1 (deprecated), 2.2.1, 2.3.2, 2.4.3 | Azure, Oracle OCI | |
| 2.3.2, 2.4, 2.4.3 | GCP | |
| Sqoop | 1.4.7 | AWS |
| 1.4.6 | AWS, Azure, Oracle OCI | |
| Tez | 0.7, 0.8.4, 0.9.1 | AWS |
| 0.7, 0.8.4 | Azure, Oracle OCI | |
| 0.8.4 | GCP | |
| Zeppelin (notebooks) | 0.6.2, 0.8.0 | AWS, Azure, Oracle OCI, GCP |
| JupyterLab (notebooks) | 1.2.3 (beta) | AWS, Azure, GCP |
Query Engine Version Lifecycle Phases¶
A query engine’s version lifecycle phases are briefly explained in this following table.
| Beta | Supported | Deprecated | Expired | |
|---|---|---|---|---|
| Production use | No Production SLAs | Available | Available but not recommended | Not available; should upgrade |
| Incident Support | Available | Available | Available | Not available; should upgrade |
| Security Updates | Available | Available | Not available; should upgrade | Not available; should upgrade |
| Bug fixes | Available | Available | Not available; should upgrade | Not available; should upgrade |
| Feature requests | Will be considered | Will be considered | Not available; should upgrade | Not available; should upgrade |
| Visibility | Visible in UI | Visible in UI | Visible in UI | Not listed in UI; cannot start new clusters with expired versions of software |
| Upgrade | Customer initiated | Customer initiated | Customer initiated | Qubole will initiate automatic upgrade upon restart |
For more information on the deprecated and end-of-life timelines of engine versions, see Deprecated Versions.
Deprecated Versions¶
The following table shows deprecated versions and corresponding timelines.
| Engines on QDS | Engine Versions | Deprecation Timeline | Expiry Timeline |
|---|---|---|---|
| Airflow | 1.7 | May 2019 | October 2019 |
| 1.8.2 | October 2019 | ETA: R60 | |
| Hadoop | 1 | October 2018 | April 2019 |
| 2.8 | May 2019 | November 2019 | |
| Hive | 0.13 | Not supported for new customers. For legacy customers, support ended in February 2019. | February 2019 |
| 1.2 | May 2020 | ETA: R60 | |
| Presto | 0.180 | October 2019 | May 2020 |
| 0.193 | February 2020 | ETA: R60 | |
| Spark | 1.3.1, 1.4.0, 1.4.1, 1.5.0 | 2017 | ETA: R60 |
| 1.5.1, 1.6.0, 1.6.1, 2.0.0, 2.1.0 | November 2018 | ETA: R60 |
Spark Version Support¶
QDS supports two major versions of Spark, 1.x and 2.x. As of April 2017, Qubole began phasing out support for older versions. Those versions are marked deprecated in the drop-down list of available versions on the Clusters page of the QDS UI. You can still launch clusters running a version marked deprecated, but:
- No new features or bug fixes will be applied to this version.
- The version will no longer be eligible for Qubole Customer Support; tickets will not be addressed.
Note
In the Spark Version drop-down list on the Clusters page of the QDS UI, latest means the latest open-source maintenance version. If you choose latest, Qubole Spark is automatically upgraded when a new maintenance version is released. For example, if 2.x latest currently points to Spark 2.x.y, then when 2.x.(y+1) is released, QDS clusters running 2.x latest automatically start using 2.x.(y+1) when the cluster is restarted.