What’s New

Important new features and improvements are as follows.

Note

A link in blue text next to a description in these Release Notes indicates the launch state, availability, and default state of the item (for example, Beta). The link provides more information. Unless otherwise stated, features are generally available, available as self-service (without intervention by Qubole support), and enabled by default.

Administration

Cluster Management

Data Analytics

Data Engineering

Hive

Notebooks and Dashboards

Presto

Spark

• You can now enable and disable features at the account level through a self-service platform.
• Qubole has updated its API throttling policy. Gradual Rollout

Learn more.

• QDS now supports an account-level node bootstrap. Learn more. | Cluster Restart Required
• QDS has made major improvements in API throttling policies.
• QDS has improved its handling of cluster terminations. Learn more.| Gradual Rollout | Cluster Restart Required

• Workbench is now generally available, with improvements.

• QDS now supports Apache Airflow version 1.10.9QDS. This Airflow version is supported only with Python 3.7.
• The Airflow CLI is available as Open Source software.
• GIT is now integrated with Airflow clusters through the DAG explorer.
• The Jupyter Notebook Command is now available in QuboleOperator as jupytercmd; users can schedule their Jupyter Notebooks in Airflow (Cluster Restart Required).
• For Airflow version 1.10.9QDS, QDS exports metrics via statsd. Prometheus scrapes metrics from the statsd and QDS displays them on Grafana.

Learn more.

• Hive 1.2 is deprecated.

• Jupyter V2 notebooks provide Qviz with the Spark driver. Qviz allows you to visualize dataframes with improved charting options and Python plots. Gradual Rollout.
• Jupyter V2 provides autocomplete and Intellisense with docstring help.
• Jupyter V2 provides notebook workflows with %run.
• The Package Management UI has been redesigned with improvements. Users of existing accounts should contact Qubole Support to enable the new UI. Learn more.
• Package Management now supports private Conda channels.

Learn more about the new features in Jupyter Notebooks.

• Presto version 317 is now generally available. Learn more. | Cluster Restart Required
• The BigQuery connector is now available in Presto 317. Learn more.
• New automated workload management and operational governance capabilities provide improved performance, reliability, and TCO.
• Dynamic concurrency and hybrid autoscaling. Learn more. | Gradual Rollout | Cluster Restart Required
• Enhancements for JDBC and ODBC drivers. Learn more.
• Improvements in Dynamic Filtering. Learn more. | Cluster Restart Required
• Improvements in reading Hive ACID tables. Learn more.
• New Datadog (beta) alerts. Learn more.
• Group quotas in Resource Group-based dynamic cluster sizing. Learn more.

• Dynamic Partition Pruning: Using Dynamic Filtering values, Dynamic Partition Pruning selects the specific partitions within the table that need to be read at runtime. This improves job performance for queries in which the join condition is on the partitioned column, by significantly reducing the amount of data read and processed. Dynamic Partition Pruning is available in Spark 2.4.3 and later versions. Gradual Rollout.