What’s New

Important new features and improvements are as follows.


A link in blue text next to a description in these Release Notes indicates the launch state, availability, and default state of the item (for example, Beta). The link provides more information. Unless otherwise stated, features are generally available, available as self-service (without intervention by Qubole support), and enabled by default.

  • You can now enable and disable features at the account level through a self-service platform.
  • Qubole has updated its API throttling policy. Gradual Rollout

Learn more.

  • QDS now supports Apache Airflow version 1.10.9QDS. This Airflow version is supported only with Python 3.7.
  • The Airflow CLI is available as Open Source software.
  • GIT is now integrated with Airflow clusters through the DAG explorer.
  • The Jupyter Notebook Command is now available in QuboleOperator as jupytercmd; users can schedule their Jupyter Notebooks in Airflow (Cluster Restart Required).
  • For Airflow version 1.10.9QDS, QDS exports metrics via statsd. Prometheus scrapes metrics from the statsd and QDS displays them on Grafana.

Learn more.

  • Hive 1.2 is deprecated.
  • Jupyter V2 notebooks provide Qviz with the Spark driver. Qviz allows you to visualize dataframes with improved charting options and Python plots. Gradual Rollout.
  • Jupyter V2 provides autocomplete and Intellisense with docstring help.
  • Jupyter V2 provides notebook workflows with %run.
  • The Package Management UI has been redesigned with improvements. Users of existing accounts should contact Qubole Support to enable the new UI. Learn more.
  • Package Management now supports private Conda channels.

Learn more about the new features in Jupyter Notebooks.

  • Dynamic Partition Pruning: Using Dynamic Filtering values, Dynamic Partition Pruning selects the specific partitions within the table that need to be read at runtime. This improves job performance for queries in which the join condition is on the partitioned column, by significantly reducing the amount of data read and processed. Dynamic Partition Pruning is available in Spark 2.4.3 and later versions. Gradual Rollout.