Attach a Cluster to the Package Management Environment

PUT /api/v1.2/package/<env ID>/attach_cluster

Use this API to attach a cluster to a package management environment. A cluster can have only one environment attached to it. You can attach a cluster to an environment only when it is down/inactive.

When you create a new Spark cluster, by default a package environment gets created and is attached to the cluster. This feature is not enabled by default. Create a ticket with Qubole Support to enable this feature on the QDS account.

In addition to attaching Spark clusters, you can attach environments with Python 3.5 to Airflow clusters that use Airflow version 1.8.2 only if the environment is detached from the cluster. For more information, see Configuring an Airflow Cluster.

A Conda virtual environment gets created for Python and R environments. In the Spark cluster, Python and R Conda environments are located in /usr/lib/envs/. The spark.pyspark.python configuration in /usr/lib/spark/conf/spark-defaults.conf points to the Python version installed in the Conda virtual environment for a Spark cluster.

In a Spark notebook associated with a cluster attached to the package management environment, configure these in its interpreter settings to point to the virtual environment:

  • Set zeppelin.R.cmd to cluster_env_default_r
  • Set zeppelin.pyspark.python to cluster_env_default_py

Required Role

The following users can make this API call:

  • Users who belong to the system-admin or system-user group.
  • Users who belong to a group associated with a role that allows updating an environment. See Managing Groups and Managing Roles for more information.

Parameters

Note

Parameters marked in bold are mandatory. Others are optional and have default values.

Parameter Description
cluster_id It is the Spark cluster’s unique ID to which you want to attach the environment. One Spark cluster can only have one environment attached to it.

Request API Syntax

curl -X PUT -H "X-AUTH-TOKEN: <API-TOKEN>" -H "Content-Type: application/json" -H "Accept: application/json" \
-d '{"cluster_id": "<Cluster ID>"}` \
"https://gcp.qubole.com/api/v1.2/package/<env ID>/attach_cluster"

Sample API Request

Here is attaching a cluster to the environment with 100 as its ID.

curl -X PUT -H "X-AUTH-TOKEN: <API-TOKEN>" -H "Content-Type: application/json" -H "Accept: application/json" \
-d '{"cluster_id": "125"}` \
"https://gcp.qubole.com/api/v1.2/package/100/attach_cluster"