Overview¶
The Qubole Data Service (QDS) is accessible via REST APIs.
To write and test applications that use these APIs, you can use any HTTP client in any programming language to interact with Qubole Data Service. The detailed syntax of the calls is described in subsequent sections.
You can also use the QDS Python SDK or the QDS Java SDK to interact with the Qubole Data Service. The Python SDK also provides an easy-to-use command-line interface.
Note
Qubole now supports only HTTPS. All HTTP requests are now redirected to HTTPS. This is aimed at better security for Qubole users.
Access URL¶
All URLs referenced in the documentation have the following base:
https://gcp.qubole.com/api/${V}/
where ${V} refers to the version of the API. Valid values of ${V} for the GCP APIs are currently v2 for the Account and Cluster APIs, and v1.2 all other GCP APIs.
API Versions Supported on QDS-on-GCP¶
The APIs for Qubole on Google Cloud Platform are supported as follows.
- Only the Cluster API and the Account API are supported on v2. The documentation for these APIs is based on v2.
- All other APIs are only supported on v1.2. The documentation for these APIs only describes the syntax and sample API calls with respect to v1.2.
cURL is a useful tool for testing out Qubole REST API calls from the command line.
The Qubole API is served over HTTPS and Qubole redirects all HTTP access to HTTPS. Qubole now supports only HTTPS access, which is aimed at better security for Qubole users. When using HTTPS, ensure that the certificates on the client machine are up-to-date.
Authentication¶
API calls must be authenticated with a Qubole API Token.
Navigate to Control Panel in the QDS UI and click the My Accounts tab on the left pane. Click Show for the account and copy the API token that is displayed.
Set the value of this API token to the AUTH_TOKEN environment variable when running the API examples via curl.
API Types¶
The APIs for Qubole on Google Cloud Platform are divided into the following categories:
Account API¶
These APIs let you to create a new account, edit and delete an existing account, view users, and set Hive bootstrap. See Account API for more information.
Apps API¶
These APIs allow you to create, list, and delete a Spark Job Server App. For more information, see Apps API.
Cluster API¶
These APIs allow you to create a new cluster, edit, restart, and delete an existing clusters. These APIs also allow you to view the list of clusters in a Qubole account, and add and remove nodes in some clusters. See Cluster API for more information.
Command API¶
The Command APIs let you submit queries and commands, check the status of commands, retrieve results and logs, or cancel commands. See Command API for more information. The Qubole Data Service currently supports these command types:
Custom Metastore API¶
These APIs allow you to connect to a custom metastore, edit and view the metastore details. For more information, see Custom Metastore API.
DbTap API¶
A DbTap identifies an external end point for import/export of data from QDS, such as a MySQL instance. The DbTap APIs let you create, view, edit or delete a DbTap. For more information, see DbTap API.
Folder API¶
This API is mainly used to create and manage a Notebook/Dashboard folders. For more information, see Folder API.
Groups API¶
These APIs allow you to create groups, add/delete users in a group, and assign/unassign roles to a group in a Qubole account. For more information, see Groups API.
Hive Metadata API¶
These APIs provide a set of read-only views that describe your Hive tables and the metadata. For more information, see Hive Metadata API.
Notebook API¶
These APIs allow you to create a notebook, clone, configure, run, import, and delete a notebook. For more information, see Notebook API.
Object Policy API¶
These APIs allow you to create object-level access policies for notebooks and clusters. For more information, see Object Policy API.
Reports API¶
These APIs let you view aggregated statistical and operational data for your commands. For more information, see Reports API.
Roles API¶
These APIs allow you to create and delete a role to perform a set of actions in a Qubole account. For more information, see Roles API.
Scheduler API¶
The Scheduler APIs let you schedule any command or workflow to run at regular intervals. For more information, see Scheduler API.
Sensor API¶
These APIs allow you to create file and partition sensors to monitor the file and Hive partition’s availability. For more information, see Sensor API.