Through CI/CD (GitHub, GitLab, or Bitbucket)

  1. Configure GitHub, GitLab, or Bitbucket as Repository
  2. Create a Configuration File
  3. Deploy DAGs and Plugins using GitHub, GitLab, or Bitbucket as Repository

Configure GitHub, GitLab, or Bitbucket as Repository

Prerequisite

To select Git (GitHub or GitLab) or Bitbucket as the deployment repository for DAGs, you should first configure the Version Control Settings on the Account Settings page. For more information on how to configure GitHub, GitLab, or Bitbucket version control settings, see Configuring Version Control Systems.

Configuration

Follow the instructions below to manage your Airflow DAGs through Git or Bitbucket:

  1. Navigate to Home > Cluster.

  2. On the Clusters page, click Edit for the Airflow cluster to change its deployment repository.

  3. On the cluster details page, select Advanced Configuration tab.

  4. From the Deployment Source drop-down list (under the AIRFLOW CLUSTER SETTINGS section), select GIT Repository.

    ../../../../_images/github-repo.png

  5. Enter repository location in the Repository URL field.

  6. Enter the branch name in the Repository Branch.

  7. To create a new Airflow cluster or edit an existing one, click Create or Update and Push respectively.

You have successfully configured Git (GitHub or GitLab) or Bitbucket as the deployment repository for Airflow DAGs.

Create a Configuration File

After you configure the repository, you require the configuration file (.yaml) in your repository to enable a set of instructions for the deployment.

  1. Create a configuration file (.yaml) with this name - qubole-airflow.yaml and ensure that it has all the parameters defined as shown in the example below. Presence of this file helps to pass the Config Validation during the deployment.
  2. Specify the requirements parameter in the configuration file (.yaml). It is the relative path to the file with the list of requirements. It installs the packages through the Package Installation and clones your git repository on the cluster through Cloning Repository during the deployment.
  3. Specify the scripts parameter that helps in completion of the Running Scripts during the deployment. scripts is an array of simple commands. They are run before copying the DAGs and plugins.
  4. Specify the dags parameter to provide the relative path to the DAG files. This folder of DAG files is recursively copied to the DAGs folder on the cluster through Copy Dags during the deployment.
  5. Specify the plugins parameter to provide the relative path to the plugin files. This folder of plugin files is recursively copied to the plugins folder on the cluster through Copy Plugin during the deployment.

Note

sync-directory and dags are the mandatory parameters.

Example

steps:
  requirements: tempdir/requirements.txt
  scripts:
    - 'ls'
    - 'python setup.py'
  sync-directory:
    dags: testproject/dags
    plugins: testproject/plugins

Deploy DAGs and Plugins using GitHub, GitLab, or Bitbucket as Repository

After you create the configuration file, follow the instructions below to start deployment from Git:

  1. On the cluster details page, select Deployment tab at the left pane.

    Important

    To start deployment from GitHub, GitLab, or Bitbucket, the respective Airflow cluster must be up and running.

  2. Click Deploy. A popup window appears to ask you for confirmation.

  3. Click OK. QDS starts deployment.

    ../../../../_images/airflow-git.png