Using Default or Custom Failure Handling

Airflow executors submit tasks to Qubole and keep track of them. These executors (task-instances) also register heartbeats with the Airflow database periodically. A task-instance is marked as zombie if it fails to register the heartbeat in a configured amount of time.

The Airflow scheduler checks for zombie processes in the system and if necessary invokes the failure-handler for the task in question. Qubole ships a default failure-handler with the Qubole operator; this checks for the corresponding Qubole command and kills it if necessary. If the command succeeds, the handler changes the task-instance state to Success.

You can override this behaviour by providing a custom failure-handler in the task definition, as shown in the following example:

def my_failure_handler(context):
"""
custom logic to handle command failures
"""

hive_task = QuboleOperator(task_id='hive_show_table',
                        command_type='hivecmd',
                        query='show tables',
                        on_failure_callback=my_failure_handler,
                        dag=dag)