Using Default or Custom Failure HandlingΒΆ
Airflow executors submit tasks to Qubole and keep track of them. These executors (task-instances) also register heartbeats with the Airflow database periodically. A task-instance is marked as zombie if it fails to register the heartbeat in a configured amount of time.
The Airflow scheduler checks for zombie processes in the system and if necessary invokes
the failure-handler for the task in question. Qubole ships a default failure-handler
with the Qubole operator; this checks for the corresponding Qubole command and kills
it if necessary. If the command succeeds, the handler changes the task-instance state to Success
.
You can override this behaviour by providing a custom failure-handler in the task definition, as shown in the following example:
def my_failure_handler(context):
"""
custom logic to handle command failures
"""
hive_task = QuboleOperator(task_id='hive_show_table',
command_type='hivecmd',
query='show tables',
on_failure_callback=my_failure_handler,
dag=dag)