Airflow
Airflow can be integrated with Unravel by Installing the Unravel Airflow agent in the same instance as that of the Airflow instance or an instance that has access to the Airflow host. The metadata information from Airflow is pushed to the agent. The agent pulls additional information such as logs, if necessary, consolidates all the DAG information, and forwards it to Unravel.
After integrating, you can observe the details of the pipelines in Airflow from Pipeline Health & Observability.
Do the following to integrate Airflow with Unravel:
Download and install the Unravel Airflow agent in the same instance as that of the Airflow instance or in the instance, which can be accessible via a network by Airflow.
Download the
airflow-agent.tar.gz
file.Extract the
airflow-agent.tar.gz
file.tar -zxf airflow-agent.tar.gz
As unravel user, do the following before you run the Unravel Airflow agent:
Go to the location where the Unravel Airflow agent is installed. For example:
home/unravel/airflow-agent
From
$AGENT_HOME/src/sample_airflow_configuration/config
, open the folder based on the Python version of your Airflow application and copy the file in the folder. The following folders are listed:greater_than_py3.6
lesser_equalto_py3.6
For example, if you have Airflow running with Python 3.8 version, copy the file in the folder
$AGENT_HOME/src/sample_airflow_configuration/config/greater_than_py3.6
with the following command.```cp $AGENT_HOME/src/sample_airflow_configuration/config/greater_than_py3.6/airflow_local_settings.py $AIRFLOW_HOME/config```
Note
If the Unravel Airflow agent is in a different instance than your Airflow application, open the copied file
$AIRFLOW_HOME/config/airflow_local_settings.py
and edit line number 14.```airflow_agent_url = "http://localhost:5002"```
Change
localhost
to the instance name where the Unravel Airflow agent is running.Open the
airflow.cfg
file, which is located in theAIRFLOW_HOME
folder, and edit the value of the key auth_backends. For the Airflow to support API calls specify the value for auth_backends asairflow.api.auth.backend.basic_auth
.Note
From Airflow version 2.3.0, multiple values are supported for the auth_backends key.
Restart the Airflow application. The DAGs start sending callbacks during each task execution.
Go to
$AGENT_HOME/src/config.py
and edit the following configs. These are needed for making API calls to Airflow.airflow_url: change to the URL where the Airflow webserver is running.
airflow_username: specify your Airflow application username.
airflow_password: specify your Airflow application password.
Run the Unravel Airflow agent. Go to
$AGENT_HOME/src
and execute thestart.sh
file.bash start.sh
Download the Pipeline Health and Observability from here and install it. Refer to Installing Pipeline health and observability.
From the Pipeline Health & Observability interface, you can now view the pipeline run information.