Skip to main content


Part 2: Enabling additional instrumentation

This topic explains how to configure Unravel to retrieve additional data from Hive, Tez, Spark and Oozie, such as Hive queries, application timelines, Spark jobs, YARN resource management data, and logs. You'll do this by generating Unravel's JARs and distributing them to every node that runs queries in the cluster. Later, after JARs are distributed to the nodes, you'll integrate Hive, Tez, and Spark data with Unravel.

1. Generate and distribute Unravel's Hive Hook and Spark Sensor JARs
2. Configure Ambari to work with Unravel
  1. Hive configurations

    1. Hive

      Click Hive > Configs > Advanced > Advanced hive-env. In the hive-env template, towards the end of line, add:

      export AUX_CLASSPATH=${AUX_CLASSPATH}:/usr/local/unravel-jars/unravel-hive-1.2.0-hook.jar 
    2. Hive Hook

      In Ambari's general properties, append ,com.unraveldata.dataflow.hive.hook.UnravelHiveHook, to the following properties:


      Be sure to append with no space before or after the comma, for example, property=existingValue,newValue


      For example, com.unraveldata.dataflow.hive.hook.UnravelHiveHook

    3. Custom

      In Ambari's custom hive-site editor set to unravel-gateway-internal-IP-hostname

      For example,

    4. Optional: Hive LLAP if it is enabled


      Edit hive-site.xml manually, not through Ambari Web UI.

      1. Copy the settings in Custom hive-interactive-site and paste them into /etc/hive/conf/hive-site.xml.

      2. Copy the settings in Advanced hive-interactive-env and paste them into /etc/hive/conf/hive-site.xml.


    If you have an Unravel version older than, create HDFS Hive Hook directories for Unravel:

    hdfs dfs -mkdir -p /user/unravel/HOOK_RESULT_DIR
    hdfs dfs -chown unravel:hadoop /user/unravel/HOOK_RESULT_DIR
    hdfs dfs -chmod -R 777 /user/unravel/HOOK_RESULT_DIR
  2. Configure HDFS.

    Click HDFS > Configs > Advanced > Advanced hadoop-env. In the hadoop-env template, look for export HADOOP_CLASSPATH and append Unravel's JAR path as shown.

    export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/local/unravel-jars/unravel-hive-1.2.0-hook.jar
  3. Configure the BTrace agent for Tez

    In the tez-site.xml configuration file, append the following Java options to and tez.task.launch.cmd-opts:

    -javaagent:/usr/local/unravel-jars/btrace-agent.jar=libs=mr,config=tez -Dunravel.server.hostport=unravel-host:4043


    In a Kerberos environment you need to modify property with the "run as" user or *.

  4. Configure the Application Timeline Server (ATS).


    From Unravel v4.6.1.6, this step is not mandatory.

    1. In yarn-site.xml:

      yarn.timeline-service.version=1.5 or yarn.timeline-service.versions=1.5f,2.0f
    2. If yarn.acl.enable is true, add unravel to yarn.admin.acl.

    3. In, add:

      Use ATS Logging: true
    4. In tez-site.xml, add:

      tez.history.logging.service.class=org.apache.tez.dag.history.logging.ats.ATSV15HistoryLoggingService"run-as"-user or *


      From HDP version 3.1.0 onwards, this Tez configuration must be done manually.

  5. Configure Spark-on-Yarn


    For unravel-host, use Unravel Server's fully qualified domain name (FQDN) or IP address.

    1. Add the location of the the Spark JARs.

      Click Spark > Configs > Custom spark-defaults > Add Property and use bulk.png Bulk property add mode, or edit spark-defaults.conf as follows:


      • If your cluster has only one Spark 1.X version, spark-defaults.conf is in /usr/hdp/current/spark-client/conf.

      • If your cluster is running Spark 2.X, spark-defaults.conf is in /usr/hdp/current/spark2-client/conf.

      This example uses default locations for Spark JARs. Your environment may vary.

    2. Enable Spark streaming.

  6. Configure Oozie

3. Configure the Unravel Host

Define the following properties in /usr/local/unravel/etc/ If you do not find the properties add them.

  1. Tez.

  2. Set these if the Application Timeline Server (ATS) requires authentication.

4. Optional: Confirm that Unravel UI shows Tez data.
  1. Run /usr/local/unravel/install_bin/ on the HDP cluster or on any cloud environment where hive.execution.engine=tez.

  2. Log into Unravel server and go to the Applications page. Check for Tez jobs.

    Unravel UI may take a few seconds to load Tez data.