Hortonworks Data Platform (HDP)

Before installing Unravel, ensure to check and complete the installation requirements. See the following instructions to download, install, and set up Unravel for the HDP platform.

Notice

The following instructions are for a single cluster environment. For installing Unravel on a multi-cluster environment, refer to Multi-cluster install.

1. Download Unravel

2. Deploy Unravel binaries

Unravel binaries are available as a tar file or RPM package. You can deploy the Unravel binaries in any directory on the server. However, the user who installs Unravel must have the write permissions to the directory where the Unravel binaries are deployed.

If the binaries are deployed to <Unravel_installation_directory>, Unravel will be available in <Unravel_installation_directory>/unravel. The directory layout for the Tar and RPM will be unravel/versions/<Directories and files>.

Option 1: Deploy Unravel from a tar file

The following steps to deploy Unravel from a tar file must be performed by a user who will run Unravel.

Create an Installation directory.
```
mkdir /path/to/installation/directory
```
For example: mkdir /opt/unravel
Note
Some locations may require root access to create a directory. In such a case, after the directory is created, change the ownership to unravel user and continue with the installation procedure as the unravel user.
```
chown -R username:groupname /path/to/installation/directory
```
For example: chown -R unravel:unravelgroup /opt/unravel
Extract and copy the Unravel tar file to the installation directory, which was created in the first step. After you extract the contents of the tar file, unravel directory is created within the installation directory.
```
tar -zxf unravel-<version>tar.gz -C /<Unravel-installation-directory>
```
For example: tar -zxf unravel-4.7.x.x.tar.gz -C /opt The Unravel directory will be available within /opt

Option 2: Deploy Unravel from an RPM package

Important

A root user should perform the following steps to deploy Unravel from an RPM package. After the RPM package is deployed, the remaining installation procedures should be performed by the unravel user.

Create an installation directory.
```
mkdir /usr/local/unravel
```
Run the following command:
```
rpm -i unravel-<version>.rpm
```
For example: rpm -i unravel-4.7.x.x.rpm
In case you want to provide a different location, you can do so by using the --prefix command. For example:
```
mkdir /opt/unravel
chown -R username:groupname /opt/unravel
rpm -i unravel-4.7.0.0.rpm --prefix /opt
```
The Unravel directory is available in /opt.
Grant ownership of the directory to a user who runs Unravel. This user executes all the processes involved in Unravel installation.
```
chown -R username:groupname /usr/local/unravel
```
For example: chown -R unravel:unravelgroup /usr/local/unravel The Unravel directory is available in /usr/local.
Continue with the installation procedures as Unravel user.

3. Install Unravel

You can install Unravel either with Interactive Precheck utility or manually.

Note

Unravel recommends installation with Interactive Precheck. However, if you have missed setting any configuration with this method, you can set them later manually.

Install Unravel with Interactive Precheck

The Interactive Precheck utility is run to validate the required configurations before installing Unravel. When you run the Interactive Precheck utility various checks are prompted for gathering configuration information. The responses you provide for these checks are used to generate a bootstrap configuration file. This file, which contains the configuration information, is then used to install Unravel.

Do the following to install and configure Unravel with Interactive Precheck.

After you download and deploy the Unravel, run the precheck.sh script from unravel/versions/X.Y.Z/healthcheck/.
For example:
/opt/unravel/versions/X.Y.Z/healthcheck/precheck.sh

Enter the necessary details when you are prompted for the following configuration information:

This section covers general information about your Unravel install. You are prompted for the following:

Data platform you want to monitor.
For Hadoop: type of Unravel node you want to configure.
For edge nodes: core node location and test connectivity.

You must answer the following prompts:

-- General information
Which data platform are you installing for?
   1- Hadoop
   2- EMR
   3- HDI
   4- Databricks
   5- Dataproc
   6- BigQuery

Select one of the above [Hadoop]: 

## You can choose a number corresponding to the platform. For example:
    Select one of the above [Hadoop]: 1
    Selected: Hadoop

Which type of Unravel node is this?
   1- Single node installation
   2- Core node - Multi cluster installation
   3- Edge node - Multi cluster installation

## You can choose a number corresponding to the type of Unravel deployment. For example:
    Select one of the above [Single node installation]: 1
    Selected: Single node installation
    User that will run unravel [unravel]:

This check allows you to configure database-related information and an external database for Unravel.

-- Database configuration
Unravel comes with PostgreSQL and can configure it automatically as its database.
Optionally you can:- Configure it to use an external database (MySQL, MariaDB or PostgreSQL).
- Have it configure and manage MySQL or MariaDB for you.


Configure an external database? (y/n) [No]: n

Do you wish to use an Unravel managed MySQL or MariaDB (no = Internal Postgresql)? (y/n) [No]: n

Will Unravel connect to a MySQL or MariaDB database (ex: hive metastore) ? (y/n) [No]: n

If you answer No, you are further prompted to select an Unravel-managed database for the installation.

If you answer Yes, you are further prompted for the type of external database that you want to configure.

-- Database configuration
   
   Configure an external database? (y/n) [No]: y

   Type
   1- PostgresQL
   2- MySQL
   3- MariaDB

If you choose a specific type of external database, you are prompted for the following database information and test connectivity to that database. Refer to Integrating Database for more details. For example: Integrate database (On-prem)

-- Database configuration
  
   Configure an external database? (y/n) [No]: y

   Type
   1- PostgresQL
   2- MySQL
   3- MariaDB

   Select one of the above []: 1
   Selected: PostgresQL

   Database hostname [None]: unrave

   Database port (integer) [None]: 666

   Database schema [None]: unravel

   Does the database use TLS (y/n) [No]: n

   Database username [None]: admin

   Database password [None] (no echo):

   Do you wish to test connecting to the external database? (y/n) [Yes]:-- Database configuration
 Configure an external database? (y/n) [No]:

If you choose MySQL or MariaDB database you are further prompted for extra packages. If you answer Yes, the Extra packages section searches for the required JDBC drivers.
```
-- Database configuration
 Will Unravel connect to a MySQL or MariaDB database (ex: hive metastore) ? (y/n) [No]: 
```

The Extra packages check shows if you use Unravel-managed MySQL/MariaDB or need JDBC drivers. Else, this check is automatically skipped.

-- Extra package location

*** JDBC drivers are required for Unravel managed MySQL or MariaDB.
*** Database software package is required for Unravel managed MySQL or MariaDB.

External package location [None]: /<my-extra-packages>
##This is the path to the directory where the required packages are located.

If the required packages are located, then the following message is shown:

The following packages will be installed:
   Database server: /my-extra-packages/mysql-5.7.27-linux-glibc2.12-x86_64.tar.gz
   JDBC driver:
     - /my-extra-packages/mysql-connector-java-5.1.48.tar.gz

 External package: Ok

If the required packages are not found, then the following error message is showing:

External package: ERROR
 - ERROR: Couldn't find jdbc drivers in /my-extra-packages
 - ERROR: Looked for:
   mysql-connector-java-*.tar.gz
   mysql-connector-java-*.jar
   mariadb-java-client-*.jar
 - ERROR: Couldn't find database server package in /my-extra-packages
 - ERROR: Looked for:
   mysql-*-linux-glibc2.12-x86_64.tar.gz
   mariadb-*-linux-x86_64.tar.gz

This check allows you to configure and test HTTPS for the unravel UI. This check prompts you for the certificate, key, password, and hostname details used to access Unravel.

Use HTTPS to access unravel? (y/n) [Yes]:  
##If you answer “Yes”, you are prompted for the path to the certificate and key. Unravel uses this information to configure TLS during installation. 
##If you answer “No”, you are shown a warning message for confirmation.

The information provided is verified for the following:

If the Key and Certificate match
If the certificate is valid
If the certificate applies for the provided hostname

This check allows you to set the Unravel UI port and verify the connectivity.

-- Unravel default port
  
  Port number (integer) [3000]: 

  Do you want to test if the port is accessible? (y/n) [Yes]: 
    
  This will open port 3000 and listen for connection for 120 seconds.
  Use your browser to test if the Unravel UI will be accessible on that port.
    
  We have detected the following hostnames:
  - some.host.example
    
 Browse to: http://some.host.example:3000
    
 ATTENTION: This address is an example. You should test with the URL that will be used to access Unravel.

A connection on port 3000 is tried and established. If the connection is successful, Unravel Port Test: OK is shown on the browser, and Unravel port: Reached is shown on the server.

This check allows you to set a custom data directory and verify the access if the directories exist. You will always find the software location where you deploy the Unravel binaries. In this check, only the space and access are tested. That data location that you have configured will be used.

-- Unravel directories

  Software [/opt/unravel]: 

  Data [/opt/unravel/data]: 

  Directories: ERROR
  - OK: 33 GB of free disk space for software.
  - ERROR: SYSH0026: Space for data 33 GB is low, recommended minimum is 100 GB.

This check allows you to configure and test email. You are prompted for host and credentials, and the following items are tested:

Connectivity
Authentication, only if provided.
Optional: Send test mail.

Following is a sample:

-- Mail server (SMTP) configuration

Unravel can send notification and alert emails.
   This will allow you to configure and test connection to a SMTP server.
   Optionally, it can also send a test email.

   You will have to provide:
   - Protocol, hostname and port
   - Credentials if required



   Configure a SMTP server? (y/n) [No]: y

   SMTP hostname [None]: smtphostname.gmail.com

   SMTP port (usually 25 for clear text, 465 for SSL, 587 for STARTLS) (integer) [None]: 587

   Security protocol
   1- None
   2- SSL
   3- StartTLS

   Select one of the above [None]: 3
   Selected: StartTLS

   Authentication required? (y/n) [Yes]: y

   Username [None]: daemon@unraveldata.com

   Password [None] (no echo):

   From [None]: daemon@unraveldata.com

   To [None]: user@unraveldata.com

   Send test email (y/n) [No]: y

Note

For more information, refer to Using the Interactive Precheck utility.

The responses that you have provided for the configuration information are used to generate a configuration file. You can use this configuration file when you run the setup command to install and configure Unravel.

After you have completed the responses, you are prompted to confirm if you want to generate the bootstrap configuration file. Press ENTER if you want to generate the bootstrap configuration file.
```
-- Unravel bootstrap configuration

   Generate a unravel bootstrap configuration file? (y/n) [Yes]:
```
The bootstrap configuration file is generated and located at /tmp/unravel-interactive-precheck/unravel-bootstrap.yaml.

Install Unravel with the bootstrap configuration file.

<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --bootstrap /tmp/unravel-interactive-precheck/unravel-bootstrap.yaml

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start all the services.

<unravel_installation_directory>/unravel/manager start

Check the status of services.
```
<unravel_installation_directory>/unravel/manager report 
```
The following service statuses are reported:
- OK: Service is up and running.
- Not Monitored: Service is not running. (Has stopped or has failed to start)
- Initializing: Services are starting up.
- Does not exist: The process unexpectedly disappeared. A restart will be attempted ten times.
You can also get the status and information for a specific service. Run the manager report command as follows:
```
<unravel_installation_directory>/unravel/manager report <service> 
```
For example: /opt/unravel/manager report auto_action

Install Unravel manually

To install Unravel manually, do the following:

1. Run setup command

About setup command

The setup command allows you to do the following:

Runs Precheck automatically to detect possible issues that prevent a successful installation. Suggestions are provided to resolve issues. Refer to Precheck filters for the expected value for each filter.
Let you run extra parameters to integrate the database of your choice.
The setup command allows you to use a managed database shipped with Unravel or an external database. When you run the setup command without additional parameters, the Unravel managed PostgreSQL database is used. Otherwise, you can specify any of the following databases, which is supported by Unravel, with the setup command:
- MySQL (Unravel managed as well as external MySQL database)
- MariaDB (Unravel managed as well as external MariaDB database)
- PostgreSQL (External PostgreSQL)
Refer to Integrate database for details.
Let you specify a separate path for the data directory other than the default path.
The Unravel data and configurations are located in the data directory. By default, the installer maintains the data directory under <Unravel installation directory>/data. You can also change the data directory's default location by running additional parameters with the setup command. To install Unravel with the setup command.
Provides more options for setup.

Notice

The Unravel user who owns the installation directory should run the setup command to install Unravel.

To install Unravel with the setup command, do the following:

Switch to Unravel user.
```
  su - <unravel user>
```
Run setup command:
Note
Refer to setup Options for all the additional parameters that you can run with the setup command
Refer to Integrate database topic and complete the pre-requisites before running the setup command with any other database other than Unravel managed PostgreSQL, which is shipped with the product. Extra parameters must be passed with the setup command when using another database.
Tip
Optionally, if you want to provide a different data directory, you can pass an extra parameter (--data-directory) with the setup command as follows:
```
<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --data-directory /the/data/directory
```
Similarly, you can configure separate directories for other unravel directories —contact support for assistance.
- PostgreSQL
  - Unravel managed PostgreSQL
    <unravel_installation_directory>/unravel/versions/<Unravel version>/setup
  - External PostgreSQL
    <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --external-database postgresql <HOST> <PORT> <SCHEMA> <USERNAME> <PASSWORD>
    The HOST, PORT, SCHEMA, USERNAME, and PASSWORD are optional fields and are prompted if missing. For example: /opt/unravel/versions/abcd.992/setup --external-database postgresql xyz.unraveldata.com 5432 unravel_db_prod unravel unraveldata
- MySQL
  - Unravel managed MySQL
    <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --extra /tmp/mysql
  - External MySQL
    <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --extra /tmp/<MySQL-directory> --external-database mysql <HOST> <PORT> <SCHEMA> <USERNAME> <PASSWORD>
    The HOST, PORT, SCHEMA, USERNAME, and PASSWORD are optional fields and are prompted if missing.
- MariaDB
  - Unravel managed MariaDB
    <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --extra /tmp/mariadb
  - External MariaDB
    <unravel_installation_directory>unravel/versions/<Unravel version>/setup --extra /tmp/<MariaDB-directory> --external-database mariadb <HOST> <PORT> <SCHEMA> <USERNAME> <PASSWORD>
    The HOST, PORT, SCHEMA, USERNAME, and PASSWORD are optional fields and are prompted if missing.
Precheck is automatically run when you run the setup command. Refer to Precheck filters for the expected value for each filter.

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start all the services.

<unravel_installation_directory>/unravel/manager start

Check the status of services.
```
<unravel_installation_directory>/unravel/manager report 
```
The following service statuses are reported:
- OK: Service is up and running.
- Not Monitored: Service is not running. (Has stopped or has failed to start)
- Initializing: Services are starting up.
- Does not exist: The process unexpectedly disappeared. A restart will be attempted ten times.
You can also get the status and information for a specific service. Run the manager report command as follows:
```
<unravel_installation_directory>/unravel/manager report <service>
```
For example: /opt/unravel/manager report auto_action

Precheck

The Precheck output displays the issues that prevent a successful installation and provides suggestions to resolve them. You must resolve each of the issues before proceeding. See Precheck filters.

After resolving the precheck issues, you must re-login or reload the shell to execute the setup command again.

Note

You can skip the precheck using the setup --skip-precheck command in certain situations.

For example:

/opt/unravel/versions/<Unravel version>/setup --skip-precheck

You can also skip the checks that you know can fail. For example, if you want to skip the Check limits option and the Disk freespace option, pick the command within the parenthesis corresponding to these failed options and run the setup command as follows:

setup --filter-precheck ~check_limits,~check_freespace

Tip

Run --help with the setup command and any combination of the setup command for complete usage details.

<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --help

Precheck filters

Filters	Description	Expected Value
System
Check uptime	Verifies the period since the last server reboot.	>24h
Clock sync	Verifies if the clock synchronization service is running on the server.	The clock synchronization service is up and running.
CPU requirement	Verifies if the server has enough CPUs to run Unravel efficiently.	Check requirements.
Memory requirement	Verifies that the server has enough memory to run Unravel efficiently.	Check requirements.
Disk access	Verifies that the user who runs unravel has access to the configured disk locations.	Unravel users can access the configured disk locations.
Disk Freespace	Verifies if the disk locations have enough free space.	Check requirements
Kerberos tools	Verifies that the Kerberos tools are available on the server to support kerberized environments.	Kerberos tools are installed.
Network ports	Verifies that the network ports used by Unravel are available.	Check requirements
OS libraries	Verifies the required libraries if run with Unravel managed MySQL.	You must install the following packages for fulfilling the OS level requirements for MySQL: `numactl-libs` (for libnuma.so) `libaio` (for libaio.so)
OS release	Verifies that the OS distribution is supported.	Check compatibility matrix
OS settings	Verifies vm.max_map_count recommended.	Check requirements
SELinux	Verifies if the SELinux status is enabled or not and provides the mode (Permissive, Disabled, Enforcing).	Check product documentation.
Check limits	Verifies that user limits are set to values	Check requirements
Hadoop
Clients	Ensure that the following Hadoop clients are installed and configured on the server: Apache Hadoop Hadoop Distributed File System (HDFS) Apache Hadoop Yarn Apache Hive Apache Beeline You can ignore the following Precheck limitations on MapR: The Hadoop client check reports missing clients (HDFS and beeline) Any check that depends on the HDFS client (for example, HDFS Access check) reports the following message: `HADH0070: hdfs client is not available` Ensure that Unravel has access to files in the MapR file system or needs to provide access manually.	Check compatibility matrix
Distribution	Verifies that the Hadoop distribution is a supported version.	Check compatibility matrix
RM HA Enabled/Disabled	Verifies if RM is running in HA mode
Healthcheck report bundle	healthcheck report tarball. This report provides the summary and information gathered by the healthcheck with the location.

Precheck Sample

/opt/unravel/versions/abcd.1004/setup 
2021-04-05 15:51:30 Sending logs to: /tmp/unravel-setup-20210405-155130.log
2021-04-05 15:51:30 Running preinstallation check...
2021-04-05 15:51:31 Gathering information ................. Ok
2021-04-05 15:51:51 Running checks .................. Ok
--------------------------------------------------------------------------------
system
 Check limits        : PASSED
 Clock sync          : PASSED
 CPU requirement     : PASSED, Available cores: 8 cores
 Disk access         : PASSED, /opt/unravel/versions/develop.1004/healthcheck/healthcheck/plugins/system is writable
 Disk freespace      : PASSED, 229 GB of free disk space is available for precheck dir.
 Kerberos tools      : PASSED
 Memory requirement  : PASSED, Available memory: 79 GB
 Network ports       : PASSED
 OS libraries        : PASSED
 OS release          : PASSED, OS release version: centos 7.6
 OS settings         : PASSED
 SELinux             : PASSED
--------------------------------------------------------------------------------
Healthcheck report bundle: /tmp/healthcheck-20210405155130-xyz.unraveldata.com.tar.gz
2021-04-05 15:51:53 Prepare to install with: /opt/unravel/versions/abcd.1004/installer/installer/../installer/conf/presets/default.yaml
2021-04-05 15:51:57 Sending logs to: /opt/unravel/logs/setup.log
2021-04-05 15:51:57 Instantiating templates ................................................................................................................................................................................................................................ Ok
2021-04-05 15:52:05 Creating parcels .................................... Ok
2021-04-05 15:52:20 Installing sensors file ............................ Ok
2021-04-05 15:52:20 Installing pgsql connector ... Ok
2021-04-05 15:52:22 Starting service monitor ... Ok
2021-04-05 15:52:27 Request start for elasticsearch_1 .... Ok
2021-04-05 15:52:27 Waiting for elasticsearch_1 for 120 sec ......... Ok
2021-04-05 15:52:35 Request start for zookeeper .... Ok
2021-04-05 15:52:35 Request start for kafka .... Ok
2021-04-05 15:52:35 Waiting for kafka for 120 sec ...... Ok
2021-04-05 15:52:37 Waiting for kafka to be alive for 120 sec ..... Ok
2021-04-05 15:52:42 Initializing pgsql ... Ok
2021-04-05 15:52:46 Request start for pgsql .... Ok
2021-04-05 15:52:46 Waiting for pgsql for 120 sec ..... Ok
2021-04-05 15:52:47 Creating database schema ................. Ok
2021-04-05 15:52:50 Generating hashes .... Ok
2021-04-05 15:52:52 Loading elasticsearch templates ............ Ok
2021-04-05 15:52:55 Creating kafka topics .................... Ok
2021-04-05 15:53:36 Creating schema objects ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... Ok
2021-04-05 15:54:03 Request stop ....................................................... Ok
2021-04-05 15:54:16 Done
[unravel@xyz ~]$

Setup Options

setup Options	Description
-h, --help	Shows help for setup.
--config CONFIG	Specify a different path to the configuration file. <unravel_installation_directory>/unravel/versions/`<Unravel version>`/setup --config `path/to/config/directory`
--enable-core	Enables core node support for non-Hadoop clusters.
--cluster-access (Edge node parameter)	Enables cluster access to the core node in a multi-cluster environment.
--data-forwarder host:port cluster-type cluster-id	Data forwarder, main unravel node.
--data-directory	Specify a different path to the data directory.
--external-database [param [param ...]]	Enable external database.
--external-database-ssl	Enable external database with SSL.
--log-file	Setup log file location. Default is `/tmp/unravel-setup-YYYYMMDD-HHMMSS.log`.
--extra DIR, -e DIR	Specify extra packages location.
--precheck	Run the preinstallation check

2. Add configurations

Run manager config auto command to automatically pull in all the Hadoop configurations. You will be prompted to provide the location and credentials for Cloudera Manager or Ambari UI.
```
<unravel_installation_directory>/unravel/manager config auto
```
If there are more than one clusters that are handled by Cloudera Manager or Ambari, you will be prompted to enable the cluster that you want to monitor. Run the following command to enable a cluster.
```
<unravel_installation_directory>/unravel/manager config cluster enable <CLUSTER_KEY>
```
Example: /opt/unravel/manager config cluster enable cluster1
Tip
Here <CLUSTER_KEY> is the name of the cluster that you want to enable for Unravel monitoring. This can be retrieved from the output shown for the manager config auto command.
The Hive metastore database password can be recovered automatically only for a cluster manager with an administrative account. Otherwise, it must be set manually as follows:
```
<Unravel installation directory>/unravel/manager config hive metastore password <CLUSTER_KEY> <HIVE_KEY> <PASSWORD>
```
Example: /opt/unravel/manager config hive metastore password cluster1 HIVE p@P@SsWorD
Tip
Here <CLUSTER_KEY> is the name of the cluster where you want to set the Hive configurations.
Also, refer to Connecting to Hive metastore in a single cluster environment.

Optional: Set up Kerberos to authenticate Hadoop services.

If you are using Kerberos authentication, set the principal path and keytab, enable Kerberos authentication, and apply the changes.

<Unravel installation directory>/unravel/manager config kerberos set --keytab </path/to/keytab file> --principal <server@example.com>
<Unravel installation directory>/unravel/manager config kerberos enable
<unravel_installation_directory>/manager config apply

If you are using Truststore certificates, run the following steps from the manager tool to add certificates to the Truststore:

Download the certificates to a directory.
Provide permissions to the user, who installs unravel, to access the certificates directory.
```
chown -R username:groupname /path/to/certificates/directory
```

Upload the certificates.

## Option 1
<unravel_installation_directory>/unravel/manager config tls trust add </path/to/the/certificate/files

or 

## Option 2
<unravel_installation_directory>/unravel/manager config tls trust add --pem </path/to/the/certificate/files>
<unravel_installation_directory>/unravel/manager config tls trust add --jks </path/to/the/certificate/files>
<unravel_installation_directory>/unravel/manager config tls trust add --pkcs12 </path/to/the/certificate/files>

Enable the Truststore

<unravel_installation_directory>/unravel/manager config tls trust <enable|disable>
<unravel_installation_directory>/unravel/manager config apply

Verify the connection.

<unravel_installation_directory>/unravel/manager verify connect <Cluster Manager-host> <Cluster Manager-port>

For example:

/opt/unravel/manager verify connect xyz.unraveldata.com 7180
-- Running: verify connect xyz.unraveldata.com 7180
 - Resolved IP: 111.17.4.123
 - Reverse lookup: ('xyz.unraveldata.com', [], ['111.17.4.123'])
 - Connection:   OK
 - TLS:      No
-- OK

If you are using TLS protocol, refer to Enabling Transport Layer Security (TLS) for Unravel UI.

Apply changes.

<unravel_installation_directory>/unravel/manager config apply

Start all the services.

<unravel_installation_directory>/unravel/manager start

Check the status of services.
```
<unravel_installation_directory>/unravel/manager report 
```
The following service statuses are reported:
- OK: Service is up and running
- Not Monitored: Service is not running. (Has stopped or has failed to start)
- Initializing: Services are starting up.
- Does not exist: The process unexpectedly disappeared. Restarts will be attempted 10 times.
You can also get the status and information for a specific service. Run the manager report command as follows:
```
<unravel_installation_directory>/unravel/manager report <service> 
```
For example: /opt/unravel/manager report auto_action
Enable additional instrumentation for HDP.
Set additional configurations, if required.
Optionally, you can run healthcheck, at this point, to verify that all the configurations and services are running successfully.
```
<unravel_installation_directory>/unravel/manager healthcheck
```
Healthcheck is run automatically on an hourly basis in the backend. You can set your email to receive the healthcheck reports.

4. Enable additional instrumentation for HDP

This topic explains how to configure Unravel to retrieve additional data from Hive, Tez, Spark, and Oozie, such as Hive queries, application timelines, Spark jobs, YARN resource management data, and logs. You can do this by generating Unravel's JARs and distributing them to every node that runs queries in the cluster. Later, after the JARs are distributed to the nodes, you can integrate Hive, Tez, and Spark data with Unravel.

1. Generate and distribute Unravel's Hive Hook and Spark Sensor JARs

Create a directory, for example, /usr/local/unravel-jars, for the JARs on the Unravel node in a cluster to set the permission.
```
mkdir /usr/local/unravel-jars
```
Example:
```
chown -R unravel:unravel /usr/local/unravel-jars
```
Generate the JARs using the unravel_hdp_setup.py script.
```
<Unravel installation directory>/unravel/manager run script unravel_hdp_setup.py --sensor-only --unravel-server <unravel-host>:3000 --spark-version <spark-version> --hive-version <hive-version> --ambari-server <ambari-host> --btrace-dir /usr/local/unravel-jars --hive-hook-dir /usr/local/unravel-jars
```
Replace the values for unravel-host, spark-version, hive-version, and ambari-host with appropriate values.
Tip
For unravel-host, specify the protocol (HTTP or HTTPS) and use the fully qualified domain name (FQDN) or IP address of Unravel Server. For example, https://playground3.unraveldata.com:3000.
For spark-version, use a Spark version that is compatible with this version of Unravel. For example,
spark-1.6 for Spark 1.6.x
spark-2.0 for Spark 2.0.x
spark-2.1 for Spark 2.1.x
spark-2.2 for Spark 2.2.x
spark-2.3 for Spark 2.3.x
spark-2.4 for Spark 2.4.x
spark-3.0 for Spark 3.0.x
For hive-version, use a Hive version that is compatible with this version of Unravel. For example,
HDP 3.x
3.1.0 for Hive 3.1.0
HDP 2.x
1.2.0 for Hive 1.2.0 or 1.2.1
0.13.0 for Hive 0.13.0
Example:
```
<Unravel installation directory>/unravel/manager run script unravel_hdp_setup.py --sensor-only --unravel-server unravel-node.domain.com:3000 --spark-version 2.4 --hive-version 3.1 --ambari-server ambari-manager-node.domain.com --btrace-dir /usr/local/unravel-jars --hive-hook-dir /usr/local/unravel-jars
```

Distribute /usr/local/unravel-jars to all worker, edge, and master nodes that run the queries.

scp -r /usr/local/unravel-jars/* root@hostname:/usr/local/unravel-jars

Examples:

Distribute JARs to	Examples
Master node	scp -r /usr/local/unravel-jars root@ambari-master-node.domain.com:/usr/local/unravel-jars
Worker node	scp -r /usr/local/unravel-jars root@ambari-worker-node.domain.com:/usr/local/unravel-jars
Edge node	scp -r /usr/local/unravel-jars root@ambari-master-edge-node.domain.com:/usr/local/unravel-jars

Distribute JARs to

Examples

Master node

scp -r /usr/local/unravel-jars root@ambari-master-node.domain.com:/usr/local/unravel-jars

Worker node

scp -r /usr/local/unravel-jars root@ambari-worker-node.domain.com:/usr/local/unravel-jars

Edge node

scp -r /usr/local/unravel-jars root@ambari-master-edge-node.domain.com:/usr/local/unravel-jars

Make sure the node can reach port 4043 of Unravel Server.
Upload JAR files to HDFS.
Example:
```
hadoop fs -put /home/hduser/1.txt /
```

2. Configure Ambari to work with Unravel

Hive configurations
1. Import the hive hook sensor jar into the classpath
  On the Ambari UI, click Hive > Configs > Advanced > Advanced hive-env. In the hive-env template, towards the end of line, add:
```
export AUX_CLASSPATH=${AUX_CLASSPATH}:<path to unravel hive hook sensor jar>/unravel-hive-<version>-hook.jar 
```
  For example:
```
export AUX_CLASSPATH=${AUX_CLASSPATH}:/usr/local/unravel-jars/unravel-hive-1.2.0-hook.jar 
```
2. Configure hive hook
  On the Ambari UI, click Hive > Configs > Advanced. In the General section, search for the following hive hooks:
  hive.exec.failure.hooks
  hive.exec.post.hooks
  hive.exec.pre.hooks
  hive.exec.run.hooks
  Copy the ,com.unraveldata.dataflow.hive.hook.UnravelHiveHook property against each of the hooks.
  Important
  Ensure to add the property without any space before or after the comma, for example, property=existingValue,newValue
  For example:
```
hive.exec.failure.hooks=existing-value,com.unraveldata.dataflow.hive.hook.UnravelHiveHook
hive.exec.post.hooks=existing-value,com.unraveldata.dataflow.hive.hook.UnravelHiveHook
hive.exec.pre.hooks=existing-value,com.unraveldata.dataflow.hive.hook.UnravelHiveHook
hive.exec.run.hooks=existing-value,com.unraveldata.dataflow.hive.hook.UnravelHiveHook
```
  If you do not find these hive hooks, go to the Custom hive-site section, click Add Property and add these as key and value per line in the Properties text box.
  For example:
```
hive.exec.pre.hooks=com.unraveldata.dataflow.hive.hook.UnravelHiveHook
```
  Similarly, in the Custom hive-site section. ensure to set com.unraveldata.host: to <unravel-gateway-internal-IP-hostname>

Configure HDFS

Click HDFS > Configs > Advanced > Advanced hadoop-env. In the hadoop-env template, look for export HADOOP_CLASSPATH and append Unravel's JAR path as shown.

export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:<Unravel sensor installation directory>/unravel-hive-<version>-hook.jar

For example:

export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/local/unravel-jars/unravel-hive-3.1.0-hook.jar

Configure the BTrace agent for Tez
From the Ambari UI, go to Tez > config and in the General section, append the Java options below to tez.am.launch.cmd-opts and tez.task.launch.cmd-opts:
```
-javaagent:/usr/local/unravel-jars/jars/btrace-agent.jar=libs=mr,config=tez -Dunravel.server.hostport=<unravel-host>:4043
```
Tip
In a Kerberos environment, you must modify tez.am.view-acls property with the "run as" user or *.
Configure the Application Timeline Server (ATS)
Note
From Unravel v4.6.1.6, this step is not mandatory.
1. In yarn-site.xml:
```
yarn.timeline-service.enabled=true
yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes=org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl
yarn.timeline-service.version=1.5 or yarn.timeline-service.versions=1.5f,2.0f
```
2. If yarn.acl.enable is true, add unravel to yarn.admin.acl.
3. In hive-env.sh, add:
```
Use ATS Logging: true
```
4. In tez-site.xml, add:
```
tez.dag.history.logging.enabled=true
tez.am.history.logging.enabled=true
tez.history.logging.service.class=org.apache.tez.dag.history.logging.ats.ATSV15HistoryLoggingService
tez.am.view-acls=*
```
  If tez-site.xml is not available, you can also add these properties from the Ambari UI. Go to Tez > config > Custom tez-site and add the above properties as key and value per line in the Properties text box.
  Note
  From HDP version 3.1.0 onwards, this Tez configuration must be done manually.
Configure Spark-on-Yarn
Tip
For unravel-host, use Unravel Server's fully qualified domain name (FQDN) or IP address.
For spark-version, use a Spark version that is compatible with this version of Unravel. For example,
- spark-1.6 for Spark 1.6.x
- spark-2.0 for Spark 2.0.x
- spark-2.1 for Spark 2.1.x
- spark-2.2 for Spark 2.2.x
- spark-2.3 for Spark 2.3.x
- spark-2.4 for Spark 2.4.x
- spark-3.0 for Spark 3.0.x
1. Add the location of the Spark JARs.
  Click Spark > Configs > Custom spark-defaults > Add Property and use Bulk property add mode, or edit spark-defaults.conf as follows:
  Tip
  If your cluster has only one Spark 1.X version, spark-defaults.conf is in /usr/hdp/current/spark-client/conf.
  If your cluster runs Spark 2.X, spark-defaults.conf is in /usr/hdp/current/spark2-client/conf.
  This example uses default locations for Spark JARs. Your environment may vary.
```
spark.unravel.server.hostport=unravel-host:4043
spark.driver.extraJavaOptions=-javaagent:/usr/local/unravel-jars/jars/btrace-agent.jar=config=driver,libs=<spark-version>
spark.executor.extraJavaOptions=-javaagent:/usr/local/unravel-jars/jars/btrace-agent.jar=config=executor,libs=<spark-version>
spark.eventLog.enabled=true 
```
  For example:
```
spark.unravel.server.hostport=xyznode.unraveldata.com:4043
spark.driver.extraJavaOptions=-javaagent:/usr/local/unravel-jars/jars/btrace-agent.jar=config=driver,libs=spark-2.3
spark.executor.extraJavaOptions=-javaagent:/usr/local/unravel-jars/jars/btrace-agent.jar=config=executor,libs=spark-2.3
spark.eventLog.enabled=true 
```
  Note
  If you have multiple Spark services in the same cluster, you must set the Spark default configuration on each.
2. Enable Spark streaming.
Configure Oozie
1. In Ambari, click Oozie >Configs > Advanced .
2. In the Filter box, search for oozie.service.WorkflowAppService.system.libpath and check the path shown.
3. From a terminal application on the Unravel node, pick up the ShareLib Root Directory directory path with the latest timestamp.
```
hdfs dfs -ls <path to ShareLib directory>
```
  For example: hdfs dfs -ls /user/oozie/share/lib/
  Important
  The jars must be copied to the lib directory (with the latest timestamp), which is shown in ShareLib Root Directory.
4. From a terminal application, copy the Hive Hook JAR /usr/local/unravel-jars/unravel-hive-<version>-hook.jar and the Btrace JAR, /usr/local/unravel-jars/jars/btrace-agent.jar to the specified path in ShareLib Root Directory.
```
hdfs dfs -copyFromLocal /usr/local/unravel-jars/unravel-hive-<version>-hook.jar /user/oozie/share/lib/<latest timestamp lib directory>/
```
  For example: hdfs dfs -copyFromLocal /usr/local/unravel-jars/unravel-hive-3.1.0-hook.jar /user/oozie/share/lib/lib_20210504054909
```
hdfs dfs -copyFromLocal /usr/local/unravel-jars/jars/btrace-agent.jar /user/oozie/share/lib/<latest timestamp lib directory>/
```
  For example: hdfs dfs -copyFromLocal /usr/local/unravel-jars/jars/btrace-agent.jar /user/oozie/share/lib/lib_20210504054909
  Caution
  Jobs controlled by Oozie 2.3+ fail if you do not copy the Hive Hook and BTrace JARs to the HDFS shared library path.

3. Configure the Unravel Host

Define the Tez and ATS properties using the manager service.

Stop Unravel.

<Unravel installation directory>/unravel/manager stop

Set the properties, shown in the tables below, for TEZ and the Application Timeline Server (ATS) if it requires authentication. Use the manager config properties set command to set the properties.

<Unravel installation directory>/unravel/manager config properties set <property> <key>

For example:

/opt/unravel/manager config properties set yarn.ats.webapp.username user1 /opt/unravel/manager config properties set yarn.ats.webapp.password pa$$w0rD

Tez

Property/Description	Set by user	Unit	Default
com.unraveldata.yarn.timeline-service.webapp.address The HTTP address of the Timeline service web application.	Optional	string (URL)	-
com.unraveldata.yarn.timeline-service.port Timeline service port.		number	8188

Property/Description

Set by user

Unit

Default

com.unraveldata.yarn.timeline-service.webapp.address

The HTTP address of the Timeline service web application.

Optional

string

(URL)

com.unraveldata.yarn.timeline-service.port

Timeline service port.

number

8188

Note

In a multi-cluster environment, you must add these properties to the Edge node.

Application Timeline Server (ATS) requires authentication

Property/Description	Set by user	Unit	Default
yarn.ats.webapp.username Username required for authentication to the Application Timeline Server (if authentication is required).	Optional	string	-
yarn.ats.webapp.password Password required for authentication to the Application Timeline Server (if authentication is required).	Optional	string	-

Property/Description

Set by user

Unit

Default

yarn.ats.webapp.username

Username required for authentication to the Application Timeline Server (if authentication is required).

Optional

string

yarn.ats.webapp.password

Password required for authentication to the Application Timeline Server (if authentication is required).

Optional

string

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start Unravel.

<Unravel installation directory>/unravel/manager start

4. Optional: Confirm that Unravel UI shows Tez data

Run the hive_test_simple.sh script on the HDP cluster or any cloud environment where hive.execution.engine=tez.
```
<Unravel installation directory>/unravel/manager run script hive_test_simple.sh
```
Log into Unravel server and go to the Applications page. Check for Tez jobs.
Unravel UI may take a few seconds to load Tez data.

5. Add more configurations

Refer to Additional configurations.

In this section:

Home

Hortonworks Data Platform (HDP)

Notice

1. Download Unravel

Important

2. Deploy Unravel binaries

Option 1: Deploy Unravel from a tar file

Note

Option 2: Deploy Unravel from an RPM package

Important

3. Install Unravel

Note

Install Unravel with Interactive Precheck

Note

Install Unravel manually

Notice

Note

Tip

Note

Tip

Tip

Tip

4. Enable additional instrumentation for HDP

1. Generate and distribute Unravel's Hive Hook and Spark Sensor JARs

Tip

2. Configure Ambari to work with Unravel

Important

Tip

Note

Note

Tip

Tip

Note

Important

Caution

3. Configure the Unravel Host

Note

4. Optional: Confirm that Unravel UI shows Tez data

5. Add more configurations

Search results