MapR

Before installing, ensure to check and complete the installation requirements. Follow the below instructions to download, install, and set up Unravel for the MapR platform.

Notice

The following instructions are for a single cluster environment. For installing Unravel on a multi-cluster environment, refer to Multi-cluster install.

1. Download Unravel

2. Deploy Unravel binaries

Unravel binaries are available as a tar file or RPM package. You can deploy the Unravel binaries in any directory on the server. However, the user who installs Unravel must have write permissions to the directory where the Unravel binaries are deployed.

After the Unravel binaries are deployed, the directory layout for both the Tar and RPM will be unravel/versions/<Directories and files>. The binaries are deployed to <Unravel_installation_directory> and Unravel will be available in <Unravel_installation_directory>/unravel.

Option 1: Deploy Unravel from a tar file

The following steps to deploy Unravel from a tar file should be performed by a user, who will run Unravel.

Create an Installation directory.
```
mkdir /path/to/installation/directory
## For example: mkdir /opt/unravel
```
Note
Some locations may require root access to create a directory. In such a case, after the directory is created, change the ownership to unravel user and continue with the installation procedure as the unravel user.
```
chown -R username:groupname /path/to/installation/directory
## For example: chown -R unravel:unravelgroup /opt/unravel
```
Extract and copy the Unravel tar file to the installation directory, which was created in the first step. After you extract the contents of the tar file, unravel directory is created within the installation directory.
```
tar zxf unravel-<version>tar.gz -C /path/to/installation/directory
## For example: tar zxf unravel-4.7.2.3.tar.gz -C /opt
## The unravel directory will be available within /opt
```

Option 2: Deploy Unravel from an RPM package

Important

The following steps, to deploy Unravel from an RPM package, should be performed by a root user. After the RPM package is deployed, the remaining installation procedures should be performed by the unravel user.

Create an installation directory.
```
mkdir /usr/local/unravel
```

Run the following command:

rpm -i unravel-<version>.rpm
## For example: rpm -i unravel-4.7.2.3.rpm 
## The unravel directory will be available in /usr/local

In case you want to provide a different location, you can do so by using the --prefix command. For example:

mkdir /opt/unravel
chown -R username:groupname /opt/unravel
rpm -i unravel-4.7.2.3.rpm --prefix /opt

## The unravel directory will be available in /opt

Grant ownership of the directory to a user who will run Unravel. This user executes all the processes involved in Unravel installation.
```
chown -R username:groupname /usr/local/unravel
## For example: chown -R unravel:unravelgroup /usr/local/unravel
```
Continue with the installation procedures as unravel user.

3. Run the setup

You can run the setup command to install Unravel. The setup command allows you to do the following:

Runs Precheck automatically to detect possible issues that prevent a successful installation. Suggestions are provided to resolve issues. Refer to Precheck filters for the expected value for each filter.
Let you run extra parameters to integrate the database of your choice.
The setup command allows you to use a managed database shipped with Unravel or an external database. The setup uses the Unravel managed PostgreSQL database when run without any additional parameters. Otherwise, you can specify one of the following types of databases in the setup command:
- MySQL (Unravel managed as well as external MySQL database)
- MariaDB (Unravel managed as well as external MariaDB database)
- PostgreSQL (External PostgreSQL)
Refer to Integrate database for details.
Let you specify a separate path for the data directory other than the default path.
You can locate the Unravel data and configurations in the data directory. By default, the installer maintains the data directory under <Unravel installation directory>/data. You can also change the data directory's default location by running additional parameters with the setup command. To install Unravel with the setup command.
Provides more setup options.

Notice

The Unravel user who owns the installation directory should run the setup command to install Unravel.

To install Unravel with the setup command, do the following:

Switch to Unravel user.
```
  su - <unravel user>
```
Run setup command:
Note
Refer to setup Options for all the additional parameters that you can run with the setup command
Refer to Integrate database topic and complete the pre-requisites before running the setup command with any other database other than Unravel managed PostgreSQL, which is shipped with the product. Extra parameters must be passed with the setup command when using another database.
Tip
Optionally, if you want to provide a different data directory, you can pass an extra parameter (--data-directory) with the setup command as follows:
```
<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --data-directory /the/data/directory
```
Similarly, you can configure separate directories for other unravel directories —contact support for assistance.
- PostgreSQL
  - Unravel managed PostgreSQL
    <unravel_installation_directory>/unravel/versions/<Unravel version>/setup
  - External PostgreSQL
    <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --external-database postgresql <HOST> <PORT> <SCHEMA> <USERNAME> <PASSWORD>
    The HOST, PORT, SCHEMA, USERNAME, and PASSWORD are optional fields and are prompted if missing. For example: /opt/unravel/versions/abcd.992/setup --external-database postgresql xyz.unraveldata.com 5432 unravel_db_prod unravel unraveldata
- MySQL
  - Unravel managed MySQL
    <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --extra /tmp/mysql
  - External MySQL
    <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --extra /tmp/<MySQL-directory> --external-database mysql <HOST> <PORT> <SCHEMA> <USERNAME> <PASSWORD>
    The HOST, PORT, SCHEMA, USERNAME, and PASSWORD are optional fields and are prompted if missing.
- MariaDB
  - Unravel managed MariaDB
    <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --extra /tmp/mariadb
  - External MariaDB
    <unravel_installation_directory>unravel/versions/<Unravel version>/setup --extra /tmp/<MariaDB-directory> --external-database mariadb <HOST> <PORT> <SCHEMA> <USERNAME> <PASSWORD>
    The HOST, PORT, SCHEMA, USERNAME, and PASSWORD are optional fields and are prompted if missing.
Precheck is automatically run when you run the setup command. Refer to Precheck filters for the expected value for each filter.

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start all the services.

<unravel_installation_directory>/unravel/manager start

Check the status of services.
```
<unravel_installation_directory>/unravel/manager report 
```
The following service statuses are reported:
- OK: Service is up and running.
- Not Monitored: Service is not running. (Has stopped or has failed to start)
- Initializing: Services are starting up.
- Does not exist: The process unexpectedly disappeared. A restart will be attempted ten times.
You can also get the status and information for a specific service. Run the manager report command as follows:
```
<unravel_installation_directory>/unravel/manager report <service>
```
For example: /opt/unravel/manager report auto_action

Precheck

The Precheck output displays the issues that prevent a successful installation and provides suggestions to resolve them. You must resolve each of the issues before proceeding. See Precheck filters.

After resolving the precheck issues, you must re-login or reload the shell to execute the setup command again.

Note

You can skip the precheck using the setup --skip-precheck command in certain situations.

For example:

/opt/unravel/versions/<Unravel version>/setup --skip-precheck

You can also skip the checks that you know can fail. For example, if you want to skip the Check limits option and the Disk freespace option, pick the command within the parenthesis corresponding to these failed options and run the setup command as follows:

setup --filter-precheck ~check_limits,~check_freespace

Tip

Run --help with the setup command and any combination of the setup command for complete usage details.

<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --help

Precheck filters

Filters	Description	Expected Value
System
Check uptime	Verifies the period since the last server reboot.	>24h
Clock sync	Verifies if the clock synchronization service is running on the server.	The clock synchronization service is up and running.
CPU requirement	Verifies if the server has enough CPUs to run Unravel efficiently.	Check requirements.
Memory requirement	Verifies that the server has enough memory to run Unravel efficiently.	Check requirements.
Disk access	Verifies that the user who runs unravel has access to the configured disk locations.	Unravel users can access the configured disk locations.
Disk Freespace	Verifies if the disk locations have enough free space.	Check requirements
Kerberos tools	Verifies that the Kerberos tools are available on the server to support kerberized environments.	Kerberos tools are installed.
Network ports	Verifies that the network ports used by Unravel are available.	Check requirements
OS libraries	Verifies the required libraries if run with Unravel managed MySQL.	You must install the following packages for fulfilling the OS level requirements for MySQL: `numactl-libs` (for libnuma.so) `libaio` (for libaio.so)
OS release	Verifies that the OS distribution is supported.	Check compatibility matrix
OS settings	Verifies vm.max_map_count recommended.	Check requirements
SELinux	Verifies if the SELinux status is enabled or not and provides the mode (Permissive, Disabled, Enforcing).	Check product documentation.
Check limits	Verifies that user limits are set to values	Check requirements
Hadoop
Clients	Ensure that the following Hadoop clients are installed and configured on the server: Apache Hadoop Hadoop Distributed File System (HDFS) Apache Hadoop Yarn Apache Hive Apache Beeline You can ignore the following Precheck limitations on MapR: The Hadoop client check reports missing clients (HDFS and beeline) Any check that depends on the HDFS client (for example, HDFS Access check) reports the following message: `HADH0070: hdfs client is not available` Ensure that Unravel has access to files in the MapR file system or needs to provide access manually.	Check compatibility matrix
Distribution	Verifies that the Hadoop distribution is a supported version.	Check compatibility matrix
RM HA Enabled/Disabled	Verifies if RM is running in HA mode
Healthcheck report bundle	healthcheck report tarball. This report provides the summary and information gathered by the healthcheck with the location.

Setup Options

setup Options	Description
-h, --help	Shows help for setup.
--config CONFIG	Specify a different path to the configuration file. <unravel_installation_directory>/unravel/versions/`<Unravel version>`/setup --config `path/to/config/directory`
--enable-core	Enables core node support for non-Hadoop clusters.
--cluster-access (Edge node parameter)	Enables cluster access to the core node in a multi-cluster environment.
--data-forwarder host:port cluster-type cluster-id	Data forwarder, main unravel node.
--data-directory	Specify a different path to the data directory.
--external-database [param [param ...]]	Enable external database.
--external-database-ssl	Enable external database with SSL.
--log-file	Setup log file location. Default is `/tmp/unravel-setup-YYYYMMDD-HHMMSS.log`.
--extra DIR, -e DIR	Specify extra packages location.
--precheck	Run the preinstallation check.

Precheck Sample

/opt/unravel/versions/abcd.1004/setup 
2021-04-05 15:51:30 Sending logs to: /tmp/unravel-setup-20210405-155130.log
2021-04-05 15:51:30 Running preinstallation check...
2021-04-05 15:51:31 Gathering information ................. Ok
2021-04-05 15:51:51 Running checks .................. Ok
--------------------------------------------------------------------------------
system
 Check limits        : PASSED
 Clock sync          : PASSED
 CPU requirement     : PASSED, Available cores: 8 cores
 Disk access         : PASSED, /opt/unravel/versions/develop.1004/healthcheck/healthcheck/plugins/system is writable
 Disk freespace      : PASSED, 229 GB of free disk space is available for precheck dir.
 Kerberos tools      : PASSED
 Memory requirement  : PASSED, Available memory: 79 GB
 Network ports       : PASSED
 OS libraries        : PASSED
 OS release          : PASSED, OS release version: centos 7.6
 OS settings         : PASSED
 SELinux             : PASSED
--------------------------------------------------------------------------------
Healthcheck report bundle: /tmp/healthcheck-20210405155130-xyz.unraveldata.com.tar.gz
2021-04-05 15:51:53 Prepare to install with: /opt/unravel/versions/abcd.1004/installer/installer/../installer/conf/presets/default.yaml
2021-04-05 15:51:57 Sending logs to: /opt/unravel/logs/setup.log
2021-04-05 15:51:57 Instantiating templates ................................................................................................................................................................................................................................ Ok
2021-04-05 15:52:05 Creating parcels .................................... Ok
2021-04-05 15:52:20 Installing sensors file ............................ Ok
2021-04-05 15:52:20 Installing pgsql connector ... Ok
2021-04-05 15:52:22 Starting service monitor ... Ok
2021-04-05 15:52:27 Request start for elasticsearch_1 .... Ok
2021-04-05 15:52:27 Waiting for elasticsearch_1 for 120 sec ......... Ok
2021-04-05 15:52:35 Request start for zookeeper .... Ok
2021-04-05 15:52:35 Request start for kafka .... Ok
2021-04-05 15:52:35 Waiting for kafka for 120 sec ...... Ok
2021-04-05 15:52:37 Waiting for kafka to be alive for 120 sec ..... Ok
2021-04-05 15:52:42 Initializing pgsql ... Ok
2021-04-05 15:52:46 Request start for pgsql .... Ok
2021-04-05 15:52:46 Waiting for pgsql for 120 sec ..... Ok
2021-04-05 15:52:47 Creating database schema ................. Ok
2021-04-05 15:52:50 Generating hashes .... Ok
2021-04-05 15:52:52 Loading elasticsearch templates ............ Ok
2021-04-05 15:52:55 Creating kafka topics .................... Ok
2021-04-05 15:53:36 Creating schema objects ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... Ok
2021-04-05 15:54:03 Request stop ....................................................... Ok
2021-04-05 15:54:16 Done
[unravel@xyz ~]$

4. Configuring Unravel for MapR

You must manually configure Unravel properties after installing Unravel. Do the following to manually configure the required set of properties.

Create a file containing the required properties for MapR installation and their corresponding values. Here is a sample file with the mandatory properties for MapR installation. Ensure to replace the highlighted items with the corresponding values.

# site-specific unravel properties
com.unraveldata.is_mapr=true
fs.defaultFS=maprfs://
hadoop.conf.dir=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop
com.unraveldata.job.collector.done.log.base=/var/mapr/cluster/yarn/rm/staging/history/done
com.unraveldata.spark.eventlog.location=maprfs:///apps/spark
com.unraveldata.mr.parse.input.splits=false
com.unraveldata.cluster.name=my.cluster.com
com.unraveldata.job.collector.log.aggregation.base=/tmp/logs/*/logs/

# Hive metastore properties
javax.jdo.option.ConnectionURL=jdbc:mysql://<hive_metastore_host>:<hive_metastore_port>/hive?createDatabaseIfNotExist=true&amp;useLegacyDatetimeCode=false&amp;serverTimezone=<server_time_zone>
javax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver
javax.jdo.option.ConnectionUserName=<hive_metastore_user>
javax.jdo.option.ConnectionPassword=<hive_metastore_password>

# Resource Manager (RM) properties. Enables https access to Resource Manager.
https.protocols=TLSv1.2
yarn.resourcemanager.webapp.address=https://<yarn_resource_manager_host>:<yarn_resource_manager_port>
yarn.resourcemanager.webapp.username=<yarn_resource_manager_user>
yarn.resourcemanager.webapp.password=<yarn_resource_manager_password>
     
# oozie properties
oozie.server.url=http://example.localdomain:11000/oozie

The following tables provide descriptions and details about all the properties that can be set for MapR.

Property/Description	Set by user	Unit	Default
com.unraveldata.is_mapr Indicate if the cluster is of MapR. The value must be set to true.	Required	boolean	false
fs.defaultFS Specifies the default file system and defaults to your local file system. The value must be set to `maprfs` For example: fs.defaultFS=maprfs://	Required	string	Local file system
hadoop.conf.dir The path to hadoop configuration files. For example: `/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop`	Required	string	-
com.unraveldata.mr.parse.input.splits This must be set to false for MapR.	Required	boolean	true

Note

Make sure the cluster type is set correctly, the default is CDH.

Property/Description	Set by user	Unit	Default
com.unraveldata.cluster.name Cluster to connect to if there are multiple clusters. Required in multi-cluster environments only. Unravel first attempts to match on the cluster ID, and then falls back to matching on the display name. For Ambari, the cluster ID and the display name are equivalent, which is the "cluster_name" attribute from the "/clusters" endpoint, for example, http://HOST:8080/api/v1/clusters/. For Cloudera Manager, the cluster ID is the "name" attribute from the "/clusters" endpoint, for example, http://HOST:7180/api/v17/clusters/.		CSL	default
com.unraveldata.cluster.type Possible values are CDH, HDP, CDP, MAPR, DB, EMR, and HDI Ensure to enter the values in capital case.		set member	CDH

Property/Description

Set by user

Unit

Default

com.unraveldata.cluster.name

Cluster to connect to if there are multiple clusters.

Required in multi-cluster environments only. Unravel first attempts to match on the cluster ID, and then falls back to matching on the display name.

For Ambari, the cluster ID and the display name are equivalent, which is the "cluster_name" attribute from the "/clusters" endpoint, for example, http://HOST:8080/api/v1/clusters/.

For Cloudera Manager, the cluster ID is the "name" attribute from the "/clusters" endpoint, for example, http://HOST:7180/api/v17/clusters/.

CSL

default

com.unraveldata.cluster.type

Possible values are CDH, HDP, CDP, MAPR, DB, EMR, and HDI

Ensure to enter the values in capital case.

set member

CDH

Unravel collects HDFS logs for analysis. Hence, you must set the following properties, to point Unravel server to the logs on HDFS.

Property/Description	Unit	Default
com.unraveldata.job.collector.done.log.base HDFS path to `done` directory of MR logs as per cluster configuration. Don't include the hdfs:// prefix For example: com.unraveldata.job.collector.done.log.base=/mr-history/done.	string	/user/history/done
com.unraveldata.job.collector.log.aggregation.base HDFS path to the aggregated container logs (logs to process). Do not include the hdfs://prefix. The log format defaults to TFile. You can specify multiple logs and log formats (TFile or IndexedFormat.) Example: com.unraveldata.job.collector.log.aggregation.base=TFile:/tmp/logs//logs/,IndexedFormat:/tmp/logs//logs-ifile/.	CSL	/tmp/logs/*/logs/
com.unraveldata.spark.eventlog.location Comma-separated list of HDFS paths to the Spark event logs as per cluster configuration. Each path must include the hdfs:/// prefix. For example: com.unraveldata.spark.eventlog.location=hdfs:///spark1-history/,hdfs:///spark2-history/	CSL	hdfs:///user/spark/applicationHistory/

Property/Description

Set by user

Unit

Default

com.unraveldata.job.collector.done.log.base

HDFS path to done directory of MR logs as per cluster configuration. Don't include the hdfs:// prefix For example: com.unraveldata.job.collector.done.log.base=/mr-history/done.

string

/user/history/done

com.unraveldata.job.collector.log.aggregation.base

HDFS path to the aggregated container logs (logs to process). Do not include the hdfs://prefix. The log format defaults to TFile. You can specify multiple logs and log formats (TFile or IndexedFormat.)

Example: com.unraveldata.job.collector.log.aggregation.base=TFile:/tmp/logs/*/logs/,IndexedFormat:/tmp/logs/*/logs-ifile/.

CSL

/tmp/logs/*/logs/

com.unraveldata.spark.eventlog.location

Comma-separated list of HDFS paths to the Spark event logs as per cluster configuration. Each path must include the hdfs:/// prefix. For example: com.unraveldata.spark.eventlog.location=hdfs:///spark1-history/,hdfs:///spark2-history/

CSL

hdfs:///user/spark/applicationHistory/

Property/Description	Set by user	Unit	Default
javax.jdo.option.ConnectionDriverName JDBC Driver class name for the hive metastore containing the metadata Examples: MySQL:`com.mysql.jdbc.Driver` Oracle:`oracle.jdbc.driver.OracleDriver` Microsoft:`com.microsoft.sqlserver.jdbc.SQLServerDriver`	Required	string	-
javax.jdo.option.ConnectionPassword Password used to access the metastore.	Required	string	-
javax.jdo.option.ConnectionUserName Username used to access the metastore.	Required	string	-
javax.jdo.option.ConnectionURL JDBC connection string for the metastore containing the metadata of the form: jdbc:`DB_Driver`://`HOST`:`PORT`/hive Example: Oracle:`jdbc:oracle:thin:@//prodHost:1521/ORCL` Microsoft:`jdbc:sqlserver://jdbc_url`	Required	string (URL)	-

Property/Description	Set by user	Unit	Default
oozie.server.url The Oozie server URL to be monitored by Unravel. If multiple servers exist, the server URL can be a comma-separated string where each part is IP of 1 Oozie server, e.g., http://ip-10-0-0-110.ec2.internal:11000,http://ip-10-0-0-114.ec2.internal:11000	Required	CSL	-
oozie.server.username	Optional	string	-
oozie.server.password	Optional	string	-
oozie.log.length The maximum number of characters in the Oozie workflow log that Unravel fetches. Any log longer than this is trimmed from the beginning and only last x characters are kept.		bytes	1000000 (1MB)
com.unraveldata.oozie.disable Whether to disable bringing in Oozie workflows into Unravel. The underlying jobs will not be affected.		boolean	false
com.unraveldata.oozie.fetch.num Number of workflows to pull in each API call.		count	100
com.unraveldata.oozie.fetch.interval.sec Controls the rate the Oozie REST server is polled. Seconds between intervals for fetching Oozie workflow status.		s	120
com.unraveldata.oozie.retry.sec		s	600
com.unraveldata.oozie.staleworkflow.interval.sec The frequency of the task.		s	120
com.unraveldata.oozie.staleworkflow.fetch.num The maximum number of stale workflows to update in a single run.		count	100
com.unraveldata.oozie.staleworkflow.fetch.logs Whether to fetch Oozie logs for a stale workflow.		boolean	false
com.unraveldata.oozie.staleworkflow.starttimedelta.min The minimum time interval that determines if a workflow is stale.		min	30
com.unraveldata.oozie.fetch.logs Whether to fetch the Oozie logs for a workflow.		boolean	true

Property/Description	Set by user	Unit	Default
https.protocols Protocol to access the Yarn resource manager	Required	string	-
yarn.resourcemanager.webapp.address The HTTP address of the Yarn resource manager.	Required	URL	-
yarn.resourcemanager.webapp.username Credentials to access Yarn resource manager	Required	string	-
yarn.resourcemanager.webapp.password Credentials to access Yarn resource manager	Required	string	-

From the installation directory, where Unravel binaries are installed, run the following command and provide the path to the properties file.
Note
Ensure that the properties file should be accessible by the user who installs Unravel that is the user should have read permissions for the properties file.
```
<Unravel installation directory>/unravel/manager config properties import <path to the properties file>

#For example:
/opt/unravel/manager config properties import /opt/properties.txt
```
Note
You can set individual properties using the manager utility. Run the following command:
```
<Unravel installation directory>/unravel/manager config properties set <property> <value>

For example:
/opt/unravel/manager config properties set com.unraveldata.cluster.type MAPR
```

Configuring MapR Ticket.

Stop Unravel.

<Unravel installation directory>/unravel/manager stop

Use an editor to open <Installation_directory>/unravel/data/conf/unravel.yaml file.

Find the unravel: block and add the following:

  environment:
    MAPRTICKET_LOCATION: /path/to/MapR_Ticket

For example:

unravel:
... other existing configuration attributes ...
  environment:
    MAPRTICKET_LOCATION: /path/to/MapR_Ticket

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Configure tagging.
1. Stop Unravel
```
<Unravel installation directory>/unravel/manager stop
```
2. Run the following to enable tagging.
```
<Unravel installation directory>/unravel/manager config tagging enable
```
3. Set the tagging method via the python script. Refer to Writing a Python scriptfor more details.
```
manager config tagging set <tagging_method> </path/to/tagging/script>
```
  <tagging method> is the method to call in the tagging script.
4. Apply the changes.
  Note
  Ensure to stop Unravel before applying the changes.
```
<Unravel installation directory>/unravel/manager config apply
```

Start Unravel

<Unravel installation directory>/unravel/manager start

5. Enable additional instrumentation (MapR)

You can use the following steps to enable additional instrumentation for MapR:

Note

After the files are updated on the Unravel host, you can use the scp command to copy them to other hosts in case of a multi-node MapR cluster. Back up your original files in case you need to roll back changes.

Deploy sensors.
1. Download the Hive Hook jar.
  Create /usr/local/unravel_client directory, if it does not exist already, and download the Hive Hook jar to this directory.
```
mkdir -p /usr/local/unravel_client
cd /usr/local/unravel_client
wget <protocol>://<unravel_base_url>:<port>/hh/unravel-hive-<hive_version>-hook.jar 
```
2. Download and extract Spark Sensor zip.
  Create /usr/local/unravel-agent directory, if it does not exist, and enter the directory. Download and extract Spark Sensor zip to this directory.
```
mkdir -p /usr/local/unravel-agent
cd /usr/local/unravel-agent
wget <protocol>://<unravel_base_url>:<port>/hh/unravel-agent-pack-bin.zip
unzip unravel-agent-pack-bin.zip
```
3. (Optional) In the case of a multi-node MapR cluster, you must distribute the sensor to all the hosts using SCP.
  - SCP Hive Hook sensor
    scp -i <ssh_key> -r /usr/local/unravel_client/ ssh_user@host:/usr/local
  - SCP Spark/MR sensor
    scp -i <ssh_key> -r /usr/local/unravel-agent/ ssh_user@host:/usr/local

Update hive-env.sh.

In /opt/mapr/hive/hive-<hive-version>/conf/hive-env.sh, append these lines:

export AUX_CLASSPATH=${AUX_CLASSPATH}:/usr/local/unravel_client/unravel-hive-<hive-version>-hook.jar 
export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}:/usr/local/unravel_client

Configure hive-site.xml

In /opt/mapr/hive/hive-<hive-version>/conf/hive-site.xml, add the following properties:

 </property>
<!--<property>    
<name>com.unraveldata.hive.hook.tcp</name>    
<value>true</value>    
<source>yarn-site.xml</source>  
</property><property>    
<name>hive.exec.failure.hooks</name>    
<value>com.unraveldata.dataflow.hive.hook.UnravelHiveHook</value>    
<source>yarn-site.xml</source>  
</property><property>    
<name>com.unraveldata.hive.hdfs.dir</name>    
<value>/user/unravel/HOOK_RESULT_DIR</value>    
<source>yarn-site.xml</source>  
</property><property>    
<name>hive.exec.driver.run.hooks</name>    
<value>com.unraveldata.dataflow.hive.hook.UnravelHiveHook</value>    
<source>yarn-site.xml</source>  
</property><property>    
<name>hive.exec.post.hooks</name>    
<value>com.unraveldata.dataflow.hive.hook.UnravelHiveHook</value>    
<source>yarn-site.xml</source>  
</property><property>    
<name>com.unraveldata.host</name>    
<value><unravel host name></value>    
<source>yarn-site.xml</source>  
</property>  <property>    
<name>hive.exec.pre.hooks</name>    
<value>com.unraveldata.dataflow.hive.hook.UnravelHiveHook</value>    
<source>yarn-site.xml</source>  
</property>-->

Update spark-defaults.conf.

In /opt/mapr/spark/spark-<spark-version>/conf/spark-defaults.conf, append these lines:

spark.unravel.server.hostport <unravel-host>:4043 spark.eventLog.dir maprfs:///apps/spark 
spark.history.fs.logDirectory maprfs:///apps/spark 
spark.driver.extraJavaOptions -javaagent:/usr/local/unravel-agent/btrace-agent.jar=libs=spark-<spark-version>,config=driver 
spark.executor.extraJavaOptions -javaagent:/usr/local/unravel-agent/btrace-agent.jar=libs=spark-<spark-version>,config=executor

Note

To enable live monitoring of Spark Streaming applications, add the following property in spark.driver.extraJavaOptions

script=StreamingProbe.btclass

#For example:
spark.driver.extraJavaOptions=-javaagent:/usr/local/unravel-agent/btrace-agent.jar=libs=spark-<spark-version>,script=StreamingProbe.btclass,config=driver

Update hadoop-env.sh.
In /opt/mapr/hadoop/hadoop-<hadoop-version>/etc/hadoop/hadoop-env.sh, append these lines:
```
export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/local/unravel_client/unravel-hive-<hive-version>-hook.jar
```

Update mapred-site.xml.

In /opt/mapr/hadoop/hadoop-hadoop-version/etc/hadoop/mapred-site.xml, append these lines:

<property>
	<name>mapreduce.task.profile</name>
	<value>true</value>
</property>
<property>
	<name>mapreduce.task.profile.maps</name>
	<value>0-5</value>
</property>
<property>
	<name>mapreduce.task.profile.reduces</name>
	<value>0-5</value>
</property>
<property>
	<name>mapreduce.task.profile.params</name>
	<value>-javaagent:/usr/local/unravel-agent/btrace-agent.jar=libs=mr -Dunravel.server.hostport=<unravel-host>:4043</value>
</property>
<property>
	<name>yarn.app.mapreduce.am.command-opts</name>
	<value>-javaagent:/usr/local/unravel-agent/btrace-agent.jar=libs=mr -Dunravel.server.hostport=<unravel-host>:4043</value>
</property>

Note

Make sure the original value of yarn.app.mapreduce.am.command-opts is preserved, by appending the Java agent setup rather than replacing the original value.

Update tez-site.xml.

In /opt/mapr/tez/tez-version/conf/tez-site.xml, append these lines:

<property>
  <name>tez.task.launch.cmd-opts</name>
  <value>-javaagent:/usr/local/unravel-agent/btrace-agent.jar=libs=mr,config=tez -Dunravel.server.hostport=<unravel-host>:4043</value>
  <description />
</property>

<property>
  <name>tez.am.launch.cmd-opts</name>
  <value>-javaagent:/usr/local/unravel-agent/btrace-agent.jar=libs=mr,config=tez -Dunravel.server.hostport=<unravel-host>:4043</value>
  <description />
</property>

Confirm and adjust the settings in yarn-site.xml

Check specific properties in /opt/mapr/hadoop/hadoop-<hadoop-version>/etc/hadoop/yarn-site.xml to be sure that these settings are present:

yarn.resourcemanager.webapp.address

<property>
	<name>yarn.resourcemanager.webapp.address</name>
	<value><your-resource-manager-webapp-ip-address>:8088</value>
	<source>yarn-site.xml</source>
</property>

yarn.log-aggregation-enable

<property>
	<name>yarn.log-aggregation-enable</name>
	<value>true</value>
	<description>For log aggregations</description>
</property>

Run the MapR setup script.
1. Run the unravel_mapr_setup.py script.
  Replace the values for unravel-host, spark-version, hive-version, and ambari-host with appropriate values.
```
<Unravel installation directory>/unravel/manager run script unravel_mapr_setup.py --unravel-server <unravel-host>:3000 --spark-version <spark-version> --hive-version <hive-version> -v
```
2. Copy the following to all MapR nodes.
```
/usr/local/unravel_client
```
```
/usr/local/unravel-agent/
```
3. Ensure the following modified configuration files in the Unravel node are applied to other nodes:
  - /opt/mapr/hive/hive-<hive-version>/conf/hive-env.sh
  - /opt/mapr/hive/hive-<hive-version>/conf/hive-site.xml
  - /opt/mapr/spark/spark-<spark-version>/conf/spark-defaults.conf
  - /opt/mapr/hadoop/hadoop-<hadoop-version>/etc/hadoop/hadoop-env.sh
  - /opt/mapr/hadoop/hadoop-<hadoop-version>/etc/hadoop/mapred-site.xml
  - /opt/mapr/tez/tez-version/conf/tez-site.xml
  - /opt/mapr/hadoop/hadoop-<hadoop-version>/etc/hadoop/yarn-site.xml
Go to MapR Control System UI > Services and restart all the services.

6. Log onto Unravel UI

Find Unravel URL from the machine where Unravel is installed.
```
echo "http://$(hostname -f):3000/"
```
If you are using an SSH tunnel or HTTP proxy, you might need to make adjustments.
Using a supported web browser (see Unravel's MapR compatibility matrix, navigate to Unravel URL. For example: http://unravel-host:3000 and log in with username admin with password unraveldata.
Unravel UI displays the collected data.

In this section:

Home

MapR

Notice

1. Download Unravel

Important

2. Deploy Unravel binaries

Option 1: Deploy Unravel from a tar file

Note

Option 2: Deploy Unravel from an RPM package

Important

3. Run the setup

Notice

Note

Tip

Precheck

Note

Tip

Setup Options

Precheck Sample

4. Configuring Unravel for MapR

Note

Note

Note

Note

5. Enable additional instrumentation (MapR)

Note

Note

Note

6. Log onto Unravel UI

Search results