Prerequisites - Multi-cluster (On-prem)

To deploy Unravel, ensure that your environment meets these requirements:

In a multi-cluster deployment of Unravel, you must fulfill the following requirements for the host on the core node and edge node:

Core node
- Accessible to Unravel Admins.
- The server should be dedicated only to Unravel. Must have no other Hadoop service or third-party applications installed.
Edge node
- Be managed by Ambari/Cloudera.
- Must have Hadoop clients pre-installed.
- Must have no other Hadoop service or third-party applications installed.
- Accessible to only Hadoop and Unravel Admins.
- PATH includes the path to the HDFS+Hive+YARN+Spark client/gateway, Hadoop commands, and Hive commands.
- Clock synchronization service (such as NTP) is running and in sync with the cluster.

Database connectivity
Ensure to fulfill the following prerequisites for database connectivity:
- MySQL
  1. Create a mysql directory in /tmp. Provide permissions and make them accessible to the user who installs Unravel.
  2. Download the following tar files to /tmp/mysql directory:
    mysql-5.7.27-linux-glibc2.12-x86_64.tar.gz
    mysql-connector-java-<version>.tar.gz
    For an external MySQL database, add the JDBC connector to /tmp/<MySQL-directory>/<jdbcconnector> directory. This can be either a tar file or a jar file.
- MariaDB
  1. Create a mariadb directory in /tmp. Provide permissions and make them accessible to the user who installs Unravel.
  2. Download the following tar files to /tmp/mariadb directory:
    mariadb-10.4.13-linux-x86_64.tar.gz
    mariadb-java-client-2.6.0.jar
    For external MariaDB, add the JDBC connector to /tmp/<MariaDB-directory>/<jdbcconnector> directory. This can be either a tar file or a jar file.
Unravel managed database service
Ensure to install the following for fulfilling the OS level requirements for Unravel managed database service.
- numactl-libs (for libnuma.so)
- libaio (for libaio.so)

Minimum requirements to install Unravel:

Cores: 8
RAM: 96 GB

The following tables provide estimated sizing details for default data retention and lookback settings. For calculating accurate server requirements for your environment, contact Unravel Support.

The Unravel server/ core node sizing table lists the estimated server requirements in a typical environment.
The Unravel data server sizing table lists the estimated server requirements for a data server.

Tip

In production environments, you can keep the Unravel software and Data directory on separate disks. Unravel recommends putting data on high bandwidth/low latency networks.

Table 3. Unravel server/ core node sizing

YARN jobs per day	Impala jobs per day	vCores	RAM	I/O	Software	Retention		Storage
YARN jobs per day	Impala jobs per day	vCores	RAM	I/O	Software	ES	MySQL	ES	MySQL
Less than 50,000	Less than 50,000	1 x 8 vCores	96 GB	15K IOPS	8 GB free	6 months	6 months	0.5 TB	1.5 TB
100,000	100,000	1 x 16 vCores	128 GB	15K IOPS	8 GB free	6 months	30 days	1 TB	3 TB
200,000	200,000	1 x 24 vCores	192 GB	15K IOPS	8 GB free	6 months	30 days	2 TB	6 TB
400,000	400,000	1 x 56 vCores	256 GB	15K IOPS	8 GB free	6 months	30 days	4 TB	10 TB

Table 4. MySQL server sizing table

YARN jobs per day	Impala jobs per day	vCores	RAM	I/O	Retention	Storage
YARN jobs per day	Impala jobs per day	vCores	RAM	I/O	Retention	Less than 50,000	Less than 50,000	16 vCores	96 GB	10K IOPS	30 days	1.5 TB
100,000	100,000	24 vCores	128 GB	10K IOPS	30 days	3 TB
200,000	200,000	24 vCores	192 GB	10K IOPS	30 days	6 TB
400,000	400,000	36 vCores	256 GB	10K IOPS	30 days	10 TB

Notice

If specific features such as Kafka and HBase monitoring are enabled, the memory requirements will increase and vary from the above table.

Architecture: x86_64
vm.max_map_count is set to 262144
nproc limit is set to unlimited.
Add unravel.conf file with the following settings to /etc/security/limits.d:
```
unravel    soft  nproc unlimited
```

Core node
- Create an Installation directory and grant ownership of the directory to the user who installs Unravel. This user executes all the processes involved in running Unravel.
- If you are using Kerberos, you must create a principal and keytab for Unravel daemons to use.
- Unravel needs access to the YARN Resource Manager's REST API.
- Unravel needs read-only access to the database used by the Hive metastore.
- Unravel users should have read-only access to hive server2.
- URL and credentials of the Cluster Manager (Cloudera/Ambari).

Edge node
- Create an Installation directory and grant ownership of the directory to the user who installs Unravel. This user executes all the processes involved in Unravel installation.
- If you are using Kerberos, you must create a principal and keytab for Unravel daemons to use.
- URL and credentials of the Cluster Manager (Cloudera/Ambari).
- Unravel must have read access to these HDFS resources:
  - MapReduce logs (hdfs://user/history)
  - YARN's log aggregation directory (hdfs://tmp/logs)
  - Spark and Spark2 event logs (hdfs://user/spark/applicationHistory and hdfs://user/spark/spark2ApplicationHistory)
  - File and partition sizes in the Hive warehouse directory (typically hdfs://apps/hive/warehouse)
- Unravel needs access to the YARN Resource Manager's REST API.
- Unravel needs read-only access to the database used by the Hive metastore.
- Unravel users should have read-only access to hive server2.
- Unravel users should keep the cluster access ID handy.
- If you plan to use Unravel's move or kill AutoActions, the Unravel username needs to be added to YARN's yarn.admin.acl property.
- If you're using Impala, Unravel needs access to the Cloudera Manager API. Read-only access is sufficient.

Note

You can customize all the Unravel ports. Refer to Configuring custom ports.

On the new node, open the following ports.

Port(s)	Direction	Description
3000	Both	Traffic to and from Unravel UI
3316	Both	Database traffic. This can be MySQL or PostgreSQL database traffic. For MySQL, the default port is 3306. For PostgreSQL, the default port is 5432.
4020	Both	Unravel APIs
4021	Both	Host monitoring of JMX on `localhost`
4043	In	UDP and TCP ingest traffic from the entire cluster to Unravel Server(s)
4044-4049	In	UDP and TCP ingest spares for `unravel_lr*`
4091-4099	Both	Kafka brokers
4171-4174, 4176-4179	Both	ElasticSearch; localhost communication between Unravel daemons or Unravel Servers in a multi-host deployment
4181-4189	Both	Zookeeper daemons
4210	Both	Cluster access service
5432		Database traffic. The default port for PostgreSQL.
HDFS ports	Both	Traffic to/from the cluster to Unravel Server(s)
Hive metastore port	Out	For YARN only. Traffic from Hive to Unravel Server(s) for partition reporting.
8088	Out	Traffic from Unravel Server(s) to the Resource Manager API
11000	Out	For Oozie only. Traffic from Unravel Server(s) to the Oozie server

For HDFS, you must provide access to the NameNode and DataNode. The default value for NameNode is 8020 , and that of DataNode is 9866 and 9867. However, these can be configured to any other ports.

Services	Default port	Direction	Description
NameNode	8020	Both	Traffic to/from the cluster to Unravel servers.
DataNode	9866,9867	Both	Traffic to/from the cluster to Unravel servers.

CDH specific port requirements

Port(s)	Direction	Description
3000	Both	Traffic to and from Unravel UI If you plan to use Cloudera Manager to install Unravel's sensors, the Cloudera Manager service must also be able to reach the Unravel host on port 3000.
7180 (or 7183 for HTTPS)	Out	Traffic from Unravel Server(s) to Cloudera Manager

Port(s)

Direction

Description

3000

Both

Traffic to and from Unravel UI

If you plan to use Cloudera Manager to install Unravel's sensors, the Cloudera Manager service must also be able to reach the Unravel host on port 3000.

7180 (or 7183 for HTTPS)

Out

Traffic from Unravel Server(s) to Cloudera Manager

In this section:

Home