Home

Unravel server reference

This reference covers both on-premises and cloud deployments. Some daemons and properties only apply to on-premises deployments.

Daemons

The Unravel service is composed of many daemons which are summarized in the next table. The single script /etc/init.d/unravel_all.sh can be used with a start or stop or restart argument to control all the daemons on a host, in the correct order. The suffix _N means 1, 2, 3, or 4 separate daemons.

Daemon logs are located in /usr/local/unravel/logs,

  • There are two UI logs: unravel_ngui.log and forever_ngui.log.

  • All other logs named with the following format, unravel_daemon_name.log. For example, the log for unravel_jcw2_1 is unravel_jcw2_1.log.

Daemon Logical Name

Description

unravel_db

bundled database (on a custom port)

unravel_ds

Datastore REST API HTTP server

unravel_ew_N

Event Worker

unravel_hhwe

Hive Hook Worker EMR

unravel_hl

Hitdoc Loader

unravel_host_N

Host monitor

unravel_ja

"Job Analyzer" summarizes jobs

unravel_jcs2

Job Collector Sensor YARN

unravel_jcse2

Job Collector Sensor YARN for EMR

unravel_jcw2_N

Job Collector Sensor Worker YARN

unravel_k

bundled Kafka (on a custom port)

unravel_km

Kafka Monitor

unravel_lr

Log Receiver

unravel_ma_N

Metrics Analyzer

unravel_ngui

aNGular Web UI

unravel_os4

Oozie v4 Sensor

unravel_pw

Partition Worker

unravel_s_N

Elasticsearch

unravel_sw_N

Spark Worker

unravel_tc

bundled TomCat (port 4020), internal REST API

unravel_td

"Tidy Dir" cleans up and archives hdfs directories, db retention cleaner.

unravel_tw

Table Worker

unravel_ud

User Digest (report generator)

unravel_us_N

Universal sensor/Impala

unravel_zk_N

bundled Zookeeper (on a custom port)

Adjustable properties

The file /usr/local/unravel/etc/unravel.properties contains settings that can be preserved during an RPM upgrade. These properties are described in the following table:

Property Type/Property Name/Description

Default Value

General Unravel

com.unraveldata.tmpdir

The base location for Unravel process control files where Unravel's temp files reside.

/srv/unravel/tmp

HDFS

com.unraveldata.hdfs.batch.monitoring.interval.sec

Number of seconds between checks for presence of hive queries and MR logs to load into Unravel for batch visibility; should be between 300 and 1800 (inclusive).

300

com.unraveldata.hdfs.interactive.monitoring.interval.sec

Number of seconds between checks for presence of hive queries and MR logs to load into Unravel for interactive visibility; should be between 5 and 60 (inclusive).

30

JDBC

unravel.jdbc.username

Unravel database user.

Set by user

unravel.jdbc.password

Password for unravel.jdbc.username.

Set by user

unravel.jdbc.url

URL for jdbc, determined by your database.

Example:

jdbc:mysql://127.0.0.1:3306/unravel_mysql_prod

Set by user

Kafka

com.unraveldata.kafka.broker_list

Embedded

127.0.0.1:4091

Oozie

com.unraveldata.oozie.fetch.interval.sec

Controls the rate the Oozie REST server is polled. Seconds between intervals for fetching Oozie workflow status.

120

com.unraveldata.oozie.fetch.num

Number of workflows to pull in each API call.

100

oozie.server.url

The Oozie server URL to be monitored by Unravel. If multiple servers exist, the server URL can be a comma-separated string where each part is IP of 1 Oozie server, e.g., http://ip-10-0-0-110.ec2.internal:11000,http://ip-10-0-0-114.ec2.internal:11000

-

Zookeeper

com.unraveldata.zk.quorum

Embedded Zookeeper ensemble in form host1:port1,host2:port2.

127.0.0.1:4181

Adjustable environment settings

Unravel runs the source command on /usr/local/unravel/etc/unravel.ext.shto allow site-specific environment variables to be set. The following table shows possible choices:

Environment Variable

Description

Default

JAVA_HOME

The standard way to specify the home directory of Oracle Java so that $JAVA_HOME/bin/java is the jvm.

See thecompatibility matrix for compatible JAVA versions.

Should use update-alternatives to make correct Java first choice.

JAVA_EXT_OPTS

Last chance arguments to jvm to override other settings.

Unset

HADOOP_CONF_DIR

The directory containing the hadoop config files core-site.xml , hdfs-site.xml , and mapred-site.xml.

As discovered by running hadoop fs -ls.

UNRAVEL_DATA_DIR

A base directory owned by user unravel. During installation Unravel creates multiple sub directories for holding persistent data (db_data , k_data,, zk_data), and also tmp_data if property com.unraveldata.tmpdir is not set.

/srv/unravel

UNRAVEL_LISTEN_PORT

The Web UI port on the primary or standalone Unravel installation (service unravel_ngui) which listens on 0.0.0.0.

3000

UNRAVEL_LOG_DIR

A destination directory owned by run-as user for log files.

/usr/local/unravel/logs

Adjustable root environment settings

Unravel's init.d script runs the source command on /etc/unravel_ctl to allow site-specific environment variables for the "run-as" and group membership of the daemons that Unravel Server runs:

Environment Variable

Description

Default if not Set

RUN_AS

The /etc/init.d/unravel_* initialization scripts use this to determine which user runs the daemons.

unravel

USE_GROUP

The primary group membership of the user that runs the daemons.

unravel

Directories and files

The following is a cross-reference of notable directories and files used by Unravel Server:

Path/Purpose

Expected Size

Notes

/etc/unravel_ctl

Control file for run-as.

n/a

Must be owned by root for security reasons.

/etc/init.d/unravel_all.sh

Convenience script for Unravel start, restart, status, and stop ; controls daemons in proper order.

n/a

For multi-host installations, run this script on both primary and secondary Unravel host.

/usr/local/unravel

Unravel Server installation directory.

1-2.5GB

This directory is created by installing the Unravel RPM; this is a fixed destination.

/usr/local/unravel/etc/unravel.ext.sh

An optional file for overriding JAVA_HOME or other settings as shown in table above.

n/a

Optional; example syntax:

export JAVA_HOME=/path

/usr/local/unravel/etc/unravel.properties

Site-specific settings for Unravel.

n/a

Keep a "golden" copy of this file; rpm -U upgrades don't overwrite this.

/usr/local/unravel/etc/unravel.version.properties

Version-specific values for Unravel like version number, build timestamp.

n/a

Updated during upgrades; do not modify this reference file in order to preserve traceability

/usr/local/unravel/logs

Logs written by Unravel daemons.

~3.5GB max

Each daemon has a maximum of 100MB of logs, auto-rolled; use a symlink to put on another partition.

/srv/unravel/

Base directory for Unravel Server data kept separately from installation directory; contains messaging data for process coordination, bundled db, Elasticsearch indexes, temporary files.

2-900GB; depending on activity level, retention

This directory or its subdirectories can be a symlink(s) to other volumes for disk io performance reasons to distribute load over multiple volumes. If this is an EBS on AWS, then it must be provisioned for max available IOPS and the Unravel Server must be EBS optimized.