Installing Unravel on HDP
This topic explains how to deploy Unravel on Hortons Data Platform (HDP).
Important
If you have not already done so, confirm your new node meets Unravel's hosting requirements.
1. Create a new node on your cluster
Using Ambari, add a new node (host) to your cluster. This new node will be Unravel Server's host machine.
Verify that Ambari installed the following clients on the new host:
Atlas Metadata
HCat
HDFS
Hive
Kerberos
MapReduce2
Oozie
Pig
Slider
Spark
Spark2
Tez
YARN
ZooKeeper
4. Check Kerberos settings.
If the cluster is kerberized, check Kerberos settings with the kinit and klist commands:
klist -kt /etc/security/keytabs/unravel.keytab Keytab name: FILE:/etc/security/keytabs/unravel.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 3 03/08/2019 15:20:13 unravel/abcd25.unraveldata.com@lab.localdomain
kinit -kt /etc/security/keytabs/unravel.keytab unravel/abcd25.unraveldata.com@lab.localdomain
klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: unravel/abcd25.unraveldata.com@lab.localdomain Valid starting Expires Service principal 03/14/2019 16:02:49 03/15/2019 16:02:49 krbtgt/lab.localdomain@lab.localdomain # groups unravelunravel : unravel hadoop # kvno unravel/abcd25.unraveldata.com@lab.localdomainunravel/abcd25.unraveldata.com@lab.localdomain: kvno = 3
Check the
unravel
user's network access.If the cluster is kerberized, make sure Unravel can access
http://timeline-host:8188/ws/v1/timeline
. Thecurl
command below requires a successful kinit command for theunravel
user. The-u
option is a fake user and is ignored when relying on GSS-API.curl --negotiate -v -u :-X GET http://timeline-host:8188/ws/v1/timeline ... > Authorization: Negotiate
token
> User-Agent: curl/7.29.0 > Host: congo52.unraveldata.com:8188 > Accept: */* > < HTTP/1.1 200 OK ...
6. (Optional) Enable additional daemons for high-volume workloads.
7. Configure Unravel Server with basic options
Edit the Unravel properties through automatic or manual configuration. Following is a reference list of the basic properties that you can configure for HDP installation.
If Kerberos is enabled, do the following:
Create or identify a principal and keytab for Unravel to use to access the HDFS resources listed in the table below and the REST API.
Use the manager tool to access the configuration UI and run the following commands to set these properties:
manager config kerberos set --keytab /path/to/keytab --principal
<service@REALM>
manager config kerberos enableAlso, refer to Kerberos configurations.
If Ranger is enabled, do the following:
Check/set Ranger permissions for the runtime user.
Create or identify a principal and keytab for Unravel daemons to access the HDFS resources listed in the table below.
If you are using a virus scanner,
It is recommended to disable your virus scanner from scanning the elasticsearch directories which are located under
Unravel installation directory/data
.In the Ambari dashboard, set YARN ACLs.
Select YARN | Configs | Advanced.
Set yarn.acl.enable to
true
.Add the Unravel user specified in com.unraveldata.kerberos.principal to yarn.admin.acl.
Save your changes.
8. Set hive metastore password
Use the manager tool to access the configuration UI and set the hive metastore password manually:
manager config properties set javax.jdo.option.ConnectionPassword --<encrypted>
9. Verify properties
Log in to the Ambari web UI and verify if the values of the following properties match in the configuration.
Unravel Property | Cluster Property |
---|---|
com.unraveldata.job.collector.done.log.base | mapreduce.jobhistory.done-dir |
com.unraveldata.job.collector.done.log.aggregation.base | yarn.nodemanager.remote-app-log-dir |
javax.jdo.option.ConnectionURL | jdbc:mysql:// |
javax.jdo.option.ConnectionDriverName | JDBC Driver Class |
javax.jdo.option.ConnectionUserName | Database Username |
10. Start Unravel
Run the following command to start Unravel:
<Unravel installation directory>/manager start
This completes the basic/core configuration.
11. Log into Unravel UI
Using a supported web browser (see Unravel's HDP compatibility matrix), navigate to http://
and log in with username unravel-host
:3000admin
with password unraveldata
.