Configuring external hive metastore (EMR)
You can configure multiple EMR clusters' hive metastore for Unravel monitoring.
To configure the hive metastore, do the following:
Create a
.txt
or.prop
file containing the following properties and their corresponding values in any location that is accessible to Unravel users. Following is a sample of the file:com.unraveldata.hive.metastore.list=
<EMR-cluster1name>
,<EMR-cluster2name>
hive.metastore.cluster.ids=default hive.metastore.<EMR_cluster1name>
.cluster.ids=<EMR_cluster1name>
javax.jdo.option.<EMR-cluster1name>.ConnectionURL=<jdbc connection URL>
javax.jdo.option.<EMR-cluster1name>.ConnectionDriverName=<jdbc driver name>
javax.jdo.option.<EMR-cluster1name>.ConnectionPassword=<jdbc connection password>
javax.jdo.option.<EMR-cluster1name>.ConnectionUserName=<jdbc connection username>
hive.metastore.<EMR_cluster2name>
.cluster.ids=<EMR_cluster2name>
javax.jdo.option.<EMR-cluster2name>.ConnectionURL=<jdbc connection URL>
javax.jdo.option.<EMR-cluster2name>.ConnectionDriverName=<jdbc driver name>
javax.jdo.option.<EMR-cluster2name>.ConnectionPassword=<jdbc connection password>
javax.jdo.option.<EMR-cluster2name>.ConnectionUserName=<jdbc connection username>
Properties
Description
EMR cluster name
Name of the EMR cluster.
JDBC connection URL
The JDBC connection string for the hive metastore containing the metadata of the form: jdbc:
DB_Driver
://HOST
:PORT
/hiveJDBC driver name
JDBC Driver class name for the hive metastore containing the metadata
JDBC connection password
Password used to access the hive metastore.
JDBC connection username
The username used to access the hive metastore.
For example:
com.unraveldata.hive.metastore.list=data-analysis,data-science,al-ml-cluster-emr-version-1-5,al-ml-EMR-version-2-0 hive.metastore.cluster.ids=default hive.metastore.al-ml-cluster-emr-version-1-5.cluster.ids=al-ml-cluster-emr-version-1-5. hive.metastore.data-analysis.cluster.ids=data-analysis javax.jdo.option.al-ml-cluster-emr-version-1-5.ConnectionDriverName=com.mysql.jdbc.Driver javax.jdo.option.al-ml-cluster-emr-version-1-5.ConnectionPassword=hive-password javax.jdo.option.al-ml-cluster-emr-version-1-5.ConnectionURL=jdbc:mysql://al-ml-cluster-emr-version-1-5-m/metastore javax.jdo.option.al-ml-cluster-emr-version-1-5.ConnectionUserName=hive javax.jdo.option.al-ml-emr-version-2-0.ConnectionDriverName=com.mysql.jdbc.Driver javax.jdo.option.al-ml-emr-version-2-0.ConnectionPassword=hive-password javax.jdo.option.al-ml-emr-version-2-0.ConnectionURL=jdbc:mysql://al-ml-emr-version-2-0-m/metastore javax.jdo.option.al-ml-emr-version-2-0.ConnectionUserName=hive javax.jdo.option.data-analysis.ConnectionDriverName=com.mysql.jdbc.Driver javax.jdo.option.data-analysis.ConnectionPassword=hive-password javax.jdo.option.data-analysis.ConnectionURL=jdbc:mysql://data-analysis-m/metastore javax.jdo.option.data-analysis.ConnectionUserName=hive javax.jdo.option.data-science.ConnectionDriverName=com.mysql.jdbc.Driver javax.jdo.option.data-science.ConnectionPassword=hive-password javax.jdo.option.data-science.ConnectionURL=jdbc:mysql://data-science-m/metastore javax.jdo.option.data-science.ConnectionUserName=hive
From the installation directory, where Unravel binaries are installed, run the following command and provide the path to the properties file.
<Unravel installation directory>/unravel/manager config properties import <path to the properties file> For example: /opt/unravel-install/unravel/manager config properties import /opt/properties.txt
Ensure to stop Unravel.
<Unravel installation directory>/unravel/manager stop
Apply the changes.
<Unravel installation directory>/unravel/manager config apply
Start Unravel.
<Unravel installation directory>/unravel/manager start