Skip to main content

Home

Configuring external hive metastore (EMR)

You can configure multiple EMR clusters' hive metastore for Unravel monitoring.

Copy database jar for hive metastore configuration

Ensure that the database jar file for MySQL, MariaDB, or Oracle DB is copied to the following location before you configure Hive metastore. 

<Unravel installation directory>/unravel/share/java/

For example:

/opt/unravel/share/java/

To configure the hive metastore, do the following:

  1. Create a .txt or .prop file containing the following properties and their corresponding values in any location accessible to Unravel users. Following is a sample of the file:

    com.unraveldata.hive.metastore.list=<EMR-cluster1name>,<EMR-cluster2name> 
hive.metastore.cluster.ids=default
hive.metastore.<EMR_cluster1name>.cluster.ids=<EMR_ClusterId>
javax.jdo.option.<EMR-cluster1name>.ConnectionURL=<jdbc connection URL>
javax.jdo.option.<EMR-cluster1name>.ConnectionDriverName=<jdbc driver name>
javax.jdo.option.<EMR-cluster1name>.ConnectionPassword=<jdbc connection password>
javax.jdo.option.<EMR-cluster1name>.ConnectionUserName=<jdbc connection username>
hive.metastore.<EMR_cluster2name>.cluster.ids=<EMR_cluster2name>
javax.jdo.option.<EMR-cluster2name>.ConnectionURL=<jdbc connection URL>
javax.jdo.option.<EMR-cluster2name>.ConnectionDriverName=<jdbc driver name>
javax.jdo.option.<EMR-cluster2name>.ConnectionPassword=<jdbc connection password>
javax.jdo.option.<EMR-cluster2name>.ConnectionUserName=<jdbc connection username>

    Properties

    Description

    EMR cluster name

    Name of the EMR cluster. For example, EMR-jsmith

    You can retrieve this value either from the Unravel UI or from AWS console > EMR > (Your EMR Cluster > Summary tab).

    EMR cluster ID

    An ID of the EMR cluster. For example, j-1JKWNI4WFD432.

    You can retrieve this value either from Unravel UI or from AWS console > EMR > (Your EMR Cluster > Summary tab).

    JDBC connection URL

    The JDBC connection string for the hive metastore containing the metadata of the form: jdbc:DB_Driver://HOST:PORT/hive

    You can retrieve this value from hive.conf of the master node of the EMR cluster or clusters.

    JDBC driver name

    JDBC Driver class name for the hive metastore containing the metadata.

    You can retrieve this value from hive.conf of the master node of the EMR cluster or clusters.

    JDBC connection password

    Password used to access the hive metastore.

    You can retrieve this value from hive.conf of the master node of the EMR cluster or clusters.

    JDBC connection username

    The username used to access the hive metastore.

    You can retrieve this value from hive.conf of the master node of the EMR cluster or clusters.

    For example:

    com.unraveldata.hive.metastore.list=EMR-jsmith,EMR-jscott,data-analysis,data-science
hive.metastore.cluster.ids=default
hive.metastore.EMR-jsmith.cluster.ids=j-1JKWNI4WFD432
javax.jdo.option.EMR-jsmith.ConnectionDriverName=com.mysql.jdbc.Driver
javax.jdo.option.EMR-jsmith.ConnectionPassword=hive-password
javax.jdo.option.EMR-jsmith.ConnectionURL=jdbc:mysql://al-ml-cluster-emr-version-1-5-m/metastore
javax.jdo.option.EMR-jsmith.ConnectionUserName=hive
javax.jdo.option.EMR-jscott.ConnectionDriverName=com.mysql.jdbc.Driver
javax.jdo.option.EMR-jscott.ConnectionPassword=hive-password
javax.jdo.option.EMR-jscott.ConnectionURL=jdbc:mysql://al-ml-emr-version-2-0-m/metastore
javax.jdo.option.EMR-jscott.ConnectionUserName=hive
javax.jdo.option.data-analysis.ConnectionDriverName=com.mysql.jdbc.Driver
javax.jdo.option.data-analysis.ConnectionPassword=hive-password
javax.jdo.option.data-analysis.ConnectionURL=jdbc:mysql://data-analysis-m/metastore
javax.jdo.option.data-analysis.ConnectionUserName=hive
javax.jdo.option.data-science.ConnectionDriverName=com.mysql.jdbc.Driver
javax.jdo.option.data-science.ConnectionPassword=hive-password
javax.jdo.option.data-science.ConnectionURL=jdbc:mysql://data-science-m/metastore
javax.jdo.option.data-science.ConnectionUserName=hive

  2. From the installation directory, where Unravel binaries are installed, run the following command and provide the path to the properties file.

    <Unravel installation directory>/unravel/manager config properties import <path to the properties file>
For example:
/opt/unravel-install/unravel/manager config properties import /opt/properties.txt

  3. Ensure to stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  4. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply

  5. Start Unravel.

    <Unravel installation directory>/unravel/manager start
In this section: