Home

Configuring the Dataproc external hive metastore in Unravel

You can configure multiple Dataproc clusters' hive metastore for Unravel monitoring.

To configure the hive metastore, do the following:

  1. Create a .txt or .prop file containing the following properties and their corresponding values in any location that is accessible to Unravel user. Following is a sample of the file:

    com.unraveldata.hive.metastore.list=<Dataproc-cluster1name>,<Dataproc-cluster2name>
hive.metastore.cluster.ids=default
javax.jdo.option.<Dataproc-cluster1name>.ConnectionURL=<jdbc connection URL>
javax.jdo.option.<Dataproc-cluster1name>.ConnectionDriverName=<jdbc driver name>
javax.jdo.option.<Dataproc-cluster1name>.ConnectionPassword=<jdbc connection password>
javax.jdo.option.<Dataproc-cluster1name>.ConnectionUserName=<jdbc connection username>
javax.jdo.option.<Dataproc-cluster2name>.ConnectionURL=<jdbc connection URL>
javax.jdo.option.<Dataproc-cluster2name>.ConnectionDriverName=<jdbc driver name>
javax.jdo.option.<Dataproc-cluster2name>.ConnectionPassword=<jdbc connection password>
javax.jdo.option.<Dataproc-cluster2name>.ConnectionUserName=<jdbc connection username>

    Properties

    Description

    Dataproc cluster name

    Name of the Dataproc cluster.

    JDBC connection URL

    JDBC connection string for the hive metastore containing the metadata of the form: jdbc:DB_Driver://HOST:PORT/hive

    JDBC driver name

    JDBC Driver class name for the hive metastore containing the metadata

    JDBC connection password

    Password used to access the hive metastore.

    JDBC connection username

    Username used to access the hive metastore.

    For example:

    com.unraveldata.hive.metastore.list=data-analysis,data-science,al-ml-cluster-dataproc-verion-1-5,al-ml-dataproc-veriosn-2-0
hive.metastore.cluster.ids=default
javax.jdo.option.al-ml-cluster-dataproc-verion-1-5.ConnectionDriverName=com.mysql.jdbc.Driver
javax.jdo.option.al-ml-cluster-dataproc-verion-1-5.ConnectionPassword=hive-password
javax.jdo.option.al-ml-cluster-dataproc-verion-1-5.ConnectionURL=jdbc:mysql://al-ml-cluster-dataproc-verion-1-5-m/metastore
javax.jdo.option.al-ml-cluster-dataproc-verion-1-5.ConnectionUserName=hive
javax.jdo.option.al-ml-dataproc-veriosn-2-0.ConnectionDriverName=com.mysql.jdbc.Driver
javax.jdo.option.al-ml-dataproc-veriosn-2-0.ConnectionPassword=hive-password
javax.jdo.option.al-ml-dataproc-veriosn-2-0.ConnectionURL=jdbc:mysql://al-ml-dataproc-veriosn-2-0-m/metastore
javax.jdo.option.al-ml-dataproc-veriosn-2-0.ConnectionUserName=hive
javax.jdo.option.data-analysis.ConnectionDriverName=com.mysql.jdbc.Driver
javax.jdo.option.data-analysis.ConnectionPassword=hive-password
javax.jdo.option.data-analysis.ConnectionURL=jdbc:mysql://data-analysis-m/metastore
javax.jdo.option.data-analysis.ConnectionUserName=hive
javax.jdo.option.data-science.ConnectionDriverName=com.mysql.jdbc.Driver
javax.jdo.option.data-science.ConnectionPassword=hive-password
javax.jdo.option.data-science.ConnectionURL=jdbc:mysql://data-science-m/metastore
javax.jdo.option.data-science.ConnectionUserName=hive

  2. From the installation directory, where Unravel binaries are installed, run the following command and provide the path to the properties file.

    <Unravel installation directory>/unravel/manager config properties import <path to the properties file>
For example:
/opt/unravel-install/unravel/manager config properties import /opt/properties.txt

  3. Ensure to stop Unravel.

    <Unravel installation directory>/unravel/manager stop

  4. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply

  5. Start Unravel.

    <Unravel installation directory>/unravel/manager start
In this section: