Skip to main content

Home

Configuring Hive Metastore (On-prem)

Unravel should connect to your Hive Metastore to populate the Data page by analyzing your table usage in conjunction with Hive query information. Unravel Server collects information through the Hive API, which is similar to the beeline CLI and uses the JDBC database connection protocol. Data

This topic takes you through the steps to connect Unravel to Hive Metastore. If you configure-external-hive-metastore-emr.are using Oracle, you also have to configure it to grant Unravel access.Configuring access for an Oracle database

Copy database jar for hive metastore configuration

Ensure that the database jar file for MySQL, MariaDB, or Oracle DB is copied to the following location before you configure Hive metastore. 

<Unravel installation directory>/unravel/share/java/

For example:

/opt/unravel/share/java/

Hive metastore connection can be set either with auto-configuration or manual configuration in a single cluster as well as in multi-cluster environments.

Connecting to Hive metastore in single cluster environment

In a single cluster environment, the Hive metastore connection can be set either with auto-configuration or manual configuration:

Option 1: Auto-configuration (Single cluster)

  1. Stop Unravel

    <Unravel installation directory>/unravel/manager stop

  2. Run auto-configuration. 

    <unravel_installation_directory>/unravel/manager config auto

    Note

    The Hive metastore database password can be recovered automatically only for a cluster manager with an administrative account. Otherwise, it must be set manually.

  3. To set the Hive metastore password manually run the following command. If the password is omitted, it will be prompted without an echo.

    CLUSTER_KEY is the name of the cluster where you set the Hive configurations.

    <Unravel installation directory>/unravel/manager config hive metastore password <CLUSTER_KEY> <HIVE_KEY> <password>
##Example:/opt/unravel/manager config hive metastore password cluster1 hive P@SsWorD

  4. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply

  5. Start Unravel

    <Unravel installation directory>/unravel/manager start

    Go to Unravel UI's Jobs > Applications page to confirm that Hive queries are displayed. Approximately twenty-four hours after configuration the Data page displays a list of your Hive Metastore tables along with their KPIs and other details.Data

Option 2: Manual configuration (Single cluster)

  1. Stop Unravel

    <Unravel installation directory>/unravel/manager stop

  2. Run the following command to connect to Hive metastore.

    Tip

    Ensure to download the compatible JDBC connector jar for your database and place the connector jar at <Unravel Installation directory>/unravel/share/java.

    PostgreSQL JDBC connector is already bundled with Unravel. 

    <Unravel installation directory>/unravel/manager config hive metastore set <CLUSTER_KEY> <HIVE_KEY> <DRIVER> <URL> <USER> <PASSWORD>
##For example: /opt/unravel/manager config hive metastore set my-cluster hive com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/database user Pa$$c0De

    This will set the following metastore database information:

    • JDBC driver: JDBC Driver class name for the data store containing the metadata.

      For example:

      • MySQL: com.mysql.jdbc.Driver

      • PostgreSQL: org.postgresql.Driver

      • Oracle: oracle.jdbc.driver.OracleDriver

      • Microsoft: com.microsoft.sqlserver.jdbc.SQLServerDriver

    • JDBC URL: JDBC connection string in the format expected by the driver.

      For example:

      • MySQL: jdbc:mysql://host:port/database

      • PostgreSQL: jdbc:postgresql://host:port/database

      • Oracle: jdbc:oracle:thin:@database:port:sid

      • Microsoft: jdbc:sqlserver://hostname:port;databaseName=database

    • Username: Username used to access the data store.

    • Password: Password used to access the data store.

  3. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply

  4. Start Unravel

    <Unravel installation directory>/unravel/manager start
Connecting to Hive metastore in multi-cluster environment

In a multi-cluster environment, the Hive metastore connection can be set either with auto-configuration or manual configuration:

Option 1: Auto-configuration (Multi-cluster)

  1. Stop Unravel

    <Unravel installation directory>/unravel/manager stop

  2. Run auto-configuration.

    • Run the following command from the core node:

      <unravel_installation_directory>/unravel/manager config edge auto <EDGE_KEY>
##Example: /opt/unravel/manager config edge auto my-edge

    • Run the following command from the core node , if the core node is directly monitoring the Hadoop cluster:

      <unravel_installation_directory>/unravel/manager config auto

  3. Set the Hive metastore password. The Hive metastore database password can be recovered automatically only for a cluster manager with an administrative account. Otherwise, it must be set manually.

    1. Run the manager config edge show command to get the <EDGE_KEY>, <HIVE_KEY>, and <CLUSTER_KEY>, which must be provided when you set the Hive metastore password.

      • <EDGE_KEY> is the label you provide to identify the edge node when you set the cluster.

      • CLUSTER_KEY is the name of the cluster where you set the Hive configurations.

      • <HIVE_KEY> is the definition of the Hive service. In the output of the manager config edge show command, this is shown as <SERVICE_KEY>

    2. Run the following command on the core node to set the password:

      <Unravel installation directory>/unravel/manager config edge hive metastore password EDGE_KEY> <CLUSTER-KEY> <HIVE-KEY> <password> 
##Example: /opt/unravel/manager config edge hive metastore password local-node cluster1 hive P@SsWorD

      If the core node is monitoring the Hadoop cluster directly, run the following command from the core node to set the password:

      <Unravel installation directory>/unravel/manager config hive metastore password <CLUSTER_KEY> <HIVE_KEY> <password> 
##Example: /opt/unravel/manager config edge hive metastore password clluster1 hive P@SsWorD

  4. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply

  5. Start Unravel

    <Unravel installation directory>/unravel/manager start

    Go to Unravel UI's Jobs > Applications page to confirm that Hive queries are displayed. Approximately twenty-four hours after configuration the Data page displays a list of your Hive Metastore tables along with their KPIs and other details.Data

Option 2: Manual configuration (Multi-cluster)

  1. Stop Unravel

    <Unravel installation directory>/unravel/manager stop

  2. Run the manager config edge show command to get the <EDGE_KEY>, <HIVE_KEY>, and <CLUSTER_KEY>, which must be provided when you connect to the Hive metastore.

    • <EDGE_KEY> is the label you provide to identify the edge node when you set the cluster.

    • CLUSTER_KEY is the name of the cluster where you set the Hive configurations.

    • <HIVE_KEY> is the definition of the Hive service. In the output of the manager config edge show command, this is shown as <SERVICE_KEY>.

  3. Run the following command to connect to Hive metastore.

    Tip

    Ensure to download the compatible JDBC connector jar for your database and place the connector jar at <Unravel Installation directory>/unravel/share/java.

    <Unravel installation directory>/unravel/manager config edge hive metastore set <EDGE_KEY> <CLUSTER_KEY> <HIVE_KEY> <DRIVER> <URL> <USER> <PASSWORD>
##Example: /opt/unravel/manager config edge hive metastore set my-edge my-cluster hive com.mysql.jdbc.Driver jdbc:mysql://localhost:3306/database user passcode

    This will set the following metastore database information:

    • JDBC driver: JDBC Driver class name for the data store containing the metadata.

      For example:

      • MySQL: com.mysql.jdbc.Driver

      • PostgreSQL: org.postgresql.Driver

      • Oracle: oracle.jdbc.driver.OracleDriver

      • Microsoft: com.microsoft.sqlserver.jdbc.SQLServerDriver

    • JDBC URL: JDBC connection string in the format expected by the driver.

      For example:

      • MySQL: jdbc:mysql://host:port/database

      • PostgreSQL: jdbc:postgresql://host:port/database

      • Oracle: jdbc:oracle:thin:@database:port:sid

      • Microsoft: jdbc:sqlserver://hostname:port;databaseName=database

    • Username: Username used to access the data store.

    • Password: Password used to access the data store.

  4. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply

  5. Start Unravel

    <Unravel installation directory>/unravel/manager start
In this section: