Skip to main content

Home

Integrate database (Cloud)

Refer to the following instructions to integrate Unravel with a database other than the default PostgreSQL shipped with the installer.

You can integrate Unravel managed and an external database for usage with Unravel. Unravel supports MySQL, MariaDB, and PostgreSQL (Postgres). Check the compatibility matrix for the versions of MySQL, MariaDB, and Postgres compatible with the corresponding Unravel platform version.

MySQL

You can integrate Unravel-managed MySQL as well as external MySQL with Unravel.

Unravel managed MySQL

Important

The following packages must be installed for fulfilling the OS-level requirements for MySQL:

  • numactl-libs (for libnuma.so)

  • libaio (for libaio.so)

To integrate with Unravel managed MySQL, do the following:

  1. Create a mysql directory in /tmp. Provide permissions and make them accessible to the user who installs Unravel.

  2. Download the following tar files for MySQL/MariaDB server to /tmp/mysql directory:

    • mysql-5.7.27-linux-glibc2.12-x86_64.tar.gz

    • mysql-connector-java-5.1.48.tar.gz

  3. Run the setup command, as an Unravel user, to set up Unravel with MySQL database:

    <unravel_installation_directory>/unravel/versions<Unravel version>/setup --enable-<platform> --extra /tmp/mysql

#For platform specify databricks or emr

This will configure and manage MySQL for usage with Unravel.

External MySQL

For configuring an external MySQL database with Unravel, you must provide the JDBC connector. This can either be as a tar file or as a jar file.

To integrate Unravel with an external database, do the following:

  1. Create unravel schema and user on the target database where the unravel user should have full access to the schema.

    ## For example:
CREATE DATABASE unravel_mysql_prod;
CREATE USER 'unravel'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON unravel_mysql_prod.* TO 'unravel'@'localhost';

  2. Create a directory for MySQL in /tmp. Provide permissions and make them accessible to the user who installs Unravel.

  3. Add the JDBC connector to /tmp/<MySQL-directory>/<jdbcconnector> directory. This can be either a tar file or a jar file.

  4. Run the setup command, as an Unravel user, to set up Unravel with MySQL database:

    With SSL

    1. Integrate the database using any of the following command-line options:

      • Using command-line

        <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-<platform> --extra /tmp/<MySQL-directory> --external-database-ssl mysql <HOST> <PORT> <SCHEMA> <USERNAME> PASSWORD
#For platform specify databricks or emr
##The HOST, PORT, SCHEMA, USERNAME, PASSWORD are optional fields and are prompted if missing.

      • Using interactive command-line UI

        1. Run the following command:

          <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-<platform> --extra /tmp/<MySQL-directory> --external-database-ssl mysql
#For platform specify databricks or emr

          The details required for the following are prompted on the screen:

          Host:
Port:
Schema:
Username:
Password:

        2. Specify the hostname, port number, username, and password details. Type the password, which will be masked, and press ENTER.

    2. Add the certificate files:

      <unravel_installation_directory>/unravel/manager config tls trust add <one or more certificate files>

    3. Enable the truststore and apply changes.

      <unravel_installation_directory>/unravel/manager config tls trust enable
<unravel_installation_directory>/unravel/manager config apply

    4. Start Unravel. 

      <unravel_installation_directory>/unravel/manager start

    Without SSL

    Integrate the database using any of the following options:

    • Using command-line

      Run the following command:

      <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-<platform> --extra /tmp/<MySQL-directory> --external-database mysql <HOST> <PORT> <SCHEMA> <USERNAME> 
#For platform specify data bricks or emr
##The HOST, PORT, SCHEMA, USERNAME, PASSWORD are optional fields and are prompted if missing.

    • Using interactive command-line

      1. Run the following command:

        <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-<platform> --extra /tmp/<MySQL-directory> --external-database mysql 
#For platform specify data bricks or emr

        The details required for the following are prompted on the screen:

        Host:
Port:
Schema:
Username:
Password:

      2. Specify the hostname, port number, username, and password details. Type the password, which will be masked, and press ENTER.

  5. You are prompted to enter the password. Type the password, which will be masked, and press ENTER.

This will configure and manage external MySQL for usage with Unravel.

Note

In the case of MySQL 8.0, the client does not support the authentication protocol requested by the server, and there can be a failed connection to the database. Run the following to resolve this authentication issue:

ALTER USER 'unravel'@'unravel-host' IDENTIFIED WITH mysql_native_password BY 'password';flush privileges;

Where unravel is your user, unravel-host is the URL, and password is your password.

MariaDB

You can integrate Unravel with MariaDB by providing a Java JDBC connector.

Unravel managed MariaDB

Important

The following packages must be installed for fulfilling the OS level requirements for MariaDB:

  • numactl-libs (for libnuma.so)

  • libaio (for libaio.so)

To configure with Unravel managed MariaDB, do the following:

  1. Create a mariadb directory in /tmp. Provide permissions and make them accessible to the user who installs Unravel.

  2. Download the following tar files for mariadb server to /tmp/mariadb directory:

    • mariadb-10.4.13-linux-x86_64.tar.gz

    • mariadb-java-client-2.6.0.jar

  3. Run the setup command, as an Unravel user, to set up Unravel with MariaDB database:

    <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-<platform>  --extra /tmp/mariadb
#For platform specify databricks or emr

This will configure and manage MariaDB for usage with Unravel.

External MariaDB

For configuring an external MariaDB database with Unravel, you must provide the JDBC connector. This can either be as a tar file or as a jar file.

To integrate Unravel with external MariaDB, do the following:

  1. Create unravel schema and user on the target database where the unravel user should have full access to the schema.

    ## For example:
CREATE DATABASE unravel_mariadb_prod;
CREATE USER 'unravel'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON unravel_mariadb_prod.* TO 'unravel'@'localhost';

  2. Create a directory for MariaDB in /tmp. Provide permissions and make them accessible to the user who installs Unravel.

  3. Add the JDBC connector to /tmp/<MariaDB-directory>/<jdbcconnector> directory. This can be either a tar file or a jar file.

  4. Run the setup command as an Unravel user to set up Unravel with an external MariaDB database:

    With SSL

    1. Integrate the database using any of the following command-line options:

      • Using command line

        <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-<platform> --extra /tmp/<MariaDB-directory> --external-database-ssl mariadb <HOST> <PORT> <SCHEMA> <USERNAME> PASSWORD
#For platform specify databricks or emr
##The HOST, PORT, SCHEMA, USERNAME, PASSWORD are optional fields and are prompted if missing.

      • Using interactive command line UI

        1. Run the following command:

          <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-<platform> --extra /tmp/<MariaDB-directory> --external-database-ssl mariadb
#For platform specify databricks or emr

          The details required for the following are prompted on the screen:

          Host:
Port:
Schema:
Username:
Password:

        2. Specify the hostname, port number, username, and password details. Type the password, which will be masked, and press ENTER.

    2. Add the certificate files:

      <unravel_installation_directory>/unravel/manager config tls trust add <one or more certificate files>

    3. Enable the truststore and apply changes.

      <unravel_installation_directory>/unravel/manager config tls trust enable
<unravel_installation_directory>/unravel/manager config apply

    4. Start Unravel. 

      <unravel_installation_directory>/unravel/manager start

    Without SSL

    Integrate the database using any of the following options:

    • Using command-line

      Run the following command:

      <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-<platform> --extra /tmp/<MariaDB-directory> --external-database mariadb <HOST> <PORT> <SCHEMA> <USERNAME> 
#For platform specify data bricks or emr
##The HOST, PORT, SCHEMA, USERNAME, PASSWORD are optional fields and are prompted if missing.

    • Using interactive command-line

      1. Run the following command:

        <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-<platform> --extra /tmp/<MariaDB-directory> --external-database mariadb 
#For platform specify data bricks or emr

        The details required for the following are prompted on the screen:

        Host:
Port:
Schema:
Username:
Password:

      2. Specify the hostname, port number, username, and password details. Type the password, which will be masked, and press ENTER.

This will set up Unravel with an external MariaDB.

PostgreSQL

You can configure external PostgreSQL, as well as the Unravel managed PostgreSQL.

Unravel managed PostgreSQL

To configure the Unravel managed PostgreSQL database, run the setup command:

<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-<platform> 
#For platform specify databricks or emr

This will set up Unravel with PostgreSQL.

External PostgreSQL

To configure external PostgreSQL, do the following:

  1. Create unravel schema and user on the target database where the unravel user should have full access to the schema.

  2. Run the setup command as an Unravel user to set up Unravel with an external PostgreSQL database:

    With SSL

    Note

    Appstore is not supported for PostgreSQL over SSL.

    1. Integrate the database using any of the following command-line options:

      • Using command line

        <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-<platform> --external-database-ssl postgresql <HOST> <PORT> <SCHEMA> <USERNAME> PASSWORD
#For platform specify databricks or emr
##The HOST, PORT, SCHEMA, USERNAME are optional fields and are prompted if missing.

      • Using interactive command line UI

        1. Run the following command:

          <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-<platform> --external-database-ssl postgresql
#For platform specify databricks or emr

          The details required for the following are prompted on the screen:

          Host:
Port:
Schema:
Username:
Password:

        2. Specify the hostname, port number, username, and password details. Type the password, which will be masked, and press ENTER.

    2. Add the certificate files:

      <unravel_installation_directory>/unravel/manager config tls trust add <one or more certificate files>

    3. Enable the truststore and apply changes.

      <unravel_installation_directory>/unravel/manager config tls trust enable
<unravel_installation_directory>/unravel/manager config apply

    4. Start Unravel. 

      <unravel_installation_directory>/unravel/manager start

    Without SSL

    • Using command line

      Run the following command:

      <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-<platform> --external-database postgresql <HOST> <PORT> <SCHEMA> <USERNAME> PASSWORD
#For platform specify databricks or emr
##The HOST, PORT, SCHEMA, USERNAME are optional fields and are prompted if missing.

    • Using interactive command-line

      <unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-<platform> --external-database postgresql
#For platform specify databricks or emr

      The details required for the following are prompted on the screen. Specify the hostname, port number, username, and password details. Type the password, which will be masked, and press ENTER.

      Host:
Port:
Schema:
Username:
Password:

      This will set up Unravel with an external PostgreSQL.

Set up Unravel Managed PostgreSQL for Hive metastore with MySQL

If you are using Unravel managed PostgreSQL database and the Hive metastore is using MySQL, you can use one of the following options for setup.

Option 1

You can use this option if the Hive metastore is using MySQL version 5.7 or 8.0.

  1. Create a directory. For example, mysql directory in /tmp. Provide permissions and make them accessible to the user who installs Unravel.

  2. Download the following tar file (MySQL 5.7 or 8.0) to this directory and provide the directory path when you run setup to install Unravel with Unravel managed PostgreSQL.

    • For MySQL 5.7: mysql-connector-java-5.1.x.tar.gz

    • For MySQL 8.0: mysql-connector-java-8.0.x.tar.gz

  3. Run the setup command as follows:

    • Core node with Hadoop configuration

      <Unravel installation directory>/unravel/versions/<Unravel version>/setup --extra /path/to/directory with MySQL connector file
      ##Example:
/opt/unravel/versions/abcd.1234/setup --extra /tmp/mysql

    • Core node without Hadoop configuration

      <Unravel installation directory>/unravel/versions/<Unravel version>/setup --enable-core --extra /path/to/directory with MySQL connector file
      ##Example:
/opt/unravel/versions/abcd.1234/setup --extra /tmp/mysql --enable-core 
Option 2

You can use this option if the Hive metastore is using any other version other than MySQL version 5.7 or 8.0.

  1. Install Unravel.

  2. Copy the MySQL connector file to <unravel_installation_directory>/share/java/path.

  3. Provide permissions and make the location accessible to the user who installs Unravel. 

    chown -R username:groupname /path/to/directory with MySQL connector file
In this section: