Installing Unravel for Snowflake

To install Unravel for Snowflake, complete the prerequisites and then run the following steps to Install and configure Unravel. You can configure Unravel for Snowflake based on your chosen database connection method for metadata collection.

Prerequisites (Snowflake)

To configure Unravel for Snowflake, you can choose between establishing a Real-time connection or a Snapshot connection to link up with the database for metadata collection. You must complete the prerequisites specific to your preferred connection type (Real-time or Snapshot) before proceeding with the installation of Unravel for Snowflake.

Ensure to complete the following prerequisites for Real-time connection.

Permissions

Create an unravel user in your Snowflake account.
Grant SELECT privilege to the unravel user for the following schemas:
- SNOWFLAKE.ACCOUNT_USAGE
- Information_schema.query_history (For Real-time monitoring)
Provide the Unravel user with the necessary permissions to use the warehouse (XS) for executing Unravel queries.
Authorize the Unravel user to run information schema queries on essential warehouses that require Real-time monitoring. This is optional.

Sizing requirements to install Unravel

The sizing provided is for a Snowflake account that monitors up to 10 warehouses and runs approximately 10,000 queries per day.

If your workload is different, contact the Unravel Support team for accurate sizing.

Hardware/ Virtual machine requirements

Cores: 8
RAM: 96 GB
Disk: Based on the days of storage (minimum 120 GB required)

Software requirements

OS: CentOS 7.9
Browser: Chrome 112.x or greater
Java: OpenJDK - 17.0.3
Instance type: Azure instance, AWS instance, VM with similar configuration

Network requirements

The system on which Unravel is installed should be capable of connecting to Snowflake over JDBC.
Open port 3000 for to and fro communication with Unravel UI. This port should be enabled and accessible.

Ensure to complete the following prerequisites for metadata collection.

Permissions

Create unravel user in your Snowflake account.
Grant SELECT privilege to the unravel user for schema (metadata uploaded schema).
Provide the Unravel user with the necessary permissions to use the warehouse (XS) for executing Unravel queries.

Sizing requirements to install Unravel

The sizing provided is for a Snowflake account that monitors up to 10 warehouses and runs approximately 15,000 queries for 15 days.

If your workload is different, contact the Unravel Support team for accurate sizing.

Hardware/ Virtual machine requirements

Cores: 4
RAM: 32 GB
Disk: Based on the days of storage (minimum 120 GB required)

Software requirements

OS: CentOS 7.9
Browser: Chrome 112.x
Java: OpenJDK - 17.0.3
Instance type: Azure instance, AWS instance, VM with similar configuration

Network requirements

The system on which Unravel is installed should be capable of connecting to Snowflake over JDBC.
Open port 3000 for to and fro communication with Unravel UI. This port should be enabled and accessible.

1. Downloading Unravel

Download Unravel onto the VM instance that you have created.

Important

Before downloading Unravel for your platform, ensure to get the username and password from Unravel Support.

Download the Unravel package (RPM or TAR):

You can refer to Download section for the complete list of Unravel product downloads.

For example:

Tar

curl -v https://preview.unraveldata.com/unravel/RPM/4.x.x/unravel-4.x.x.x-cloud.tar.gz -u username:password

RPM

curl -v https://preview.unraveldata.com/unravel/RPM/4.x.x/unravel-4.x.x.x.rpm -o unravel-4.x.x.x.rpm -u username:password

2. Deploying Unravel

Unravel binaries are available as a tar file or RPM package. You can deploy the Unravel binaries in any directory on the server. However, the user who installs Unravel must have the write permissions to the directory where the Unravel binaries are deployed.

After deploying the Unravel binaries, the directory layout for the Tar and RPM will be unravel/versions/<Directories and files>. The binaries are deployed to <Unravel_installation_directory>, and Unravel will be available in <Unravel_installation_directory/unravel.

The following steps to deploy Unravel from a tar file should be performed by a user who will run Unravel.

Create an Installation directory.
```
mkdir /path/to/installation/directory
```
For example: mkdir /opt/
Extract the Unravel tar file to the installation directory, which you have created in the first step. After you extract the contents of the tar file, the unravel directory is created within the installation directory.
```
tar zxf unravel-<version>tar.gz -C /path/to/installation/directory
```
For example: tar zxf unravel-4.x.x.x.tar.gz -C /opt
The unravel directory will be available within /opt.
Grant ownership of the directory to a user who will run Unravel.
```
chown -R username:groupname /path/to/installation/directory
```
For example: chown -R hadoop:hadoop /opt/unravel/

Important

A root user should perform the following steps to deploy Unravel from an RPM package. After the RPM package is deployed, the remaining installation procedures should be completed by the Unravel user.

Create an installation directory.
```
mkdir /usr/local/unravel
```
Run the following command:
```
rpm -i unravel-<version>.rpm
```
For example: rpm -i unravel-4.x.x.x.rpm
The unravel directory will be available in /usr/local.
If you want to provide a different location, use the --prefix command. For example:
```
mkdir /opt/unravel
rpm -i unravel-4.x.x.x.rpm --prefix /opt
```
The unravel directory will be available in /opt.
Grant ownership of the directory to a user who will run Unravel. This user executes all the processes involved in the Unravel installation.
```
chown -R username:groupname /usr/local/unravel
```
For example: chown -R hadoop:hadoop /usr/local/unravel
Continue with the installation procedures as unravel user.

3. Installing Unravel for Snowflake

You can install Unravel on Snowflake using the Interactive Precheck method or manually.

Installing Unravel for Snowflake with Interactive Precheck method

The Interactive Precheck utility is run to validate the required configurations before installing Unravel. When you run the Interactive Precheck utility, various checks are prompted for gathering configuration information. The responses you provide for these checks generate a bootstrap configuration file. This file, which contains the configuration information, is then used to install Unravel.

Do the following to install and configure Unravel with Interactive Precheck.

After you download and deploy the Unravel, run the precheck.sh script from unravel/versions/X.Y.Z/healthcheck/.
For example:
/opt/unravel/versions/X.Y.Z/healthcheck/precheck.sh

Enter the necessary details when you are prompted for the following configuration information:

This check allows you to configure database-related information and an external database for Unravel.

-- Database configuration
 Configure an external database? (y/n) [No]:

If you answer No, an Unravel-managed database is used for the installation.

If you answer Yes, you are further prompted for the type of external database that you want to configure.

-- Database configuration
   
   Configure an external database? (y/n) [No]: y

   Type
   1- PostgresQL
   2- MySQL
   3- MariaDB

If you choose a specific type of external database, you are prompted for the following database information and test connectivity to that database. Refer to Integrating Database for more details. For example:

-- Database configuration
  
   Configure an external database? (y/n) [No]: y

   Type
   1- PostgresQL
   2- MySQL
   3- MariaDB

   Select one of the above []: 1
   Selected: PostgresQL

   Database hostname [None]:

   Database port (integer) [None]:

   Database schema [None]:

   Does the database use TLS (y/n) [No]: 

   Database username [None]:

   Database password [None] (no echo):

   Do you wish to test connecting to the external database? (y/n) [Yes]:-- Database configuration
 Configure an external database? (y/n) [No]:

If you choose MySQL or MariaDB database you are further prompted for extra packages. If you answer Yes, the Extra packages section searches for the required JDBC drivers.
```
-- Database configuration
 Will Unravel connect to a MySQL or MariaDB database (ex: hive metastore) ? (y/n) [No]: 
```

The license option prompts you to set the Unravel license. Specify the license file location.

--License
Do you want to set the unravel license now? (y/n) [Yes]:
License file location (ex: /path/to/unravel.license) [/home/unravel/valid.lic]:
License: Ok

If the license file is valid, the OK message is displayed.
If the license file is expired, the license expired error is displayed with the license expiry date and timestamp.
If the license file content is incorrect, the invalid license error is displayed with the appropriate error message. You can fix the error in the license file and then set the license.
To resolve the errors, see Licensing error messages and troubleshooting.

Note

If the filename is not provided, the command prompts for the license information. You can provide either a path to the license file or the content of the license file.

Sample content of the license file:

##### BEGIN UNRAVEL LICENSE 
Licensee     : ACME Disintegrating Pistol Manufacturing
Valid from    :  2022-12-16 00:00:00 UTC 
Expire after  :  2023-10-16 23:59:00 UTC
License type : Enterprise
Licensed number of nodes : 1000000
Signature    : c2Uvb2JqLnRhcmdldC92OF9pbml0aWFsaXplcnMvZ2VuL3RvcnF1ZS
Revision     : 1
##### END UNRAVEL LICENSE #####

The Extra packages check shows if you use Unravel-managed MySQL/MariaDB or need JDBC drivers. Else, this check is automatically skipped.

-- Extra package location

*** JDBC drivers are required for Unravel managed MySQL or MariaDB.
*** Database software package is required for Unravel managed MySQL or MariaDB.

External package location [None]: /<my-extra-packages>
##This is the path to the directory where the required packages are located.

If the required packages are located, then the following message is shown:

The following packages will be installed:
   Database server: /my-extra-packages/mysql-5.7.27-linux-glibc2.12-x86_64.tar.gz
   JDBC driver:
     - /my-extra-packages/mysql-connector-java-5.1.48.tar.gz

 External package: Ok

If the required packages are not found, then the following error message is showing:

External package: ERROR
 - ERROR: Couldn't find jdbc drivers in /my-extra-packages
 - ERROR: Looked for:
   mysql-connector-java-*.tar.gz
   mysql-connector-java-*.jar
   mariadb-java-client-*.jar
 - ERROR: Couldn't find database server package in /my-extra-packages
 - ERROR: Looked for:
   mysql-*-linux-glibc2.12-x86_64.tar.gz
   mariadb-*-linux-x86_64.tar.gz

This check allows you to configure and test HTTPS for the unravel UI. This check prompts you for the certificate, key, password, and hostname details used to access Unravel.

Use HTTPS to access unravel? (y/n) [Yes]:  
##If you answer “Yes”, you are prompted for the path to the certificate and key. Unravel uses this information to configure TLS during installation. 
##If you answer “No”, you are shown a warning message for confirmation.

The information provided is verified for the following:

If the Key and Certificate match
If the certificate is valid
If the certificate applies for the provided hostname

This check allows you to set the Unravel UI port and verify the connectivity.

-- Unravel default port
  
  Port number (integer) [3000]: 

  Do you want to test if the port is accessible? (y/n) [Yes]: 
    
  This will open port 3000 and listen for connection for 120 seconds.
  Use your browser to test if the Unravel UI will be accessible on that port.
    
  We have detected the following hostnames:
  - some.host.example
    
 Browse to: http://some.host.example:3000
    
 ATTENTION: This address is an example. You should test with the URL that will be used to access Unravel.

A connection on port 3000 is tried and established. If the connection is successful, Unravel Port Test: OK is shown on the browser, and Unravel port: Reached is shown on the server.

This check allows you to set a custom data directory and verify the access if the directories exist. You will always find the software location where you deploy the Unravel binaries. In this check, only the space and access are tested. That data location that you have configured will be used.

-- Unravel directories

  Software [/opt/unravel]: 

  Data [/opt/unravel/data]: 

  Directories: ERROR
  - OK: 33 GB of free disk space for software.
  - ERROR: SYSH0026: Space for data 33 GB is low, recommended minimum is 100 GB.

This check allows you to configure and test email. You are prompted for host and credentials, and the following items are tested:

Connectivity
Authentication, only if provided.
Optional: Send test mail.

Following is a sample:

-- Mail server (SMTP) configuration

Unravel can send notification and alert emails.
   This will allow you to configure and test connection to a SMTP server.
   Optionally, it can also send a test email.

   You will have to provide:
   - Protocol, hostname and port
   - Credentials if required



   Configure a SMTP server? (y/n) [No]: y

   SMTP hostname [None]: smtphostname.gmail.com

   SMTP port (usually 25 for clear text, 465 for SSL, 587 for STARTLS) (integer) [None]: 587

   Security protocol
   1- None
   2- SSL
   3- StartTLS

   Select one of the above [None]: 3
   Selected: StartTLS

   Authentication required? (y/n) [Yes]: y

   Username [None]: daemon@unraveldata.com

   Password [None] (no echo):

   From [None]: daemon@unraveldata.com

   To [None]: user@unraveldata.com

   Send test email (y/n) [No]: y

Note

For more information, refer to Using the Interactive Precheck utility.

The responses that you have provided for the configuration information are used to generate a configuration file. You can use this configuration file when you run the setup command to install and configure Unravel.

After you have completed the responses, you are prompted to confirm if you want to generate the bootstrap configuration file. Press ENTER if you want to generate the bootstrap configuration file.
```
-- Unravel bootstrap configuration

   Generate a unravel bootstrap configuration file? (y/n) [Yes]:
```
The bootstrap configuration file is generated and located at $HOME/unravel-interactive-precheck/unravel-bootstrap.yaml.

Install Unravel with the bootstrap configuration file.

<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --bootstrap $HOME/unravel-interactive-precheck/unravel-bootstrap.yaml

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start all the services.

<unravel_installation_directory>/unravel/manager start

Check the status of services.
```
<unravel_installation_directory>/unravel/manager report 
```
The following service statuses are reported:
- OK: Service is up and running.
- Not Monitored: Service is not running. (Has stopped or has failed to start)
- Initializing: Services are starting up.
- Does not exist: The process unexpectedly disappeared. A restart will be attempted ten times.
You can also get the status and information for a specific service. Run the manager report command as follows:
```
<unravel_installation_directory>/unravel/manager report <service> 
```
For example: /opt/unravel/manager report auto_action

Installing Unravel on Snowflake using the Manual method

About setup command

The setup command allows you to do the following:

Runs Precheck automatically to detect possible issues that prevent a successful installation. Suggestions are provided to resolve issues. Refer to Precheck filters for the expected value for each filter.
Let you run extra parameters to integrate the database of your choice.
The setup command allows you to use a managed database shipped with Unravel or an external database. When you run the setup command without additional parameters, the Unravel-managed PostgreSQL database is used. Otherwise, you can specify any of the following databases, which are supported by Unravel, with the setup command:
- MySQL (Unravel managed as well as external MySQL database)
- MariaDB (Unravel managed as well as external MariaDB database)
- PostgreSQL (External PostgreSQL)
Refer to Integrate database for details.
Let you specify a separate path for the data directory other than the default path.
The Unravel data and configurations are located in the data directory. By default, the installer maintains the data directory under <Unravel installation directory>/data. You can also change the data directory's default location by running additional parameters with the setup command. To install Unravel with the setup command.
Provides more options for setup.

To install Unravel with the setup command, do the following:

Switch to Unravel user.
```
  su - <unravel user>
```
Notice
The Unravel user who owns the installation directory should run the setup command to install Unravel.
Run setup command with any of the following databases (PostgreSQL, MySQL, MariaDB). Refer to setup options for all the additional parameters that you can run with the setup command.
Run setup command:
```
<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-snowflake
```
Precheck is automatically run when you run the setup command. Refer to Precheck filters for the expected value for each filter. Also, refer to the Precheck sample.
Tip
- Run --help with the setup command and any combination of the setup command for complete usage details.
```
<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --help
```
- Optionally, if you want to provide a different data directory, you can pass an extra parameter (--data-directory) with the setup command as shown below:
```
<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-snowflake --data-directory /the/data/directory
```
  Similarly, you can configure separate directories for other unravel directories. Contact support for assistance.
- Refer to setup Options for all the additional parameters that can be run with the setup command
PostgreSQL
Unravel managed PostgreSQL
<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-snowflake
Notice
If you are using Unravel managed PostgreSQL database, and the Hive metastore is using MySQL, refer Set up Unravel Managed PostgreSQL for Hive metastore with MySQL
External PostgreSQL
<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-snowflake --external-database postgresql
For example: /opt/unravel/versions/abcd.992/setup --enable-snowflake --external-database postgresql
The details required for the following are prompted on the screen:
Host: Port: Schema: Username: Password:
For example:
Host:xyz.unraveldata.com Port:5432 Schema:unravel_db_prod Username:unravel Password:unraveldata
Note
The HOST, PORT, SCHEMA, USERNAME, and PASSWORD are optional fields and are prompted if missing.
MySQL
Unravel managed MySQL
<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-snowflake --extra /tmp/mysql
External MySQL
<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-snowflake --extra /tmp/<MySQL-directory> --external-database mysql
The details required for the following are prompted on the screen:
Host: Port: Schema: Username: Password:
For example:
Host:xyz.unraveldata.com Port:5432 Schema:unravel_db_prod Username:unravel Password:unraveldata
Note
The HOST, PORT, SCHEMA, USERNAME, and PASSWORD are optional fields and are prompted if missing.
MariaDB
Unravel managed MariaDB
<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-snowflake --extra /tmp/mariadb
External MariaDB
<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-snowflake --extra /tmp/<MariaDB-directory> --external-database mariadb
The details required for the following are prompted on the screen:
Host: Port: Schema: Username: Password:
For example:
Host:xyz.unraveldata.com Port:5432 Schema:unravel_db_prod Username:unravel Password:unraveldata
Note
The HOST, PORT, SCHEMA, USERNAME,and PASSWORD are optional fields and are prompted if missing.
Set the path of a license file.
```
<Unravel installation directory>/unravel/manager config license set <license filename>
```
This command takes a filename as input and performs the following actions:
- Reads the license file path and the license file
  The license YAML file contains product licensing information, license validity and expiration date, and the licensed number of clusters and nodes.
- Verifies whether it is a valid license
- Adds the com.unraveldata.license.file property to the unravel.properties file. For information, see License property.
Note
If you do not provide the license filename, the manager config license set command prompts for the license information. You can copy the content of the license file.
Sample content of the license file:
```
##### BEGIN UNRAVEL LICENSE 
Licensee     : ACME Disintegrating Pistol Manufacturing
Valid from    :  2022-12-16 00:00:00 UTC 
Expire after  :  2023-10-16 23:59:00 UTC
License type : Enterprise
Licensed number of nodes : 1000000
Signature    : c2Uvb2JqLnRhcmdldC92OF9pbml0aWFsaXplcnMvZ2VuL3RvcnF1ZS
Revision     : 1
##### END UNRAVEL LICENSE #####
```

Apply the changes and restart.

<unravel_installation_directory>/unravel/manager config apply then start

Check the status of services.
```
<unravel_installation_directory>/unravel/manager report 
```
The following service statuses are reported:
- OK: Service is up and running.
- Not Monitored: Service is not running. (Has stopped or has failed to start)
- Initializing: Services are starting up.
- Does not exist: The process unexpectedly disappeared. Restarts will be attempted 10 times.
You can also get the status and information for a specific service. Run the manager report command as follows:
```
<unravel_installation_directory>/unravel/manager report <service> 
## For example: /opt/unravel/manager report auto_action
```
Access Unravel UI using this link, http://<FQDN>:3000.

Precheck

The Precheck output displays the issues that prevent a successful installation and provides suggestions to resolve them. You must resolve each of the issues before proceeding. See Precheck filters.

After the prechecks are resolved, you must re-login or reload the shell to execute the setup command again.

Note

In certain situations, you can skip the precheck using the setup --skip-precheck command

For example:

/opt/unravel/versions/<Unravel version>/setup --skip-precheck

You can also skip the checks that you know can fail. For example, if you want to skip the Check limits option and the Disk freespace option, pick the command within the parenthesis corresponding to these failed options and run the setup command as follows:

setup --filter-precheck ~check_limits,~check_freespace

Tip

Run --help with the setup command and any combination of the setup command for complete usage details.

<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --help

Precheck filters

Filters	Description	Expected Value
System
Check uptime	Verifies the period since the last server reboot.	>24h
Clock sync	Verifies if the clock synchronization service is running on the server.	The clock synchronization service is up and running.
CPU requirement	Verifies if the server has enough CPUs to run Unravel efficiently.	Check requirements.
Memory requirement	Verifies that the server has enough memory to run Unravel efficiently.	Check requirements.
Disk access	Verifies that the user who runs unravel has access to the configured disk locations.	Unravel users can access the configured disk locations.
Disk Freespace	Verifies if the disk locations have enough free space.	Check requirements
Kerberos tools	Verifies that the Kerberos tools are available on the server to support kerberized environments.	Kerberos tools are installed.
Network ports	Verifies that the network ports used by Unravel are available.	Check requirements
OS libraries	Verifies the required libraries if run with Unravel managed MySQL.	The following packages must be installed for fulfilling the OS level requirements for MySQL: `numactl-libs` (for libnuma.so) `libaio` (for libaio.so)
OS release	Verifies that the OS distribution is supported.	Check compatibility matrix
OS settings	Verifies vm.max_map_count recommended.	Check requirements
SELinux	Verifies if the SELinux status is enabled or not and provides in which mode it is(Permissive, Disabled, Enforcing).	Check product documentation.
Check limits	Verifies that user limits are set to values	Check requirements
Healthcheck report bundle	Healthcheck report tarball. This report provides the summary and information gathered by the healthcheck with the location.

Precheck Sample

/opt/unravel/versions/abcd.1004/setup 
2021-04-05 15:51:30 Sending logs to: /tmp/unravel-setup-20210405-155130.log
2021-04-05 15:51:30 Running preinstallation check...
2021-04-05 15:51:31 Gathering information ................. Ok
2021-04-05 15:51:51 Running checks .................. Ok
--------------------------------------------------------------------------------
system
 Check limits        : PASSED
 Clock sync          : PASSED
 CPU requirement     : PASSED, Available cores: 8 cores
 Disk access         : PASSED, /opt/unravel/versions/develop.1004/healthcheck/healthcheck/plugins/system is writable
 Disk freespace      : PASSED, 229 GB of free disk space is available for precheck dir.
 Kerberos tools      : PASSED
 Memory requirement  : PASSED, Available memory: 79 GB
 Network ports       : PASSED
 OS libraries        : PASSED
 OS release          : PASSED, OS release version: centos 7.6
 OS settings         : PASSED
 SELinux             : PASSED
--------------------------------------------------------------------------------
Healthcheck report bundle: /tmp/healthcheck-20210405155130-xyz.unraveldata.com.tar.gz
2021-04-05 15:51:53 Prepare to install with: /opt/unravel/versions/abcd.1004/installer/installer/../installer/conf/presets/default.yaml
2021-04-05 15:51:57 Sending logs to: /opt/unravel/logs/setup.log
2021-04-05 15:51:57 Instantiating templates ................................................................................................................................................................................................................................ Ok
2021-04-05 15:52:05 Creating parcels .................................... Ok
2021-04-05 15:52:20 Installing sensors file ............................ Ok
2021-04-05 15:52:20 Installing pgsql connector ... Ok
2021-04-05 15:52:22 Starting service monitor ... Ok
2021-04-05 15:52:27 Request start for elasticsearch_1 .... Ok
2021-04-05 15:52:27 Waiting for elasticsearch_1 for 120 sec ......... Ok
2021-04-05 15:52:35 Request start for zookeeper .... Ok
2021-04-05 15:52:35 Request start for kafka .... Ok
2021-04-05 15:52:35 Waiting for kafka for 120 sec ...... Ok
2021-04-05 15:52:37 Waiting for kafka to be alive for 120 sec ..... Ok
2021-04-05 15:52:42 Initializing pgsql ... Ok
2021-04-05 15:52:46 Request start for pgsql .... Ok
2021-04-05 15:52:46 Waiting for pgsql for 120 sec ..... Ok
2021-04-05 15:52:47 Creating database schema ................. Ok
2021-04-05 15:52:50 Generating hashes .... Ok
2021-04-05 15:52:52 Loading elasticsearch templates ............ Ok
2021-04-05 15:52:55 Creating kafka topics .................... Ok
2021-04-05 15:53:36 Creating schema objects ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... Ok
2021-04-05 15:54:03 Request stop ....................................................... Ok
2021-04-05 15:54:16 Done
[unravel@xyz ~]$

Setup Options

setup Options	Description
-h, --help	Shows help for setup.
--config CONFIG	Specify a different path to the configuration file. <unravel_installation_directory>/unravel/versions/`<Unravel version>`/setup --config `path/to/config/directory`
--enable-core	Enables core node support for non-Hadoop clusters.
--enable(platform)	Enables the specified platform. Options are `-mapr, emr, hdi, bigquery, dataproc, databricks, snowflake, presto`
--cluster-access (Edge node parameter)	Enables cluster access to the core node in a multi-cluster environment.
--data-forwarder host:port cluster-type cluster-id	Data forwarder, main unravel node.
--enable-spark-k8s	Enable Spark on Kubernetes support
--data-directory	Location of the data directory.
--internal-tls INTERNAL_TLS	Location of directory with files, or single file with all certificates and key to enable TLS for internal communications.
--external-database [param [param ...]]	Enable external database.
--external-database [param [param ...]]	Enable external database: The `TYPE HOST PORT SCHEMA USERNAME PASSWORD` is prompted if missing.
--external-database-ssl	Enable external database with SSL.
--log-file	Setup log file location. Default is `/tmp/unravel-setup-YYYYMMDD-HHMMSS.log`.
--extra DIR, -e DIR	Specify extra packages location.
--bootstrap BOOTSTRAP_CONFIG	Configure bootstrap.
--precheck	Run the preinstallation check

4. Configure Unravel for Snowflake

The configurations for setting up Unravel for Snowflake vary based on the connection method that you have chosen to link the database for metadata collection. Based on your connection method, configure Unravel for Snowflake.

Configuring Unravel for Snowflake with Real-time connection

With the Real-time connection, you can directly connect to your Snowflake database and, after configuring your Snowflake account details, get live data insights. The real-time connection allows you to access all Unravel features at once, perform real-time monitoring, receive notification alerts, and observe live insights without any requirement for account admin access.

Also, see Choosing a connection to Upload Snowflake data

Getting live data using a Real-time connection

The following steps are involved in setting up an Unravel account.

Access the Unravel UI using the login credentials you received in the email when you signed up for the Unravel Snowflake Standard (Free) account. The data fields are empty when you access Unravel UI for the first time.
Click on the upper right side and select Snowflake configuration.
Select the Real-time option.

Enter the following details:

Field	Description
Snowflake account
User	Enter the username to access the Snowflake account.
Password	Enter the passcode to access the Snowflake account.
Account	Specify the Snowflake user account from where you want to get live data for monitoring.
Role	Specify a role associated with the user account for Unravel to access the correct Snowflake data warehouse.
Parameters
Database name	Specify the Snowflake database where Unravel can access the metadata.
Schema name	Specify the Snowflake schema in your database.
Warehouse name	Specify a warehouse for Unravel to run queries on your metadata (monitor and telemetry data). The minimum warehouse size required by Unravel is X-small.
Look back	Set how many days of metadata should be processed from when this configuration is saved. You can set the look-back period to a maximum of 180 days.Maximum is 180 das
Set Snowflake credit cost
Credit cost	Set the USD cost of your Snowflake credits.

Click Save to save the settings. After you save the setting, Unravel connects to the configured Snowflake account and starts polling the data. New data can be seen on the Queries and Warehouse pages on Unravel UI.
The duration of the entire data loading process depends on both the volume of data and the specified look-back period.

Configuring Unravel for Snowflake with Snapshot connection

With the Snapshot connection, you can manually download the metadata or telemetry data from your Snowflake account and upload this metadata to the Unravel Snowflake account. The Snapshot connection allows you to unlock Unravel features with one-time data access without any serverless cost or the need for account admin access from the Unravel Snowflake account. However, real-time monitoring is not available with this connection.

The download and upload data tasks can be executed using scripts. After uploading the data, you access the Unravel Snowflake account, detect the metadata, and start the data ingestion. The data for the last 14 days get ingested into the Unravel Snowflake account.

Also see, Choosing connection to Upload Snowflake data

The following steps are involved in setting up an Unravel account.

1. Uploading Snowflake metadata to Unravel Snowflake account

The Snowflake metadata can be uploaded using one of the following methods:

Snowflake-data-loader tool
This tool can be used in Linux, MAC, and Windows. You can execute this tool and retrieve account usage information from your Snowflake account and upload it to the Unravel Snowflake account.
Download/upload scripts
You can use the download scripts to download metadata from your Snowflake account and then run the upload scripts to upload the metadata to the Unravel Snowflake account.

Uploading Snowflake metadata using the snowflake-data-loader tool

This tool enables you to retrieve account usage information from one Snowflake account and upload it to a second account.

This tool does the following:

Creates a stage in the source Snowflake account.
Saves source Snowflake account usage to stage.
Downloads account usage information from the source Snowflake account to local.
Creates a stage in the target Snowflake account.
Uploads the account usage information to the target Snowflake account.

Prerequisites for using the snowflake-data-loader tool

The source account user must have the following permissions:

Access SNOWFLAKE.ACCOUNT_USAGE and SNOWFLAKE.INFORMATION_SCHEMA schema
Create Stage permission
Create File Format permission

Uploading Snowflake metadata using the snowflake-data-loader tool

Use a command line to run the snowflake-data-loader tool. You must pass the following arguments from the command line. Some of these are mandatory and some are optional arguments. You are prompted to address the missing mandatory arguments.

Mandatory arguments		Optional arguments
`--source_user`	Your source Snowflake account username.	`--source_login_method`	The login method for the source account. Possible options are password (default), oauth, sso, okta, or keypair.
`--source_password`	Your source Snowflake account password. If source_login_method is password, this argument is required.	`--target_login_method`	The login method for the target account. Possible options are password (default), oauth, sso, okta, or keypair.
`--private_key_path`	The path to your private key file. If source_login_method or target_login_method is `keypair`, this argument is required. The key will be used for both source and target accounts.	`--source_private_link`	The private link for the source account for example: testaccount.us-east-1.privatelink.snowflakecomputing.com.
`--source_account`	Your source Snowflake account ID.	`--target_private_link`	The private link for the target account for example: testaccount.us-east-1.privatelink.snowflakecomputing.com.
`--source_warehouse`	The name of the source warehouse from where you want to retrieve details.	`--source_okta_url`	The okta URL for the source account for example: https://testaccount.okta.com.
`--source_database`	The name of the source account database where the stage is created.	`--target_okta_url`	The okta URL for the target account for example: https://testaccount.okta.com.
`--source_schema`	The name of the source account schema where the stage is created.	`--source_passcode`	Your source Snowflake account MFA password.
`--source_role`	The name of the source account role.	`--target_passcode`	Your target Snowflake account MFA password.
`--target_user`	Your target Snowflake account username.	`--stage`	The name of the stage. The default is unravel_stage.
`--target_password`	Your target Snowflake account password.	`--out`	The directory to save output files. The default is the current directory.
`--target_account`	Your target Snowflake account ID.	`--file_format`	The name of the file format. The default is `unravel_file_format`.
`target_warehouse`	The name of the target warehouse from where you want to retrieve details.	`--debug`	Prints debug messages when set.
`target_database`	The name of the target database.	`--save-sql`	This flag saves all queries as SQL files instead of running them.
`target_schema`	The name of the target schema.	`--disable-cleanup`	This will skip the local temporary file cleanup process.
`--target_role`	The name of the target role.	`--look-back-days`	The number of days to look back for account usage information. Default is 15 days.

If any of the required arguments are missing, you will be prompted to enter them.

The script will also replace - with _ for the value of --stage argument.

You can run the snowflake-data-loader tool on any of the following operating systems:

Linux

Download the latest release of the snowflake-data-loader tool from the following location:
https://github.com/unraveldata-org/snowflake-data-loader/releases

Using a command line, execute the snowflake-data-loader script with the required arguments as follows:

./snowflake-data-loader \
--source_login_method keypair \
--target_login_method password \
--source_user <source_user> \
--private_key_path <private_key_path> \
--source_account <source_account> \
--source_warehouse <source_warehouse> \
--source_database <source_database> \
--source_schema <source_schema> \
--source_role <source_role> \
--target_user <target_user> \
--target_password <target_password> \
--target_account <target_account> \
--target_warehouse <target_warehouse> \
--target_database <target_database> \
--target_schema <target_schema> \
--target_role <target_role>

For example:

./snowflake-data-loader --source_user sf-user --source_account xyz12345.us-east-1 --source_warehouse unraveldata --source_database unraveldb --source_schema unravelschema --source_role SYSADMIN --target_user 6087e389979f870926a1dbf8d41d52b8 --target_password Cl9ATHlRf6mh --target_account ytb00868 --target_warehouse USER_WH --target_database TRIAL_DB --target_schema unravel_qayt00yvh1gru1p_sf --target_role 47572BE3A1393DDCE452900795978313

MAC

Download the latest release of the snowflake-data-loader tool from the following location:
https://github.com/unraveldata-org/snowflake-data-loader/releases
Note
In MAC, you may be prompted to trust the binary. Run the following command to trust the binary:
```
xattr -d com.apple.quarantine  <path_to_the_binary_directory>/snowflake-data-loader
```

Using a command line, execute the snowflake-data-loader script with the required arguments as follows:

./snowflake-data-loader \
--source_login_method keypair \
--target_login_method password \
--source_user <source_user> \
--private_key_path <private_key_path> \
--source_account <source_account> \
--source_warehouse <source_warehouse> \
--source_database <source_database> \
--source_schema <source_schema> \
--source_role <source_role> \
--target_user <target_user> \
--target_password <target_password> \
--target_account <target_account> \
--target_warehouse <target_warehouse> \
--target_database <target_database> \
--target_schema <target_schema> \
--target_role <target_role>

For example:

./snowflake-data-loader --source_user sf-user --source_account xyz12345.us-east-1 --source_warehouse unraveldata --source_database unraveldb --source_schema unravelschema --source_role SYSADMIN --target_user 6087e389979f870926a1dbf8d41d52b8 --target_password Cl9ATHlRf6mh --target_account ytb00868 --target_warehouse USER_WH --target_database TRIAL_DB --target_schema unravel_qayt00yvh1gru1p_sf --target_role 47572BE3A1393DDCE452900795978313

Windows

Download the latest release of the snowflake-data-loader tool from the following location:
https://github.com/unraveldata-org/snowflake-data-loader/releases

Sign in to Windows with a password. Using a command line, execute the snowflake-data-loader script, with the required arguments from the list, as follows:

snowflake-data-loader.exe \
--source_user <source_user> \
--source_password <source_password> \
--source_account <source_account> \
--source_warehouse <source_warehouse> \
--source_database <source_database> \
--source_schema <source_schema> \
--source_role <source_role> \
--target_user <target_user> \
--target_password <target_password> \
--target_account <target_account> \
--target_warehouse <target_warehouse> \
--target_database <target_database> \
--target_schema <target_schema>

For example:

snowflake-data-loader.exe --source_user sf-user --source_account xyz1234.us-east-1 --source_warehouse UNRAVELDATA --source_database SEMI_STRUCTURED_DB --source_schema SEMI_STRUCTURED_SCHEMA --source_role sysadmin --target_user 3e5968c0661f5c37b51aca02bcf827e1 --target_password BNcc5LIIISjc --target_account ytb00868 --target_warehouse USER_WH --target_database TRIAL_DB --target_schema unravel_foqt3sagsf1k1q8_sf --target_role 47572BE3A1393DDCE452900795978313

Uploading Snowflake metadata using download/upload scripts

The following scripts let you retrieve account usage information from your Snowflake account to the Unravel Snowflake account.

snowsql_download_data.sql:
snowsql_show_wareshouses.sql
snowflake_query.py
snowsql_upload_data.sql

The Snowflake metadata or telemetry data, warehouse data, and warehouse parameters data are downloaded/uploaded with these scripts.

Prerequisites for uploading data using down/upload scripts

Ensure to install the following before you start to download Snowflake metadata:
An account with access to creating state and file format.
Account user with permission to access the account_usage schema and all warehouses.

Uploading Snowflake metadata using download/upload scripts

Unravel provides the following download scripts that you can use to download Snowflake metadata, warehouse data, and warehouse parameters data.

snowsql_download_data.sql: Downloads Snowflake metadata or telemetry data.
snowsql_show_wareshouses.sql: Downloads Snowflake warehouse's data.
snowflake_query.py: Downloads Snowflake warehouse parameters

Unravel provides the following upload scripts to upload the downloaded Snowflake metadata to the Unravel Snowflake account:

snowsql_upload_data.sql

Do the following to download Snowflake metadata:

Download the download/upload scripts from this location.
Using SnowSQL connect to a Snowflake account from where you want to download the Snowflake metadata. This account must have access to the creating stage and file format.

Execute the snowsql_download_data.sql script with the required arguments to download the metadata from Snowflake #account_usage views. Refer to the following list of arguments for more details:

snowsql -f /opt/script/snowsql_download_data.sql -d ${db} -s ${schema} -r ${role} -a ${account} -u ${user} -w ${warehouse} -o variable_substitution=true -o log_file=${script output file path/filename} --variable path=$(local/path/to store/downloaded metadata) --variable stage_name=unravel_stage_name --variable file_format=unravel_file_format

For example:

snowsql -f /opt/script/snowsql_download_data.sql -d sf_source_database -s sf_schema -r sysadmin -a sf_account -u sf_user -w sf_warehouse -o variable_substitution=true -o log_file=/opt/script/snowsql_download_data.log --variable path=/opt/download_path/ --variable stage_name=unravel_stage --variable file_format=unravel_file_format

Parameter	Description
-o variable_substitution	Enable the variable substitution switch in the script. Some variables are used in the script for which the values must be passed from the CLI. Set this to true.
--variable path	Specify the local path to store the downloaded Snowflake system metadata.
--variable stage_name	Specify the stage name, which is used to keep the temporary files for download and upload.
--variable file_format	Specify the file format name, which is used by upload/download scripts.
-f	Specify the file name of the script that is executed.
-d	Specify the database used for the script execution.
-s	Specify the name of the schema name used for the script execution.
-r	Specify the role of the user who executes the script.
-a	Specify the Snowflake account, which will be used for the script execution.
-u	Specify the username of the Snowflake user who executes the script.
-w	Specify the Snowflake warehouse.
-o log_file	Specify the path to the log file that will be generated when you execute the script.
-o	Provide the output-related arguments to get the logs in the specified path and format.

After the script is executed, the Snowflake metadata gets downloaded to the specified location.

Execute the snowsql_show_wareshouses.sql script with the required arguments to download the warehouse data. Refer to the following list of arguments for more details:

snowsql -f /opt/script/snowsql_show_wareshouses.sql -d ${db} -s ${schema} -r ${role} -a ${account} -u ${user} -w ${warehouse}  -o output_format=csv -o output_file=${path}/warehouses.csv -o variable_substitution=true

snowsql -f /opt/script/snowsql_show_wareshouses.sql -d database -s database-schema -r sysadmin -a useraccount -u user1 -w warehouse1  -o output_format=csv -o output_file=/opt/unravel/warehouses.csv -o variable_substitution=true

Parameter	Description
-o variable_substitution	Enable the variable substitution switch in the script. Some variables are used in the script for which the values must be passed from the CLI. Set this to true.
-f	Specify the file name of the script that is executed.
-d	Specify the database used for the script execution.
-u	Specify the username of the Snowflake user who executes the script.
-s	Specify the name of the schema name used for the script execution.
-r	Specify the role of the user who executes the script.
-a	Specify the Snowflake account, which will be used for the script execution.
-o	Provide the output-related arguments to get the logs in the specified path and format.
-o output_file	Specify the path to the output file generated on script execution. Note The `output_file` path should be the same as the path mentioned in the snowsql_download_data.sql command
-o output_format	Specify the output file format on script execution. This is in CSV format.

After the script is executed, the warehouse data gets downloaded in CSV format at the specified output location.

Execute the snowflake_query.py script with the required arguments to download the warehouse parameters data. Refer to the following list of arguments for more details:

python3 /opt/script/snowflake_query.py --user '${user}' --password '${password}' --account '${account}' --warehouse '${warehouse}' --database '${db}' --schema '${schema}' --out '/opt/unravel' --role ${role}

For example:

python3 /opt/script/snowflake_query.py --user 'sf_user' --password 'sf_password' --account 'sf_account' --warehouse 'sf_warehouse' --database 'sf_source_database' --schema 'sf_schema' --out '/opt/download_path' --role sysadmin

Parameter	Description
`--user`	Specify the name of the Snowflake user for the script execution.
`--password`	Specify the Snowflake user account passcode.
`--schema`	Specify the name of the schema that must be used for the script execution.
`--role`	Specify the role of the user.
`--account`	Specify the Snowflake account, which will be used for the script execution.
`--out`	Specify the path to the output folder.
`--database`	Specify the database used for the script execution.

After the script is executed, the warehouse parameters data gets downloaded in CSV format at the specified output location.

Uploading metadata from the Snowflake account

Unravel provides download scripts that you can use to upload Snowflake metadata, warehouse data, and warehouse parameters data to the Unravel Snowflake account. The download scripts should be executed in SnowSQL with an account that has access to the following:

Creating a stage in the specified Snowflake database and schema
Creating file format in the specified snowflake database and schema.

Do the following to upload the Snowflake metadata:

Download the snowsql_upload_data.sql from this location:
https://github.com/unraveldata-org/snowflake-data-loader/tree/main/script
From the same location, execute the create table script: prepare_schema.sql. After the stored procedure is created, execute the stored procedure with the following command:
```
call prepare_replication_schema(DbName, SchemaName)
```

Execute the snowsql_upload_data.sql script to upload the metadata to Unravel with the required arguments. Refer to the following list of arguments for more details:

snowsql -f /opt/script/snowsql_upload_data.sql -d ${db} -s ${schema} -r ${role} -a ${account} -u ${user} -w ${warehouse} -o variable_substitution=true -o log_file=/opt/script/snowsql_upload_data.log --variable path=${path} --variable stage_name=unravel_stage_upload --variable file_format=unravel_file_format_upload

snowsql -f /opt/script/snowsql_upload_data.sql -d unravel_sf_database -s unravel_sf_schema -r unravel_sf_role -a unravel_sf_account -u unravel_sf_user -w unravel_sf_warehouse -o variable_substitution=true -o log_file=/opt/script/snowsql_upload_data.log --variable path=/opt/download_path/ --variable stage_name=unravel_stage_upload --variable file_format=unravel_file_format_upload

Parameter	Description
-o variable_substitution	Enable the variable substitution switch in the script. Some variables are used in the script for which the values must be passed from the CLI. Set this to true.
-o log_file	Specify the name of the log file that must be generated on command execution.
--variable path	Specify the local path to where the snowflake system metadata is downloaded.
--variable stage_name	Specify the stage name, which is used to keep the temporary files for upload.
--variable file_format	Specify the file format name, which is used by the upload scripts.
-f	Specify the script file that you want to execute.
-d	Specify the Unravel provided Trial Snowflake Account database name.
-u	Specify the Unravel provided Trial Snowflake Account User name
-s	Specify the Unravel provided Trial Snowflake Account Schema name.
-r	Unravel provided Trial Snowflake Account Role
-a	Specify the Unravel provided Trial Snowflake Account name
-o	Provide the output-related arguments to get the logs in the specified path and format.

After the script is executed, the Snowflake metadata gets uploaded in the specified format to the selected location.

2. Configuring Snowflake settings in Unravel

The following configurations must be set in the Unravel Snowflake account. For this, you must sign in to Unravel using the login credentials you have received in the email.

Access the Unravel UI using the login credentials you received in the email when you created the Unravel Snowflake free account. The data fields are empty when you access Unravel UI for the first time.
Click on the upper right side and select Snowflake configuration.

Enter the following details:

Field	Description
Snowflake account
User	Enter the username to access the Snowflake account.
Password	Enter the passcode to access the Snowflake account.
Account	Specify the Snowflake user account from where you want to get live data for monitoring.
Role	Specify a role associated with the user account for Unravel to access the correct Snowflake data warehouse.
Parameters
Database name	Specify the Snowflake database where Unravel can access the metadata.
Schema name	Specify the Snowflake schema in your database.
Warehouse name	Specify a warehouse for Unravel to run queries on your metadata (monitor and telemetry data). The minimum warehouse size required by Unravel is X-small.
Look back	Set how many days of metadata should be processed from when this configuration is saved. You can set the look-back period to a maximum of 180 days.
Set Snowflake credit cost
Credit cost	Set the USD cost of your Snowflake credits.

Click the Detect Metadata button to check the connection and table metadata access. After a successful connection and metadata access, you can start the data ingestion.
Click Start data ingestion. The data ingestion process begins. The data for up to 14 days gets ingested into the Unravel Snowflake account.
Click the Monitor data ingestion link. You can monitor the data ingestion status from the Data ingestion dashboard. All the dashboards will be visible under the Dashboard tab.

In this section:

Home

Installing Unravel for Snowflake

Prerequisites (Snowflake)

1. Downloading Unravel

Important

2. Deploying Unravel

Important

3. Installing Unravel for Snowflake

Note

Note

Notice

Tip

Notice

Note

Note

Note

Note

Note

Tip

4. Configure Unravel for Snowflake

Note

Note

Search results