Microsoft Azure Databricks

Before installing Unravel in Azure Databricks, check and ensure that the Unravel installation requirements are completed and follow the below instructions to install and configure Unravel:

1. Create Unravel VM, create Databricks, and install DBFS CLI

Create Azure VM

Select Virtual Machines > Add and enter the following information in the Basics tab:

Project Details
Subscription	Choose the applicable subscription.
Resource group	Create a new group or choose an existing one.
Instance Details
Virtual Machine Name:	The Unravel server name.
Region:	Select the Azure region.
Availability Options	Select `No infrastructure is redundancy required`.
Image	Select the appropriate image. Both `Centos-based 7.x`+ and `Red Hat Enterprise Linux 7.x`+ are supported.
Size	Click Change Size. In the modal select Memory-optimized image with at least `128 GB memory` and `Premium Disk support`, for example, E16s_v3 in East US 2)
Administrator account
Authentication type	Select `password` or `SSH Key`.
Username and Password	Enter your VM login information.
Inbound Port Rules
Public inbound ports	Select Allow selected ports.
Selected Inbound ports	Select both `HTTPS` and `SSH`.

Click Next: Disks > and enter the following information in the Disks tab.
Disk Options
- OS disk type: Select Premium SSD.
Data Disk
- Click Create and attach a new disk.
  Caution
  This disk is formatted so do not choose Attach an existing disk.
  - Enter a Name.
  - Select Source typeNone (empty disk).
  - Set Size to at least 512 GiB.
  - Account type: Select premium SSD.
Click Next: Networking > and enter the following information:
- Virtual network: Create new or choose an existing one.
- Subnet: Create new or choose an existing one.
- Public IP: Create new or choose an existing one.
- Select Inbound ports: Select HTTPS and SSH.
Click Review + create. Your deployment is now created.
Select Go to Resource > Networking > Inbound port rules > Add inbound port rule and include the following ports.
Rule Name
Destination
Destination IP Address
Destination Port Ranges
Unravel_3000
IP Addresses
NIC Private IP
3000
Unravel_4043
IP Addresses
NIC Private IP
4043
Click OK.

Rule Name	Destination	Destination IP Address	Destination Port Ranges
Unravel_3000	IP Addresses	NIC Private IP	3000
Unravel_4043	IP Addresses	NIC Private IP	4043

Create Azure Databricks

Select Create a resource > Azure Databricks > Create. Go directly to step#3 if you already have workspaces.
Select Workspace name, Subscription, Resource group, Location, and Pricing tier.

Review VNET Peering options to connect Databricks with Unravel VM.

Databricks Workspace	Unravel VM	Region	VNET peering option
Deployed with VNET Injection	Deployed in the same VNET as Azure Workspace.	-	-
Deployed with VNET Injection	Deployed in different VNET from Azure Workspace.	Any Azure region	Create VNET Peering between the two VNETs
Deployed with Default VNET		Any Azure region	Create VNET Peering between the two VNETs

Install DBFS CLI

Install the Databricks File System (DBFS) command-line interface using the following commands:

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install python-pip
sudo pip install databricks-cli

Note

Test connectivity using databricks-cli. In case there are any errors (Error: ValueError: Timeout value connect was Timeout) , then reinstall databricks-cli using python virtualenv as follows:

pip install databricks-cli
yum install python3
virtualenv -p /usr/bin/python3 mypy3
source mypy3/bin/activate
pip install databricks-cli

2. Download Unravel

Important

Before you download Unravel for your platform, ensure to get the username and password from Unravel Support.

Download the Unravel v4.7 package (RPM or Tar) using any of the following links:

Tar
https://preview.unraveldata.com/unravel/RPM/4.7.0/unravel-4.7.0.0-cloud.tar.gz
md5sum:
Note
You require root permissions to download and deploy the RPM package.
RPM
https://preview.unraveldata.com/unravel/RPM/4.7.0/unravel-4.7.0.0-cloud.rpm
md5sum:

For example:

Tar

curl -v https://preview.unraveldata.com/unravel/tar/4.7.0/unravel-4.7.0.0-cloud.tar.gz -o unravel-4.7.0.0-cloud.tar.gz -u username:password

RPM

curl -v https://preview.unraveldata.com/unravel/RPM/4.7.0/unravel-4.7.0.0-cloud.rpm -o unravel-4.7.0.0-cloud.rpm -u username:password

You can refer to Download section for the complete list of Unravel product downloads.

3. Deploy Unravel

Unravel binaries are available as a tar file or RPM package. You can deploy the Unravel binaries in any directory on the server. However, the user who installs Unravel must have write permissions to the directory where the Unravel binaries are deployed.

After the Unravel binaries are deployed, the directory layout for both the Tar and RPM will be unravel/versions/<Directories and files>. The binaries are deployed to <Unravel_installation_directory> and Unravel will be available in <Unravel_installation_directory/unravel.

Option 1: Deploy Unravel from a tar file
The following steps, to deploy Unravel from a tar file, should be performed by a user, who will run Unravel.
1. Create an Installation directory.
```
mkdir /path/to/installation/directory
## For example: mkdir /opt/unravel
```
  Note
  Some locations may require root access to create a directory. In such a case, after the directory is created, change the ownership to unravel user and continue with the installation procedure as the unravel user.
```
chown -R username:groupname /path/to/installation/directory
## For example:chown -R unravel:unravelgroup /opt/unravel
```
2. Extract and copy the Unravel tar file to the installation directory, which was created in the first step. After you extract the contents of the tar file, unravel directory is created within the installation directory.
```
tar zxf unravel-<version>tar.gz -C /path/to/installation/directory
## For example: tar zxf unravel-4.7.0.0.tar.gz -C /opt
## The unravel directory will be available within /opt
```
Option 2: Deploy Unravel from an RPM package
Important
The following steps to deploy Unravel from an RPM package should be performed by a root user. After the RPM package is deployed, the remaining installation procedures should be performed by the unravel user.
1. Create an installation directory.
```
mkdir /usr/local/unravel
```
2. Grant ownership of the directory to a user who will run Unravel. This user executes all the processes involved in Unravel installation.
```
chown -R username:groupname /usr/local/unravel
## For example: chown -R unravel:unravelgroup /usr/local/unravel
```
3. Run the following command:
```
rpm -i unravel-<version>.rpm
## For example: rpm -i unravel-4.7.0.0.rpm 
## The unravel directory will be available in /usr/local
```
  In case you want to provide a different location, you can do so by using the --prefix command. For example:
```
mkdir /opt/unravel
chown -R username:groupname /opt/unravel
rpm -i unravel-4.7.0.0.rpm --prefix /opt

## The unravel directory will be available in /opt
```
4. Continue with the installation procedures as unravel user.

4. Run setup

You can run the setup command to install Unravel. The setup command allows you to do the following:

Runs Precheck automatically to detect possible issues that prevent a successful installation. Suggestions are provided to resolve issues. Refer to Precheck filters for the expected value for each filter.
Let you run extra parameters to integrate the database of your choice.
The setup command allows you to use a managed database shipped with Unravel, or an external database. When run without any additional parameters, setup uses the Unravel managed PostgreSQL database. Otherwise, you can specify one of the following types of databases in the setup command:
- MySQL (Unravel managed as well as external MySQL database)
- MariaDB (Unravel managed as well as external MariaDB database)
- PostgreSQL (External PostgreSQL)
Refer to Integrate database for details.
Let you specify a separate path for the data directory other than the default path.
The Unravel data and configurations are located in the data directory. By default, the installer maintains the data directory under <Unravel installation directory>/data. You can also change the data directory's default location by running additional parameters with the setup command. To install Unravel with the setup command.
Provides more setup options.

Notice

The Unravel user who owns the installation directory should run the setup command to install Unravel.

To install Unravel with the setup command, do the following:

After deploying the binaries, if you are the root user, switch to Unravel user.
```
  su - <unravel user>
```

Run setup command:

Note

Refer to setup Options for all the additional parameters that can be run with the setup command

Refer to Integrate database topic and complete the pre-requisites before running the setup command with any other database other than Unravel managed PostgreSQL, which is shipped with the product. Extra parameters must be passed with the setup command when you use another database.

Tip

Optionally, if you want to provide a different data directory, you can pass an extra parameter (--data-directory) with the setup command as shown below:

<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --data-directory /the/data/directory

Similarly, you can configure separate directories for other unravel directories. Contact support for assistance.

PostgreSQL

Unravel managed PostgreSQL

<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-databricks

External PostgreSQL

<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-databricks --external-database postgresql <HOST> <PORT> <SCHEMA> <USERNAME> <PASSWORD>/

##The HOST, PORT, SCHEMA, USERNAME, PASSWORD are optional fields and are prompted if missing.

##For example:
/opt/unravel/versions/abcd.992/setup --enable-databricks --external-database postgresql xyz.unraveldata.com 5432 unravel_db_prod unravel unraveldata

MySQL

Unravel managed MySQL

<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-databricks --extra /tmp/mysql

External MySQL

<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-databricks --extra /tmp/<MySQL-directory> --external-database mysql <HOST> <PORT> <SCHEMA> <USERNAME> <PASSWORD>

##The HOST, PORT, SCHEMA, USERNAME, PASSWORD are optional fields and are prompted if missing.

MariaDB

Unravel managed MariaDB

<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --enable-databricks  --extra /tmp/mariadb

External MariaDB

<unravel_installation_directory>unravel/versions/<Unravel version>/setup --enable-databricks  --extra /tmp/<MariaDB-directory> --external-database mariadb <HOST> <PORT> <SCHEMA> <USERNAME> <PASSWORD>

##The HOST, PORT, SCHEMA, USERNAME, PASSWORD are optional fields and are prompted if missing.

Precheck is automatically run when you run the setup command. Refer to Precheck filters for expected value for each filter.

Start all the services.

<unravel_installation_directory>/unravel/manager start

Check the status of services.
```
<unravel_installation_directory>/unravel/manager report 
```
The following service statuses are reported:
- OK: Service is up and running.
- Not Monitored: Service is not running. (Has stopped or has failed to start)
- Initializing: Services are starting up.
- Does not exist: The process unexpectedly disappeared. Restarts will be attempted 10 times.
You can also get the status and information for a specific service. Run the manager report command as follows:
```
<unravel_installation_directory>/unravel/manager report <service> 
## For example: /opt/unravel/manager report auto_action
```

Precheck

The Precheck output displays the issues that prevent a successful installation and also provides suggestions to resolve them. You must resolve each of the issues before proceeding. See Precheck filters.

After the prechecks are resolved, you must re-login or reload the shell to execute the setup command again.

Here is a sample of the Precheck run result:

/opt/unravel/versions/abcd.1004/setup 
2021-04-05 15:51:30 Sending logs to: /tmp/unravel-setup-20210405-155130.log
2021-04-05 15:51:30 Running preinstallation check...
2021-04-05 15:51:31 Gathering information ................. Ok
2021-04-05 15:51:51 Running checks .................. Ok
--------------------------------------------------------------------------------
system
 Check limits        : PASSED
 Clock sync          : PASSED
 CPU requirement     : PASSED, Available cores: 8 cores
 Disk access         : PASSED, /opt/unravel/versions/develop.1004/healthcheck/healthcheck/plugins/system is writable
 Disk freespace      : PASSED, 229 GB of free disk space is available for precheck dir.
 Kerberos tools      : PASSED
 Memory requirement  : PASSED, Available memory: 79 GB
 Network ports       : PASSED
 OS libraries        : PASSED
 OS release          : PASSED, OS release version: centos 7.6
 OS settings         : PASSED
 SELinux             : PASSED
--------------------------------------------------------------------------------
Healthcheck report bundle: /tmp/healthcheck-20210405155130-xyz.unraveldata.com.tar.gz
2021-04-05 15:51:53 Prepare to install with: /opt/unravel/versions/abcd.1004/installer/installer/../installer/conf/presets/default.yaml
2021-04-05 15:51:57 Sending logs to: /opt/unravel/logs/setup.log
2021-04-05 15:51:57 Instantiating templates ................................................................................................................................................................................................................................ Ok
2021-04-05 15:52:05 Creating parcels .................................... Ok
2021-04-05 15:52:20 Installing sensors file ............................ Ok
2021-04-05 15:52:20 Installing pgsql connector ... Ok
2021-04-05 15:52:22 Starting service monitor ... Ok
2021-04-05 15:52:27 Request start for elasticsearch_1 .... Ok
2021-04-05 15:52:27 Waiting for elasticsearch_1 for 120 sec ......... Ok
2021-04-05 15:52:35 Request start for zookeeper .... Ok
2021-04-05 15:52:35 Request start for kafka .... Ok
2021-04-05 15:52:35 Waiting for kafka for 120 sec ...... Ok
2021-04-05 15:52:37 Waiting for kafka to be alive for 120 sec ..... Ok
2021-04-05 15:52:42 Initializing pgsql ... Ok
2021-04-05 15:52:46 Request start for pgsql .... Ok
2021-04-05 15:52:46 Waiting for pgsql for 120 sec ..... Ok
2021-04-05 15:52:47 Creating database schema ................. Ok
2021-04-05 15:52:50 Generating hashes .... Ok
2021-04-05 15:52:52 Loading elasticsearch templates ............ Ok
2021-04-05 15:52:55 Creating kafka topics .................... Ok
2021-04-05 15:53:36 Creating schema objects ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... Ok
2021-04-05 15:54:03 Request stop ....................................................... Ok
2021-04-05 15:54:16 Done
[unravel@xyz ~]$

Note

In certain situations, you can skip the precheck using the setup --skip-precheck.

For example:

/opt/unravel/versions/<Unravel version>/setup --skip-precheck

You can also skip the checks that you know can fail. For example, if you want to skip the Check limits option and check_network_ports, run the setup command as follows:

setup --filter-precheck ~check_limits,~check_network_ports

Tip

Run --help with the setup command and any combination of the setup command for complete usage details.

<unravel_installation_directory>/unravel/versions/<Unravel version>/setup --help

Precheck filters

Filters	Description	Expected Value
System
Check uptime	Verifies the period since the last server reboot.	>24h
Clock sync	Verifies if the clock synchronization service is running on the server.	The clock synchronization service is up and running.
CPU requirement	Verifies if the server has enough CPUs to run Unravel efficiently.	Check requirements.
Memory requirement	Verifies that the server has enough memory to run Unravel efficiently.	Check requirements.
Disk access	Verifies that the user who runs unravel has access to the configured disk locations.	Unravel users can access the configured disk locations.
Disk Freespace	Verifies if the disk locations have enough free space.	Check requirements
Kerberos tools	Verifies that the Kerberos tools are available on the server to support kerberized environments.	Kerberos tools are installed.
Network ports	Verifies that the network ports used by Unravel are available.	Check requirements
OS libraries	Verifies the required libraries if run with Unravel managed MySQL.	The following packages must be installed for fulfilling the OS level requirements for MySQL: `numactl-libs` (for libnuma.so) `libaio` (for libaio.so)
OS release	Verifies that the OS distribution is supported.	Check compatibility matrix
OS settings	Verifies vm.max_map_count recommended.	Check requirements
SELinux	Verifies if the SELinux status is enabled or not and provides in which mode it is(Permissive, Disabled, Enforcing).	Check product documentation.
Check limits	Verifies that user limits are set to values	Check requirements
Hadoop
Clients	Ensure that the following Hadoop clients installed and configured on the server: Apache Hadoop Hadoop Distributed File System (HDFS) Apache Hadoop Yarn Apache Hive Apache Beeline	Check compatibility matrix
Distribution	Verifies that the Hadoop distribution is a supported version.	Check compatibility matrix
RM HA Enabled/Disabled	Verifies if RM is running in HA mode
Healthcheck report bundle	healthcheck report tarball. This report provides the summary and information gathered by the healthcheck with the location.

Setup Options

setup Options	Description
-h, --help	Shows help for setup.
--config CONFIG	Specify a different path to the configuration file. <unravel_installation_directory>/unravel/versions/`<Unravel version>`/setup --config `path/to/config/directory`
--enable-core	Enables multi-cluster core node support.
--cluster-access	Enables cluster access, main unravel node.
--data-forwarder host:port cluster-type cluster-id	Data forwarder, main unravel node.
--data-directory	Specify a different path to the data directory.
--external-database [param [param ...]]	Enable external database.
--external-database-ssl	Enable external database with SSL.
--log-file	Setup log file location. Default is `/tmp/unravel-setup-YYYYMMDD-HHMMSS.log`.
--extra DIR, -e DIR	Specify extra packages location.
--precheck	Run the preinstallation check.

5. Connect Databricks cluster to Unravel

Create a Workspace token in Databricks.
1. Go to Workspace > Admin Console > Access Control and enable Personal Access Tokens. For more details, refer to Manage personal access tokens.
2. Go to Workspace > User Settings > Access Tokens and click Generate New Token. For more details, refer to Authentication using Databricks personal access tokens.

Register a new Databricks workspace or edit details of an existing Databricks workspace, using one of the following options.

Option 1: From Unravel UI

Go to the Unravel Instance UI and click Workspaces. The Workspaces Manager page is displayed.

Click Add Workspace and enter the following details.

Field	Description
Workspace Id	Databricks workspace ID.
Workspace Name	Databricks workspace name.
Instance (Region) URL	Regional URL where the Databricks workspace is deployed.
Tier	Select a subscription option: Standard or Premium.
Token	Personal access token to authenticate to and access Databricks REST APIs. Refer to Authentication using Databricks personal access tokens to create personal access tokens.

Note

After you click the Add button, it will take around 2-3 minutes to register the Databricks Workspace with Unravel.

Option 2: From command-line using manager command.

Stop Unravel

<Unravel installation directory>/unravel/manager stop

Switch to Unravel user.

Add the workspace details using the manager command as follows from the Unravel installation directory:

source <path-to-python3-virtual environment-dir>/bin/activate
<Unravel_installation_directory>/unravel/manager config databricks add --id <workspace-id> --name <workspace-name> --instance <workspace-instance> --access-token <workspace-token> --unravel-endpoint <unravel_server:port> 

##For example:
/opt/unravel/manager config databricks add --id 0000000000000000 --name myworkspacename --instance https://adb-0000000000000000.16.azuredatabricks.net --access-token xxxx --unravel-endpoint 127.0.0.1:4043 --tier premium

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start Unravel

<Unravel installation directory>/unravel/manager start

Set the cluster configurations. On the Unravel Instance UI, click Manage > Workspaces > Cluster configuration to configure the Databricks cluster with Unravel. The following configurations must be updated for every cluster (Automated /Interactive) in your workspace.

Configure the Workspace for Data page.

Ensure that at least one of the workspace is populated before you configure a workspace for the Data page.

To configure the Databricks for Data page, do the following:

Stop Unravel

<Unravel installation directory>/unravel/manager stop

Set the following property.

<Unravel installation directory>/unravel/manager config properties set hive.metastore.<X>.workspace.ids <Comma-separated list of Databricks workspaces>

##Replace <X> with  the metastore variables listed in com.unraveldata.hive.metastore.list.

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start Unravel

<Unravel installation directory>/unravel/manager start

Set additional configurations if required.
Optionally, you can run healthcheck, at this point, to verify that all the configurations and services are running successfully.
```
<unravel_installation_directory>/unravel/manager healthcheck
```
Healthcheck is run automatically on an hourly basis in the backend. You can set the healthcheck intervals and email alerts to receive the healthcheck reports.

In this section:

Home

Microsoft Azure Databricks

1. Create Unravel VM, create Databricks, and install DBFS CLI

Caution

Note

2. Download Unravel

Important

Note

3. Deploy Unravel

Note

Important

4. Run setup

Notice

Note

Tip

Note

Tip

5. Connect Databricks cluster to Unravel

Note

Search results