Part 1: Installing Unravel on a separate Azure VM

This topic explains how to install Unravel on an Azure VM with required network setup to connect the Unravel VM to your Azure Databricks workspaces.

1. Provision an Azure VM for Unravel Server

Log into the Azure portal.
Select Virtual machines, and click + Add.
On the Basics tab, enter values for the following fields:
- Subscription: Select the subscription type.
- Resource Group: Select the resource group to associate with this VM. The VM inherits configurations for lifecycle, permissions, and policies from this group.
- Virtual machine name: Enter a name, using only alphanumeric characters, hypens ("-"), and underscores ("_"). The VM name you specify here also becomes the VM's hostname.
- Region: Select a data center region for this VM. Some VM types are unavailable in some regions.
- Availability options: Select your redundancy (durability) settings.
- Image: Select a compatible underlying operating system for the VM.
- Size (required): Select a VM type that meets Unravel's requirements.
- Select your VM's Authentication type.
  Tip
  Best practice is to authenticate using an SSH public key, which you can generate using ssh-keygen. Avoid any reserved names like "admin" for the username.
- Set Inbound Port Rules:
  If you plan to allow external access to Unravel UI, select Allow selected ports and then select HTTPS and SSH.
Click Next: Disks.
On the Disks tab, enter values for the following fields:
- OS disk type: For better performance in production, we recommend a Premium SSD because it tolerates higher IOPS. For a dev/test cluster, we suggest a Standard SSD.
- Advanced: We recommend using managed disks that have better performance and reliability.
- Data disks:
  If you don't have a disk ready, click Create and attach new disk. In the dialog box, specify the disk name, size in GiB (must meet Unravel's minimum requirements), and source type of "empty disk".
  Otherwise, click Attach an existing disk.
Click Next: Networking.
On the Networking tab, enter values for the following fields:
- Virtual network (required): Select the appropriate virtual network for your cluster(s).
- Subnet (required): Select a subnet with the appropriate address range based on the number of IPs you plan to have in your network.
- NIC network security group:
  - Set this to Basic.
  - Add an inbound security policy to allow SSH access and 443 access to the Unravel node.
  - The default security policy should allow all access within the VNET. Default rules start with a priority of 65000.
Click Review + create.
Click Create.
It takes about 2 minutes to create your VM.
When Azure completes the creation of your VM, click Go to resource.
Copy the VM's public IP address.
Open an SSH session to your VM's public IP address to verify that your IP address is reachable:
```
ssh -i ssh-key user@ip-address
```

Verify that eth0 on the new VM is bound to the private IP address shown in the Azure portal.

ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0d:3a:1b:c2:48
          inet addr:10.10.1.96

2. Configure the VM

Install ntpd, start it at boot time, and check whether time is accurate.
This is necessary in order to synchronize your VM's clock.
```
sudo su -
yum install ntp
ntpd -u ntp:ntp
```
Disable Security Enhanced Linux (SELinux) permanently. This is important because HDFS maintains replication in different nodes/racks, so setting firewall rules in SELinux will lead to performance degradation.
```
sudo setenforce Permissive
```
In /etc/selinux/config, set SELINUX=permissive to make sure the settings persist after reboot:
```
SELINUX=permissive
```
Install libaio.x86_64.
Libaio has a huge performance benefit over the standard POSIX asynchronous I/O facility because the operations are performed in the Linux kernel instead of as a separate user process.
```
sudo yum -y install libaio.x86_64
```
Install lzop.x86_64.
Hadoop requires LZO compression libraries.
```
sudo yum install lzop.x86_64
```

Disable the firewall and check your iptable rules.

sudo systemctl disable firewalld
sudo systemctl stop firewalld
sudo iptables -F
sudo iptables -L

Prepare the second disk (for example, /dev/sdc) with at least 500 GB that was configured previously on Azure portal. Use fdisk -l to check any 500GB disk without partition. This step requires root privilege.

sudo su -
 
 
# List all disks and partitions
# You should see one called "sdc" if you attached a 500-1000 GB disk.
fdisk -l
fdisk /dev/sdc
# p (list current partitions)
# n (new partition)
# p (primary)
# Keep accepting rest of default configs.
# w (save)
 
# Format the disk
/usr/sbin/mkfs -t ext4 /dev/sdc
 
 
mkdir -p /srv
 
DISKUUID=`/usr/sbin/blkid |grep ext4 |grep sdc | awk '{ print $2}' |sed -e 's/"//g'`
echo $DISKUUID
 
# Mount the disk on /srv
echo "${DISKUUID}    /srv   ext4 defaults  0 0" >> /etc/fstab
mount /dev/sdc1 /srv
 
# Verify the disk space
df -hT /srv
 
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sdc1      ext4  197G   61M  187G   1% /srv
 
 
# Set permissions for Unravel and symlink Unravel's directories to the /srv mount
mkdir -p /srv/local/unravel
chmod -R 755 /srv/local
ln -s /srv/local/unravel /usr/local/unravel
chmod 755 /usr/local/unravel

3. Install MySQL

Complete the [Before Installing Unravel RPM] steps in Install and configure MySQL for Unravel.

4. Install Unravel Server on the VM

Download the Unravel Server RPM.
Install the Unravel Server RPM:
```
sudo rpm -ivh unravel-version.rpm
```
This installation creates the following directories, databases, and users:
Directories: The installation creates /usr/local/unravel/ which contains the executables, scripts, and settings (/usr/local/unravel/etc/unravel.properties).
/etc/init.d/unravel_* contains scripts for controlling the Unravel services
/etc/init.d/unravel_all.sh can be used to manually stop, start, restart, and get the status of all daemons in the proper order.
Subsequent RPM upgrades don't change /usr/local/unravel/etc/unravel.properties because your site-specific properties are put into this file.
Users: User unravel is created if it does not already exist.
DB: The initial bundled Postgres database and other durable state are in /srv/unravel/. You can switch to an external RDS later. Unravel recommends an externally managed MySQL database for production use, such as Azure SQL DB.
Config: The master configuration file is in /usr/local/unravel/etc/unravel.properties
Logs: All logs are in /usr/local/unravel/logs/
Grant access to Unravel Server:
- By default, a Public IP should be assigned to the Unravel VM.
- Create a security policy that allows SSH access on to the Unravel VM through port 443.
- It is recommended that you use an SSH key to access the Unravel node.

5. Configure MySQL

Complete the [After Installing Unravel RPM] steps in Install and configure MySQL for Unravel.

6. Configure Unravel Server with basic options

Open an SSH session to the Unravel VM.

ssh -i ssh-private-key ssh-user@unravel-host

Set correct permissions on the Unravel configuration directory.

cd /usr/local/unravel/etc
sudo chown unravel:unravel *.properties
sudo chmod 644 *.properties

In /usr/local/unravel/etc/unravel.properties, add/modify the following properties:

Set general properties:

com.unraveldata.onprem=false
com.unraveldata.cluster.type=DB
com.unraveldata.python.enabled=false
com.unraveldata.tagging.enabled=true

Optionally modify the following properties:

com.unraveldata.spark.appLoading.delayForRetry=60000
com.unraveldata.databricks.http.conn.timeout=5
com.unraveldata.databricks.http.read.timeout=10

Allow network access to the Unravel VM from Databricks workspaces:

Assign public IP and add firewall rules to open the required ports as follows.

Databricks cluster nodes are created in their own virtual network "workers-vnet". And for these cluster nodes access to Unravel VM’s secured service endpoint, you need to open Unravel VM’s secured service endpoints to the public. This means Unravel secure services ports (3000, 4043, and 4443) need to be opened to the public on Unravel VM’s virtual network and then mapped to Unravel VM, by following the steps below.

Log into the Azure portal.
Click Virtual Machines service and select the Unravel VM.
Click Networking.
The networking settings of the Unravel VM are displayed. The settings include the virtual network, subnet, and security group which the VM is currently using.
Click Inbound port rule.

Add an inbound rule to the network security group to allow port 3000, 4043 and 4443 on the internal IP address of the Unravel VM.

In the screenshot below, the internal IP address of the Unravel VM is 10.10.1.63.

You should now see the following three inbound rules added to the network security group:

Rule name	Port	Protocol	Source	Destination
Allow_port_3000_to_Unravel_VM	3000	Any	Any	Unravel_VM_internal_IP
Allow_port_4043_to_Unravel_VM	4043	Any	Any	Unravel_VM_internal_IP
Allow_port_4443_to_Unravel_VM	4443	Any	Any	Unravel_VM_internal_IP

Unravel supports unsecured SSL/TLS communication between the Unravel VM, Databricks Sensor, and Btrace agents running on Databricks nodes. To enable unnsecured SSL communication, follow these steps.

Update properties related to the log receiver:

// HTTP server port
com.unraveldata.live.logreceiver.port (default: 4043)

// HTTPS server port (negative means disabled), with positive value the SSL is enabled
com.unraveldata.live.logreceiver.port.https (default: -1) 

// KeyStore file path
com.unraveldata.server.ssl.cert_path (default: null)

// KeyStore password
com.unraveldata.server.ssl.cert_password (default: null) 

// Log network activity. Use only for debugging purposes.
com.unraveldata.server.log_network_activity (default: false)

Note

If the log receiver's certificate changes, add the new certificate to BTrace and Hive Hook trust stores (see SSL/TLS support for BTrace and SSL/TLS support for Hive Hook).

Run the databricks_setup.sh script with the unsecured SSL flag (-e true):

Generate Databricks personal access token for all workspaces.
In the Generate New Token dialog, enter these settings:
- Workspace name
- Workspace ID
- Databricks instance
- Databricks personal access token
  Tip
  Unravel uses this token to authenticate and access Databricks REST API and DBFS CLI.
  For information on how to generate the token, see https://docs.databricks.com/api/latest/authentication.html#generate-a-token
  Unlike passwords, tokens expire and can be revoked. If a token expires, replace it with a new one and re-run databricks_setup.sh. To identify expired tokens, check the Unravel logs (look for "authentication errors") and Unravel UI (look for missing Databricks metadata).
Enable token-based authentication.
Install DBFS CLI on the Azure VM host.
Note
This CLI needs a Python version newer than 2.7.9.
Run databricks_setup.sh to install Unravel agents on Azure Databricks workspaces.

7. Start Unravel services

sudo /etc/init.d/unravel_all.sh restart

8. Log into Unravel UI

Run the echo command to find the URL for Unravel UI.
If you're using an SSH tunnel or HTTP proxy, you might need to make adjustments to the host/IP of the URL:
```
echo "http://(hostname -f):3000/"
```
Create an SSH tunnel to access the Azure VM for Unravel's TCP port 3000.
```
ssh -i ssh-private-key ssh-user@unravel-host -L 3000:127.0.0.1:3000
```
Using a supported web browser, navigate to http://127.0.0.1:3000 and log in as user admin with password unraveldata.

Congratulations! Unravel Server is up and running. Proceed to Part 2: Connecting Unravel to a Databricks cluster.

In this section:

Home

Part 1: Installing Unravel on a separate Azure VM

1. Provision an Azure VM for Unravel Server

Tip

2. Configure the VM

3. Install MySQL

4. Install Unravel Server on the VM

5. Configure MySQL

6. Configure Unravel Server with basic options

Note

Tip

Note

7. Start Unravel services

8. Log into Unravel UI

Search results