Microsoft Azure Databricks
Before installing Unravel in Azure Databricks, check and ensure that the Unravel installation requirements are completed and follow the below instructions to install and configure Unravel:
1. Create Unravel VM, create Azure Databricks, and install Unravel Prerequisites on Azure VM
4. Install Unravel either with Interactive Precheck or manually
1. Create Unravel VM and Azure Databricks
Sign in to the Azure portal.
Select Virtual Machines > Add and enter the following information in the Basics tab:
Project Details
Subscription
Choose the applicable subscription.
Resource group
Create a new group or choose an existing one.
Instance Details
Virtual Machine Name:
The Unravel server name.
Region:
Select the Azure region.
Availability Options
Select
No infrastructure is redundancy required
.Image
Select the appropriate image. Both
Centos-based 7.x
+ andRed Hat Enterprise Linux 7.x
+ are supported.Size
Click Change Size. In the modal, select Memory-optimized image with at least
128 GB memory
andPremium Disk support
, for example, E16s_v3 in East US 2)Administrator account
Authentication type
Select
password
orSSH Key
.Username and Password
Enter your VM login information.
Inbound Port Rules
Public inbound ports
Select Allow selected ports.
Selected Inbound ports
Select both
HTTPS
andSSH
.Click Next: Disks > and enter the following information in the Disks tab.
Disk Options
OS disk type: Select
Premium SSD
.
Data Disk
Click Create and attach a new disk.
Caution
This disk is formatted, so do not choose the Attach an existing disk option.
Enter a Name.
Select Source type
None (empty disk)
.Set Size to at least 512 GiB.
Account type: Select
premium SSD
.
Click Next: Networking > and enter the following information:
Virtual network: Create a new one or choose an existing one.
Subnet: Create a new or choose an existing one.
Public IP: Create a new one or choose an existing one.
Select Inbound ports: Select
HTTPS
andSSH
.
Click Review + create. Your deployment is now created.
Select Go to Resource > Networking > Inbound port rules > Add inbound port rule and include the following ports.
Rule Name
Destination
Destination IP Address
Destination Port Ranges
Unravel_3000
IP Addresses
NIC Private IP
3000
Unravel_443
IP Addresses
NIC Private IP
443
Unravel_4043
IP Addresses
NIC Private IP
4043
Unravel_4443
IP Addresses
NIC Private IP
443
Click OK.
Select Create a resource > Azure Databricks > Create. Go directly to step#3 if you already have workspaces.
Select Workspace name, Subscription, Resource group, Location, and Pricing tier.
Review VNET Peering options to connect Databricks with Unravel VM.
2. Download Unravel
3. Deploy Unravel
4. Install Unravel
You can install Unravel either with Interactive Precheck or manually without Interactive Precheck.
Note
Unravel recommends installation with Interactive Precheck.
To install Unravel manually, do the following:
You can run the setup command to install Unravel.
The setup command allows you to do the following:
Runs Precheck automatically to detect possible issues that prevent a successful installation. Suggestions are provided to resolve issues. Refer to Precheck filters for the expected value for each filter.
Let you run extra parameters to integrate the database of your choice.
The setup command allows you to use a managed database shipped with Unravel or an external database. When you run the setup command run without any additional parameters, the Unravel managed PostgreSQL database is used. Otherwise, you can specify any of the following databases, which is supported by Unravel, with the setup command:
MySQL (Unravel managed as well as external MySQL database)
MariaDB (Unravel managed as well as external MariaDB database)
PostgreSQL (External PostgreSQL)
Refer to Integrate database for details.
Let you specify a separate path for the data directory other than the default path.
The Unravel data and configurations are located in the
data
directory. By default, the installer maintains the data directory under<Unravel installation directory>/data
. You can also change thedata
directory's default location by running additional parameters with the setup command. To install Unravel with the setup command.Provides more options for setup.
To install Unravel with the setup command, do the following:
After deploying the binaries, if you are the root user, switch to Unravel user.
su -
<unravel user>
Notice
Only the Unravel user who owns the installation directory should run the setup command to install Unravel.
Run setup command with any of the following databases (PostgreSQL, MySQL, MariaDB). Refer to setup options for all the additional parameters that you can run with the setup command.
Tip
Run --help with the setup command and any combination of the setup command for complete usage details.
<unravel_installation_directory>/unravel/versions/
<Unravel version>
/setup --helpRefer to Integrate database topic and complete the prerequisites before running the setup command with any other database other than Unravel managed PostgreSQL, which is shipped with the product. Extra parameters must be passed with the setup command when you use another database.
Optionally, if you want to provide a different data directory, you can pass an extra parameter (--data-directory) with the setup command as shown below:
<unravel_installation_directory>/unravel/versions/
<Unravel version>
/setup --enable-databricks --data-directory/the/data/directory
Similarly, you can configure separate directories for other unravel directories—contact support for assistance.
When you run the setup command, the Precheck utility, which identifies the issues that prevent a successful installation, is automatically run. Refer to Precheck filters list to view details of each item in the precheck run output.
Review and update Unravel Log Receiver (LR) endpoint. By default, this is set to local FQDN only visible to workspaces within the same network. If this is not the case, run the following to set the LR endpoint:
<unravel_installation_directory>/unravel/manager config databricks set-lr-endpoint
<hostname>
'' ## For example: /opt/unravel/manager config databricks set-lr-endpoint <hostname> ''After you run this command, you are prompted to specify the port number. Ensure to press ENTER and leave it empty.
Apply the changes.
<Unravel installation directory>/unravel/manager config apply <Unravel installation directory>/unravel/manager refresh databricks
Start all the services.
<unravel_installation_directory>/unravel/manager start
Check the status of services.
<unravel_installation_directory>/unravel/manager report
The following service statuses are reported:
OK: Service is up and running.
Not Monitored: Service is not running. (Has stopped or has failed to start)
Initializing: Services are starting up.
Does not exist: The process unexpectedly disappeared. A restart will be attempted ten times.
You can also get the status and information for a specific service. Run the manager report command as follows:
<unravel_installation_directory>/unravel/manager report <service> ## For example: /opt/unravel/manager report auto_action
Run the following steps to connect the Databricks cluster to Unravel.
Register workspace in Unravel.
Sign in to Unravel UI and from the upper right click
> Workspaces. The Workspaces Manager page is displayed.
Click Add Workspace and enter the following details.
Field
Description
Workspace Id
Databricks workspace ID, which can be found in the Databricks URL.
Workspace Name
Databricks workspace name, which can be found in the Databricks URL.
Instance (Region) URL
Regional URL where the Databricks workspace is deployed.
Tier
Select a subscription option: Standard or Premium.
Token
Personal access token to access Databricks REST APIs. Refer to Authentication using Databricks personal access tokens to create personal access tokens.
Note
Personal access tokens can be created with admin or non-admin roles.
Note
After you click the Add button, it will take around 2-3 minutes to register the Databricks Workspace with Unravel.
Add Unravel configuration to Databricks clusters using any of the following options:
global init script
Global init script applies the Unravel configurations at Workspace level. To set up global init scripts, open this file and follow the instructions.
cluster init script
Cluster init script applies the Unravel configurations at cluster level. To setup cluster init scripts from the cluster UI, do the following:
Go to Unravel UI, click Manage > Workspaces > Cluster configuration to get the configuration details.
Follow the instructions and update each cluster (Automated /Interactive) that you want to monitor with Unravel.
Add Unravel configuration to Databricks clusters. Go to Unravel UI and from the upper right click Manage
> Workspaces > Cluster configuration to get the configuration details. Follow the instructions and update every cluster (Automated /Interactive) in your workspace.
Tip
By default, the Ganglia metrics are enabled with Dcom.unraveldata.agent.metrics.ganglia_enabled property set to true.
Note
To add Unravel configurations to job clusters via API, refer How to setup cluster init scripts via cluster API.
Set additional configurations if required.
Configure the Workspace for Data page.
Optionally, you can run healthcheck, at this point, to verify that all the configurations and services are running successfully.
<unravel_installation_directory>/unravel/manager healthcheck
Healthcheck is run automatically on an hourly basis in the backend. You can set the healthcheck intervals and email alerts to receive the healthcheck reports.
Tip
The workspace setup can be done at any time and does not impact the running clusters or jobs.
Refer to Databricks FAQ.