Databricks FAQ

Home

Databricks FAQ

How do I verify if a workspace is registered with Unravel?

Sign in to Unravel.
Click Manager > Workspace and check if the corresponding Databricks workspace is shown in the Workspace list.

How do I verify if a cluster has Unravel configurations?

Check if Spark conf, logging, and init scripts are present.
For Spark conf, check if each and every property is correct.
Refer Unravel > Workspace Manager > Cluster Configurations.

How do I check if a cluster has the correct Unravel configuration?

Check if content in Spark conf, logging, and init scripts are correct.
Refer Unravel > Workspace Manager > Cluster Configurations.

How to check if the Log Receiver (LR) endpoint is valid?

Refer Unravel > Workspace Manager > Cluster Configurations.
Check if spark.unravel.server.hostport is a valid address.

How to verify if the LR endpoint is accessible from Databricks?

Check if the host/IP is accessible from the Databricks notebook.
```
%sh
nslookup 10.2.1.4
```
Check if port 4043 is open for outgoing traffic on Databricks.
```
%sh
nc -zv 10.2.1.4 4043
```
Check if port 4043 is open for incoming traffic on Unravel.
```
<see administrator>
```

How to verify if a workspace is accessible from Unravel?

Check if port 443 is open for outgoing traffic on Unravel

curl -X GET -H "Authorization: Bearer <token-here>" 'https://<instance-name-here>/api/2.0/dbfs/list?path=dbfs:/'

How do I check if the workspace token is valid in Unravel?

Access the unravel.properties file in Unravel and get the workspace token.

Run the following to check if the token is valid and works:

curl -X GET -H "Authorization: Bearer <token>" 'https://<instance-name>/api/2.0/dbfs/list?path=dbfs:/'

How do I verify if the workspace token is valid in Databricks?

Run the following to get workspace token from unravel_db.properties file from DBFS.

dbfs cat  
dbfs:/databricks/unravel/unravel-db-sensor-archive/etc/unravel_db.properties

Following is a sample of the output:

#Thu Sep 30 20:06:42 UTC 2021
unravel-server=01.0.0.1\:4043
databricks-instance=https\://abc-0000000000000.18.azuredatabricks.net
databricks-workspace-name=DBW-xxx-yyy
databricks-workspace-id=<workspaceid>
ssl_enabled=False
insecure_ssl=True
debug=False
sleep-sec=30
databricks-token=<databricks-token>

Check if the token is valid using the following command:

curl -X GET -H "Authorization: Bearer <token>" 'https://<instance-name>/api/2.0/dbfs/list?path=dbfs:/'

If the token is invalid, you can regenerate the token and update the workspace from Unravel UI > Manage > Workspace.

How do I verify if the workspace token is valid in Databricks?

You can register a workspace in Unravel from the command line with the manager command.

Stop Unravel

<Unravel installation directory>/unravel/manager stop

Switch to Unravel user.

Add the workspace details using the manager command as follows from the Unravel installation directory:

source <path-to-python3-virtual environment-dir>/bin/activate
<Unravel_installation_directory>/unravel/manager config databricks add --id <workspace-id> --name <workspace-name> --instance <workspace-instance> --access-token <workspace-token> --tier <tier_option>

##For example:
/opt/unravel/manager config databricks add --id 0000000000000000 --name myworkspacename --instance https://adb-0000000000000000.16.azuredatabricks.net --access-token xxxx --tier premium

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start Unravel

<Unravel installation directory>/unravel/manager start

How to set up global init scripts? (Unravel v4760 onwards)

Global init script applies the Unravel configurations to all clusters in a workspace. Do the following to set up Unravel configuration as Global init scripts.

If upgrading from a previous version of Unravel, you must remove all the existing scripts such as unravel_cluster_init.sh, unravel_spark_init.sh, etc.
On Databricks, go to Workspace > Settings > Admin Console > Global init scripts.

Click +Add and set the following:

Item	Settings
Name	Enter the name as unravel_init
Script	Copy and paste the following content in the Script box: #!/bin/bash # # Runs Unravel Init scripts COUNTER=1 while [ ! -d "/dbfs" ] && [ $COUNTER -le 20 ]; do echo "$(date) Waiting for dbfs mount: RetryCount = ${COUNTER} ....." ((COUNTER++)) sleep 0.1 done UD_ROOT=/dbfs/databricks/unravel CLUSTER_INIT=${UD_ROOT}/unravel-db-sensor-archive/dbin/unravel_cluster_init.sh SPARK_INIT=${UD_ROOT}/unravel-db-sensor-archive/dbin/unravel_spark_init.sh if [ ! -f "${CLUSTER_INIT}" ]; then echo "Unravel Cluster Init ${CLUSTER_INIT} doesn't exist!" exit 0 else cp ${CLUSTER_INIT} /tmp/ chmod a+x /tmp/unravel_cluster_init.sh /tmp/unravel_cluster_init.sh fi if [ ! -f "${SPARK_INIT}" ]; then echo "Unravel Spark Init ${SPARK_INIT} doesn't exist!" exit 0 else cp ${SPARK_INIT} /tmp/ chmod a+x /tmp/unravel_spark_init.sh /tmp/unravel_spark_init.sh fi Note Unravel supports Databricks version 11.3 and below. Newer versions can be included by setting the environment variable `DATABRICKS_RUNTIME_VERSION` at the top of this script
Enabled	Turn on the Enable toggle.

Item

Settings

Name

Enter the name as unravel_init

Script

Copy and paste the following content in the Script box:

#!/bin/bash
#
# Runs Unravel Init scripts

COUNTER=1
while [ ! -d "/dbfs" ] && [ $COUNTER -le 20 ]; do
 echo "$(date) Waiting for dbfs mount: RetryCount = ${COUNTER} ....."
 ((COUNTER++))
 sleep 0.1
done

UD_ROOT=/dbfs/databricks/unravel
CLUSTER_INIT=${UD_ROOT}/unravel-db-sensor-archive/dbin/unravel_cluster_init.sh
SPARK_INIT=${UD_ROOT}/unravel-db-sensor-archive/dbin/unravel_spark_init.sh

if [ ! -f "${CLUSTER_INIT}" ]; then
 echo "Unravel Cluster Init ${CLUSTER_INIT} doesn't exist!"
 exit 0
else
 cp ${CLUSTER_INIT} /tmp/
 chmod a+x /tmp/unravel_cluster_init.sh
 /tmp/unravel_cluster_init.sh
fi

if [ ! -f "${SPARK_INIT}" ]; then
 echo "Unravel Spark Init ${SPARK_INIT} doesn't exist!"
 exit 0
else
 cp ${SPARK_INIT} /tmp/
 chmod a+x /tmp/unravel_spark_init.sh
 /tmp/unravel_spark_init.sh
fi

Note

Unravel supports Databricks version 11.3 and below. Newer versions can be included by setting the environment variable DATABRICKS_RUNTIME_VERSION at the top of this script

Enabled

Turn on the Enable toggle.

Click Add to save the settings.

Note

Cluster logging should be enabled at the cluster level. See Logging in for instructions.

How to set up global init scripts? (Unravel v4750)

Global init script applies the Unravel configurations to all clusters in a workspace. Do the following to set up Unravel configuration as Global init scripts.

On Databricks, go to Workspace > Settings > Admin Console > Global init scripts.
If you are upgrading from a previous version of Unravel, delete all the existing init scripts; unravel_cluster_init.sh, unravel_spark_init.sh.

Click +Add and set the following:

Item	Settings
Name	Enter the name as unravel_init.
Script	Copy and paste the following content in the Script box: #!/bin/bash # # Runs Unravel Init scripts COUNTER=1 while [ ! -d "/dbfs" ] && [ $COUNTER -le 20 ]; do echo "$(date) Waiting for dbfs mount: RetryCount = ${COUNTER} ....." ((COUNTER++)) sleep 0.1 done UD_ROOT=/dbfs/databricks/unravel CLUSTER_INIT=${UD_ROOT}/unravel-db-sensor-archive/dbin/unravel_cluster_init.sh SPARK_INIT=${UD_ROOT}/unravel-db-sensor-archive/dbin/unravel_spark_init.sh if [ ! -f "${CLUSTER_INIT}" ]; then echo "Unravel Cluster Init ${CLUSTER_INIT} doesn't exist!" exit 0 else cp ${CLUSTER_INIT} /tmp/ chmod a+x /tmp/unravel_cluster_init.sh /tmp/unravel_cluster_init.sh fi if [ ! -f "${SPARK_INIT}" ]; then echo "Unravel Spark Init ${SPARK_INIT} doesn't exist!" exit 0 else cp ${SPARK_INIT} /tmp/ chmod a+x /tmp/unravel_spark_init.sh /tmp/unravel_spark_init.sh fi
Enabled	Turn on the Enable toggle.

Item

Settings

Name

Enter the name as unravel_init.

Script

Copy and paste the following content in the Script box:

#!/bin/bash
#
# Runs Unravel Init scripts

COUNTER=1
while [ ! -d "/dbfs" ] && [ $COUNTER -le 20 ]; do
 echo "$(date) Waiting for dbfs mount: RetryCount = ${COUNTER} ....."
 ((COUNTER++))
 sleep 0.1
done

UD_ROOT=/dbfs/databricks/unravel
CLUSTER_INIT=${UD_ROOT}/unravel-db-sensor-archive/dbin/unravel_cluster_init.sh
SPARK_INIT=${UD_ROOT}/unravel-db-sensor-archive/dbin/unravel_spark_init.sh

if [ ! -f "${CLUSTER_INIT}" ]; then
 echo "Unravel Cluster Init ${CLUSTER_INIT} doesn't exist!"
 exit 0
else
 cp ${CLUSTER_INIT} /tmp/
 chmod a+x /tmp/unravel_cluster_init.sh
 /tmp/unravel_cluster_init.sh
fi

if [ ! -f "${SPARK_INIT}" ]; then
 echo "Unravel Spark Init ${SPARK_INIT} doesn't exist!"
 exit 0
else
 cp ${SPARK_INIT} /tmp/
 chmod a+x /tmp/unravel_spark_init.sh
 /tmp/unravel_spark_init.sh
fi

Enabled

Turn on the Enable toggle.

Click Add to save the settings.

Note

Cluster logging should be enabled at the cluster level. See Logging in Cluster init script for instructions.

How to set up global init scripts? (Unravel v4740)

Global init script applies the Unravel configurations to all clusters in a workspace. Do the following to set up Unravel configuration as Global init scripts.

On Databricks, go to Workspace > Settings > Admin Console > Global init scripts.
Click +Add and set the following:
Item
Settings
Name
Set to unravel_cluster_init.
Script
Copy the contents from unravel_cluster_init.sh.
Enabled
Set to True.
Click Add to save the settings.

Item	Settings
Name	Set to unravel_cluster_init.
Script	Copy the contents from unravel_cluster_init.sh.
Enabled	Set to True.

Click +Add again and set the following:

Item	Settings
Name	Set to unravel_spark_init.
Script	Copy the contents from unravel_spark_init.sh. Note This script supports up to Databricks 10.4 (latest at the time). In case a new DBR is available, test and update the condition as applicable.
Enabled	Set to True.

Click Add to save the settings.

How effective the performance will be with Global init scripts?

Same as the Cluster init scripts.

How to handle performance issues that occur while using the Global init script?

Disable the Global init script in case of issues.

How to set up cluster init scripts via cluster UI? (V4760 onwards)

The cluster init script applies the Unravel configurations for each cluster. To setup cluster init scripts from the cluster UI, do the following:

On Databricks, open a cluster and go to Advanced Options.

Edit the following settings:

Item	Settings
Logging	Set Destination to DBFS and copy and paste the following path in Cluster Log Path. `dbfs:/cluster-logs`
Init Scripts	Set Destination to DBFS, copy and paste the following in Init Script Path and then click Add. `dbfs:/databricks/unravel/unravel-db-sensor-archive/dbin/install-unravel.sh`

Item

Settings

Logging

Set Destination to DBFS and copy and paste the following path in Cluster Log Path.

dbfs:/cluster-logs

Init Scripts

Set Destination to DBFS, copy and paste the following in Init Script Path and then click Add.

dbfs:/databricks/unravel/unravel-db-sensor-archive/dbin/install-unravel.sh

Note

Cluster logging should be enabled at the cluster level. See Logging in Cluster init script for instructions.

How to set up cluster init scripts via cluster UI? (V4750)

The cluster init script applies the Unravel configurations for each cluster. To setup cluster init scripts from the cluster UI, do the following:

Go to Unravel UI, click Manage > Workspaces > Cluster configuration to get the configuration details.
Follow the instructions and update each cluster (Automated /Interactive) that you want to monitor with Unravel

How to setup cluster init scripts via cluster API (V4760)

To add Unravel configurations to job clusters via API, use the JSON format as follows:

{
    "settings": {
        "new_cluster": {
            "init_scripts": [
                {
                    "dbfs": {
                        "destination": "dbfs:/databricks/unravel/unravel-db-sensor-archive/dbin/install-unravel.sh"
                    }
                },
                ...
            ],
            "cluster_log_conf": {
                "dbfs": {
                    "destination": "dbfs:/cluster-logs"
                }
            },
            ...
        },
        ...
    }
}

Note

Cluster logging should be enabled at the cluster level. See Logging in Cluster init script for instructions.

How to setup cluster init scripts via cluster API (V4750)

To add Unravel configurations to job clusters via API, use the JSON format as follows:

{
    "settings": {
        "new_cluster": {
            "spark_conf": {
                // Note: If extraJavaOptions is already in use, prepend the Unravel values. Also, for Databricks Runtime with spark 2.x.x, replace "spark-3.0" with "spark-2.4"
                "spark.executor.extraJavaOptions": "-javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-3.0",
                "spark.driver.extraJavaOptions": "-javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-3.0",
 
                // rest of your spark properties here ...
                ... 
            },
            "init_scripts": [
                {
                    "dbfs": {
                        "destination": "dbfs:/databricks/unravel/unravel-db-sensor-archive/dbin/install-unravel.sh"
                    }
                },
 
                // rest of your init scripts here ...
                ...
            ],
            "cluster_log_conf": {
                "dbfs": {
                    "destination": "dbfs:/cluster-logs"
                }
            },
 
            // rest of your cluster properties here ...
            ...
        },
        ...g
    }
}

How to update instances and prices?

Follow the instructions in this file to update instances and prices.

How to apply for a discount on instance prices?

You can configure discounted prices for VMs and DBUs in Databricks using Unravel properties. Add the properties to set the discount as a percentage value.

Stop Unravel.

<Unravel installation directory>/unravel/manager stop

From the installation directory, set the following properties as follows:

<Unravel installation directory>/unravel/manager config properties set com.unraveldata.databricks.vm.discount.percentage 
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.databricks.dbu.discount.percentage 
##Example: 
/opt/unravel/manager config properties set com.unraveldata.databricks.vm.discount.percentage 10
/opt/unravel/manager config properties set com.unraveldata.databricks.dbu.discount.percentage 20.5

Apply the changes.

<Unravel installation directory>/unravel/manager config apply

Start Unravel.

<Unravel installation directory>/unravel/manager start

How to configure LR authentication for Databricks with TLS enabled?

Refer to Configuring Log Receiver (LR) authentication.

Would you like to provide feedback? Just click here to suggest edits.

Home

Databricks FAQ

Note

Note

Note

Note

Note

Note

Search results