Home

Running the Databricks_setup.sh script
Description

This topic explains how to configure Unravel for Databricks using /usr/local/unravel/bin/databricks_setup.sh on Unravel Server. This script initializes a given workspace and provides instructions required to set up a job cluster that for Unravel to monitor.

Important

Run this script as the same username that you used to install Unravel Server.

Syntax
./databricks_setup.sh [--add-workspace -i workspace-id -n workspace-name -s workspace-instance -t workspace-token -u unravel_server:port [options]] | [-h] | [-p -u unravel_server:port [options]]

Options:

Option

Description

--add-workspace | -a

Sets up or updates a Databricks workspace for monitoring by Unravel. Valid values:

-i

ID of the workspace to be configured.

-n

Workspace name.

-s

Workspace instance. Must start with https://.

For example, https://eastus.databricks.com

-t

Personal access token for the workspace.

-u

IP address and port of Unravel Server.

For example, 0.0.0.1:4043

-e

Enables/disables SSL for Databricks sensor and agent.

Valid values: true, false.

Default: false.

-c

(Optional) Enables/disables SSL connections to Unravel endpoints without certificates. This option is only in effect if -e is set to true.

Valid values: true, false.

Default: false.

-v

(Optional) Spark version to be used.

Default: 2.3.

-d

(Optional) Enables debug logs for Unravel Databricks sensor installation.

Default: false.

-m

(Optional) Specifies the frequency in seconds in which to poll cluster metrics.

Default: 30.

--switch-to-db | -t

Switch this cluster to Azure Databricks.

--print-spark-conf | -p

Print the minimal Spark configuration required to monitor cluster using Unravel.

-u

IP address and port of Unravel Server.

For example, 0.0.0.1:4043

-e

Enables/disables SSL for Databricks sensor and agent.

Valid values: true, false.

Default: false.

-c

(Optional) Enables/disables SSL connections to Unravel endpoints without certificates. This option is only in effect if -e is set to true.

Valid values: true, false.

Default: false.

-v

(Optional) Spark version to be used. Default: 2.3.

--help | -h

Prints the usage of this script.

Note

If you generate new tokens, re-run this script to update Unravel Server.

Note

After running this script, restart Unravel services:

service unravel_all.sh restart
Examples

To connect to a new workspace with SSL:

 $ ./usr/local/unravel/bin/databricks_setup.sh --add-workspace -i 755550310254240 -n Engineering -s https://eastus2.azuredatabricks.net -t dapi490444403de2c1169e4667a3a02a694a -u azurevm001.unraveldata.com:4443  -e true
Copying .. /tmp/7674350310254240.properties  to  dbfs:/databricks/unravel/unravel-db-sensor-archive/etc/unravel_db.properties
Copied file successfully
Copying .. /tmp/agent-pack  to  dbfs:/databricks/unravel/unravel-agent-pack-bin

Copied file successfully
Copying .. /tmp/sensor_pack  to  dbfs:/databricks/unravel/unravel-db-sensor-archive
Copied file successfully
switch_to_databricks action already done. No action taken
-----------------------------------
  Cluster Spark Configuration
-----------------------------------
spark.executor.extraJavaOptions -Dcom.unraveldata.client.rest.ssl.enabled=true -Dcom.unraveldata.ssl.insecure=true -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3
spark.eventLog.enabled true
spark.unravel.server.hostport azurevm001.unraveldata.com:4443
spark.driver.extraJavaOptions -Dcom.unraveldata.client.rest.ssl.enabled=true -Dcom.unraveldata.ssl.insecure=true -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,libs=spark-2.3
spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/
spark.unravel.shutdown.delay.ms 300

-----------------------------------
  Spark Submit Parameters
-----------------------------------
"--conf", "spark.executor.extraJavaOptions=-Dcom.unraveldata.client.rest.ssl.enabled=true -Dcom.unraveldata.ssl.insecure=true -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3",
"--conf", "spark.eventLog.enabled=true",
"--conf", "spark.unravel.server.hostport=azurevm001.unraveldata.com:4443",
"--conf", "spark.driver.extraJavaOptions=-Dcom.unraveldata.client.rest.ssl.enabled=true -Dcom.unraveldata.ssl.insecure=true -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,libs=spark-2.3",
"--conf", "spark.eventLog.dir=dbfs:/databricks/unravel/eventLogs/",
"--conf", "spark.unravel.shutdown.delay.ms=300"

-----------------------------------
  Databricks Cluster Init Script
-----------------------------------
dbfs:/databricks/unravel/unravel-db-sensor-archive/dbin/install-unravel.sh

-----------------------------------
  Restart unravel daemons
-----------------------------------
service unravel_all.sh restart
------------------------------------

To connect to a new workspace with no SSL:

/usr/local/unravel/bin/databricks_setup.sh --add-workspace -i 2982641413555431 -n Finance -s https://eastus.azuredatabricks.net -t dapiba64a058e8d2e07156288314305eqassw23 -u azurevm001.unraveldata.com:4043
Copying .. /tmp/2982641413551061.properties to dbfs:/databricks/unravel/unravel-db-sensor-archive/etc/unravel_db.properties
Copied file successfully
Copying .. /tmp/agent-pack to dbfs:/databricks/unravel/unravel-agent-pack-bin
Copied file successfully
Copying .. /tmp/sensor_pack to dbfs:/databricks/unravel/unravel-db-sensor-archive
Copied file successfully
switch_to_databricks action already done. No action taken
-----------------------------------
Cluster Spark Configuration
-----------------------------------
spark.executor.extraJavaOptions -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3
spark.eventLog.enabled true
spark.unravel.server.hostport azurevm001.unraveldata.com:4043
spark.driver.extraJavaOptions -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,libs=spark-2.3
spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/
spark.unravel.shutdown.delay.ms 300

-----------------------------------
Spark Submit Parameters
-----------------------------------
"--conf", "spark.executor.extraJavaOptions= -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3",
"--conf", "spark.eventLog.enabled=true",
"--conf", "spark.unravel.server.hostport=azurevm001.unraveldata.com:4043",
"--conf", "spark.driver.extraJavaOptions= -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,libs=spark-2.3",
"--conf", "spark.eventLog.dir=dbfs:/databricks/unravel/eventLogs/",
"--conf", "spark.unravel.shutdown.delay.ms=300"

-----------------------------------
Databricks Cluster Init Script
-----------------------------------
dbfs:/databricks/unravel/unravel-db-sensor-archive/dbin/install-unravel.sh

-----------------------------------
  Restart unravel daemons
-----------------------------------
service unravel_all.sh restart
------------------------------------

To print the configuration:

/usr/local/unravel/bin/databricks_setup.sh -p -u azurevm001.unraveldata.com:4443 -e true
-----------------------------------
Cluster Spark Configuration
-----------------------------------
spark.executor.extraJavaOptions -Dcom.unraveldata.client.rest.ssl.enabled=true -Dcom.unraveldata.ssl.insecure=true -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3
spark.eventLog.enabled true
spark.unravel.server.hostport azurevm001.unraveldata.com:4443
spark.driver.extraJavaOptions -Dcom.unraveldata.client.rest.ssl.enabled=true -Dcom.unraveldata.ssl.insecure=true -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,libs=spark-2.3
spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/
spark.unravel.shutdown.delay.ms 300

-----------------------------------
Spark Submit Parameters
-----------------------------------
"--conf", "spark.executor.extraJavaOptions=-Dcom.unraveldata.client.rest.ssl.enabled=true -Dcom.unraveldata.ssl.insecure=true -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3",
"--conf", "spark.eventLog.enabled=true",
"--conf", "spark.unravel.server.hostport=azurevm001.unraveldata.com:4443",
"--conf", "spark.driver.extraJavaOptions=-Dcom.unraveldata.client.rest.ssl.enabled=true -Dcom.unraveldata.ssl.insecure=true -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,libs=spark-2.3",
"--conf", "spark.eventLog.dir=dbfs:/databricks/unravel/eventLogs/",
"--conf", "spark.unravel.shutdown.delay.ms=300"

-----------------------------------
Databricks Cluster Init Script
-----------------------------------
dbfs:/databricks/unravel/unravel-db-sensor-archive/dbin/install-unravel.sh

-----------------------------------
Restart unravel daemons
-----------------------------------
service unravel_all.sh restart
------------------------------------
In this section: