Installing Databricks sensors using Setup script
Description
This topic explains how to configure Unravel for Databricks using the /usr/local/unravel/install_bin/databricks_setup.sh script on the Unravel server. This script deploys Unravel agent binaries and prints Databricks cluster configuration that is required for Unravel monitoring. Repeat this script for each workspace you want to initialize.
Important
Run this script as the same username that you used to install the Unravel server.
Syntax
usage: databricks_setup.sh --add-workspace -i<workspace-id>-n<workspace-name>-r<workspace-instance>-t<workspace-token>-u<unravel_server:port>[options] databricks_setup.sh --print-spark-conf -u<unravel_server:port>[options] databricks_setup.sh --help
Options:
Option | Description | |
|---|---|---|
| Sets up or updates a Databricks workspace for monitoring by Unravel. Valid values: | |
| Id of the workspace to be configured. | |
| Workspace name. | |
| Workspace instance. Must start with For example, | |
| Workspace access token. | |
| Unravel LR Endpoint. (e.g: 10.0.0.4:4043) | |
| (Optional) Workspace tier. Accepted values are - premium, standard. The default is premium. | |
| Enables/disables SSL for Databricks sensor and agent. Valid values: Default: | |
| (Optional) Enables/disables SSL connections to Unravel endpoints without certificates. This option is only in effect if Valid values: Default: | |
| (Optional) Spark version to be used. Default: | |
| (Optional) Enables debug logs for Unravel Databricks sensor installation. Default: | |
| (Optional) Specifies the frequency in seconds in which to poll cluster metrics. Default: | |
| Print the minimal Spark configuration required to monitor cluster using Unravel. | |
| Unravel server URL. For example, | |
| Enables/disables SSL for Databricks sensor and agent. Valid values: Default: | |
| (Optional) Enables/disables SSL connections to Unravel endpoints without certificates. This option is only in effect if Valid values: Default: | |
| (Optional) Spark version to be used. Default: | |
| Prints the usage of this script. | |
Note
If you generate new tokens, re-run this script to update Unravel Server.
Examples
To add/edit a workspace:
/usr/local/unravel/install_bin/databricks_setup.sh --add-workspace -i 1234567890 -n DemoWorkspace -t ***** -r https://eastus.azuredatabricks.net -u 10.1.2.3:4043 -p premium Deleting directory - dbfs:/databricks/unravel/unravel-agent-pack-bin Deleted directory successfully Deleting directory - dbfs:/databricks/unravel/unravel-db-sensor-archive Deleted directory successfully Creating directory - dbfs:/databricks/unravel/logs Created directory successfully Creating directory - dbfs:/databricks/unravel/eventLogs Created directory successfully Copying /tmp/unravel_db.properties to dbfs:/databricks/unravel/unravel-db-sensor-archive/etc/unravel_db.properties Copied file successfully Copying /tmp/agent-pack to dbfs:/databricks/unravel/unravel-agent-pack-bin Copied file successfully Copying /tmp/sensor_pack to dbfs:/databricks/unravel/unravel-db-sensor-archive Copied file successfully ----------------------------------- Cluster Spark Configuration ----------------------------------- spark.executor.extraJavaOptions -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3 spark.eventLog.enabled true spark.unravel.server.hostport 10.1.2.3:4043 spark.driver.extraJavaOptions -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-2.3 spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/ spark.unravel.shutdown.delay.ms 300 ----------------------------------- Spark Submit Parameters ----------------------------------- "--conf", "spark.executor.extraJavaOptions= -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3", "--conf", "spark.eventLog.enabled=true", "--conf", "spark.unravel.server.hostport=10.1.2.3:4043", "--conf", "spark.driver.extraJavaOptions= -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-2.3", "--conf", "spark.eventLog.dir=dbfs:/databricks/unravel/event
To print the configuration:
/usr/local/unravel/install_bin/databricks_setup.sh -p -u 10.1.2.3:4043 ----------------------------------- Cluster Spark Configuration ----------------------------------- spark.executor.extraJavaOptions -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3 spark.eventLog.enabled true spark.unravel.server.hostport 10.1.2.3:4043 spark.driver.extraJavaOptions -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-2.3 spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/ spark.unravel.shutdown.delay.ms 300 ----------------------------------- Spark Submit Parameters ----------------------------------- "--conf", "spark.executor.extraJavaOptions= -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3", "--conf", "spark.eventLog.enabled=true", "--conf", "spark.unravel.server.hostport=10.1.2.3:4043", "--conf", "spark.driver.extraJavaOptions= -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-2.3", "--conf", "spark.eventLog.dir=dbfs:/databricks/unravel/eventLogs/", "--conf", "spark.unravel.shutdown.delay.ms=300" ----------------------------------- Databricks Cluster Init Script ----------------------------------- dbfs:/databricks/unravel/unravel-db-sensor-archive/dbin/install-unravel.sh