Home

Configure Databricks with Unravel

In your Databricks workspace, update the following tabs under Advanced Options for every cluster (Automated/Interactive) that you want to monitor:

  1. Spark

    Copy the following snippet to Spark > Spark Conf. Replace Unravel DNS or IP Address. This snippet is also generated by the Databricks setup script on Unravel.

    spark.eventLog.enabled true
spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/
spark.unravel.server.hostport <Unravel DNS or IP Address>:4043
spark.unravel.shutdown.delay.ms 300
spark.executor.extraJavaOptions  
-Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3
spark.driver.extraJavaOptions  
-Dcom.unraveldata.client.rest.request.timeout.ms=1000 
-Dcom.unraveldata.client.rest.conn.timeout.ms=1000 
-javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-2.3

    Note

    For spark-submit jobs, click Configure spark-submit and copy the following snippet in the Set Parameters > Parameters text box as spark-submit parameters. Replace Unravel DNS or IP Address.

    "--conf", "spark.eventLog.enabled=true",
"--conf", "spark.eventLog.dir=dbfs:/databricks/unravel/eventLogs/",
"--conf", "spark.unravel.shutdown.delay.ms=300",
"--conf", "spark.unravel.server.hostport=<Unravel DNS or IP Address>:4043",
"--conf", "spark.executor.extraJavaOptions= -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3",
"--conf", "spark.driver.extraJavaOptions= -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-2.3"

  2. Logging

    Select DBFS as Destination, and copy the following as Cluster Log Path.

    dbfs:/cluster-logs/

  3. Init Scripts

    In the Init Scripts tab, set Destination to DBFS. Copy the following as the Init script path and click Add.

    dbfs:/databricks/unravel/unravel-db-sensor-archive/dbin/install-unravel.sh
In this section: