Skip to main content

Home

Configure Azure Databricks Automated (Job) Clusters with Unravel

  1. Sign in to the Azure portal.

  2. Add the following under Job > Configure Cluster > Spark >Spark Conf.

    Note

    This is generated from a Databricks setup script on Unravel. It needs to be copied on each Automated Clusters.

    Merge the configuration as needed.

    spark.executor.extraJavaOptions -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=executor,libs=spark-2.3
spark.driver.extraJavaOptions -Dcom.unraveldata.client.rest.request.timeout.ms=1000 -Dcom.unraveldata.client.rest.conn.timeout.ms=1000 -javaagent:/dbfs/databricks/unravel/unravel-agent-pack-bin/btrace-agent.jar=config=driver,script=StreamingProbe.btclass,libs=spark-2.3
spark.eventLog.enabled true
spark.eventLog.dir dbfs:/databricks/unravel/eventLogs/
spark.unravel.server.hostport Unravel VM Private IP Address spark.unravel.shutdown.delay.ms 300

  3. Enable logging Job > Configure Cluster > Spark > Logging.

  4. Add the following under Job > Configure Cluster > Spark > Init Scripts.

    dbfs:/databricks/unravel/unravel-db-sensor-archive/dbin/install-unravel.sh
In this section: