Home

Enable/disable live monitoring of Spark Streaming applications

This topic explains how to enable and disable live monitoring of Spark Streaming applications.

Unravel uses a StreamingProbe to perform the live monitoring. By default, the probe is not enabled because monitoring can generate a lot of data and potentially overload Unravel's ES.

The probe is controlled by spark.driver.extraJavaOptions which is defined in spark-defaults.conf.

  1. To enable the probe edit spark-defaults.conf and set the property as follows:

    spark.driver.extraJavaOptions=-javaagent:/usr/local/unravel-agent/jars/btrace-agent.jar=libs=spark-2.3,script=StreamingProbe.btclass,config=driver

  2. Set spark.driver.extraJavaOptions as follows. You can comment out the above definition and just add another definition to ease toggling the live monitoring.

    // comment out
#spark.driver.extraJavaOptions=-javaagent:/usr/local/unravel-agent/jars/btrace-agent.jar=libs=spark-2.3,script=StreamingProbe.btclass,config=driver
// and add 
spark.driver.extraJavaOptions=-javaagent:/usr/local/unravel-agent/jars/btrace-agent.jar=libs=spark-2.3,config=driver
In this section: