Skip to main content

Home

Enable/disable live monitoring of Spark Streaming applications

This topic explains how to enable and disable live monitoring of Spark Streaming applications.

Note

Spark versions 1.6.x, 2.x, and 3.0.x are supported for spark streaming.

Note

Support for Spark apps using the Structured Streaming API introduced in Spark 2 is limited.

Unravel uses a StreamingProbe to perform the live monitoring. By default, the probe is not enabled because monitoring can generate a lot of data and potentially overload Unravel's ES.

The probe is controlled by spark.driver.extraJavaOptions which is defined in spark-defaults.conf.

The Spark streaming probe is disabled by default and you must enable it manually by editing spark-defaults.conf.

For CDH

  1. To enable the probe edit spark-defaults.conf and set the property as follows:

    spark.driver.extraJavaOptions=-javaagent:/opt/cloudera/parcels/UNRAVEL_SENSOR/lib/java/btrace-agent.jar=libs=spark-<spark-version>,script=StreamingProbe.btclass,config=driver

  2. Set spark.driver.extraJavaOptions as follows. You can comment out the above definition and just add another definition to ease toggling the live monitoring.

    // comment out
#spark.driver.extraJavaOptions=-javaagent:/opt/cloudera/parcels/UNRAVEL_SENSOR/lib/java/btrace-agent.jar=libs=spark-<spark-version>,script=StreamingProbe.btclass,config=driver
// and add 
spark.driver.extraJavaOptions=-javaagent:/opt/cloudera/parcels/UNRAVEL_SENSOR/lib/java/btrace-agent.jar=libs=spark-<spark-version>,config=driver
All other platforms.

Depending on your Unravel version and platform the BTrace jars are located in /usr/local/unravel-agent/jars or /usr/local/unravel-jars. Substitute your jar location for JAR_location below,

  1. To enable the probe edit spark-defaults.conf and set the property as follows:

    spark.driver.extraJavaOptions=-javaagent:JAR_location/btrace-agent.jar=libs=spark-<spark-version>,script=StreamingProbe.btclass,config=driver

  2. Set spark.driver.extraJavaOptions as follows. You can comment out the above definition and just add another definition to ease toggling the live monitoring.

    // comment out
#spark.driver.extraJavaOptions=-javaagent:JAR_location/btrace-agent.jar=libs=spark-<spark-version>,script=StreamingProbe.btclass,config=driver
// and add 
spark.driver.extraJavaOptions=-javaagent:JAR_location/btrace-agent.jar=libs=spark-<spark-version>,config=driver