com.unraveldata.spark.live.pipeline.enabled

Specifies if Unravel should process the live job data coming from sensor or not. true: The live job data will be processed as soon as it is received. false: Live job data will not be processed.

com.unraveldata.spark.live.pipeline.maxStoredStages

Maximum number of jobs/stages stored in the DB. If an application has (# jobs/stages) > maxStoredStages only the last maxStoredStages are stored.

This setting affects only the live pipeline. When processing the event log file (after the application has completed its execution) this property is not considered.

com.unraveldata.spark.master

Default spark master mode to be used if not available from Sensor.

Possible values: local, standalone or yarn (default)

com.unraveldata.jobtime.to.apptime.ratio.threshold

Defines the threshold of the ratio of jobtime to apptime and is used to determine if SQL analysis of queries should be done for an app.

com.unraveldata.num.sql.queries.to.analyze

This property defines the maximum number of queries to analyze in a spark app.

spark.unravel.sql.op.timing.enabled

This property is a unravel specific Spark conf which can be defined in spark-submit command to enable or disable the capturing of SQL timing data.

com.unraveldata.sqloperator.to.query.ratio.threshold

This property defines the ratio of operatorTime/querytime and determines whether the operator is slow.

com.unraveldata.gctime.to.querytime.ratio.threshold

This property defines the ratio of gcTime/querytime and determines if the query is spending significant time in GC.

com.unraveldata.query.to.app.ratio.threshold

This property defines the ratio of queryTime/appTime and identifies the most significant query in the app.

com.unraveldata.spark.eventlog.location

All the possible locations of the event log files. Multiple locations are supported as a comma-separated list of values.

This property is used only when the Unravel sensor is not enabled. When the sensor is enabled, the event log path is taken from the application configuration at runtime.

com.unraveldata.spark.eventlog.maxSize

Maximum size of the event log file that will be processed by the Spark worker daemon. Event logs larger than MaxSize will not be processed.

com.unraveldata.spark.eventlog.appDuration.mins

Maximum duration (in minutes) of application to pull Spark event log.

com.unraveldata.spark.hadoopFsMulti.useFilteredFiles

Specifies how to search the event log files.

true: prefix search

false: prefix + suffix search

Prefix + suffix search is faster as it avoids listFiles() API which may take a long time for large directories on HDFS. This search requires that all the possible suffixes for the event log files are known. Possible suffixes are specified by com.unraveldata.spark.hadoopFsMulti.eventlog.suffixes..

com.unraveldata.spark.hadoopFsMulti.eventlog.suffixes

Specifies suffixes used for prefix+suffix search of the event logs when com.unraveldata.spark.hadoopFsMulti.useFilteredFiles=false.

NOTE: the empty suffix (,,) be part of this value for uncompressed event log files.

com.unraveldata.spark.appLoading.maxAttempts

Maximum number of attempts for loading the event log file from HDFS/S3/ ADL/WASB etc.

com.unraveldata.spark.appLoading.delayForRetry

Delay used among consecutive retries when loading the event log files. The actual delay is not constant, it increases progressively by 2^attempt * delayForRetry.

com.unraveldata.spark.tasks.inMemoryLimit

Number of tasks to be kept in memory and DB per stage. All stats are calculated for all the task attempts but only the configured number of tasks will be kept in memory/DB.

Events Related

com.unraveldata.spark.events.enableCaching

Enables logic for executing caching events.

com.unraveldata.spark.appLoading.maxConcurrentApps

The number of applications Unravel keep metadata in Spark worker daemon memory.

com.unraveldata.spark.time.histogram

Specifies whether the timeline histogram is generated or not.

Note: Timeline histogram generation is memory intensive.

com.unraveldata.yarn.log.delayForRetry

Delay in milliseconds for consecutive retries when loading the yarn log files.

com.unraveldata.yarn.log.maxAttempts

A maximum number of attempts for loading the yarn log files.

spark.unravel.shutdown.delay.ms

Amount of time to delay shutdown so the last messages are processed (allows Btrace sensor to send all the data before the spark driver exits).

spark.unravel.live.update.interval.sec

This is the interval in seconds after which live application data is updated. It allows for the tracking of Spark tasks. The Spark APM updates on Task completion in addition Job start, and Job and Stage completion.

com.unraveldata.s3.profile.config.file.path

The path to the s3 profile file, e.g., /usr/local/unravel/etc/s3ro.properties.

com.unraveldata.spark.s3.profilesToBuckets

Comma separated list of profile to bucket mappings in the following format: <s3_profile>:<s3_bucket>, for example, com.unraveldata.spark.s3.profileToBuckets=profile-prod:com.unraveldata.dev,profile-dev:com.unraveldata.dev.

Note