Home

v4.5.x - Updates to Unravel properties

Note

This page lists Unravel configuration properties added, renamed, removed, and deprecated in 4.5.x. For a complete list of all current properties, see Unravel Properties.

4.5.5.0

Name/Description

Default

com.unraveldata.com.hive.spark.enabled

Set to true to allow Hive and Spark applications to be linked.

Only available for CDH 5.14.

Specified in hive-conf, or in the Hive CLI or beeline session. (CDH 5.14 only.)

true

Name/Description

Default

com.unraveldata.spark.live.interval.sec

This is the interval in seconds after which live application data is updated. It allows for tracking of Spark tasks. The Spark APM updates on Task completion in addition Job start, and Job and Stage completion.

This is a sensor property and is not updated unravel.properties.

60

com.unraveldata.mr.parse.input.split

Controls whether to parse splits information for map reduce jobs from YARN aggregated logs or not.

true

Updated

com.unraveldata.job.collector.log.aggregation.base

HDFS path to the aggregated container logs (logs to process). Do not include the hdfs://prefix. The log format defaults to TFile. You can specify multiple logs and log formats (TFile or IndexedFormat.)

Example: com.unraveldata.job.collector.log.aggregation.base=TFile:/tmp/logs/*/logs/,IndexedFormat:/tmp/logs/*/logs-ifile/.

/tmp/logs/*/logs/

Name/Description

Default

com.unraveldata.action.killmove.enabled

Allows users to kill and move apps.

true: users can kill and move appls

false: only admins can kill and move apps.

false

Kafka

com.unraveldata.ext.kafka.aggs.broker

100

com.unraveldata.ext.kafka.aggs.consumer_group

1000

com.unraveldata.ext.kafka.aggs.broker_metrics

1000

Property/Description

Default

Airflow

airflow.look.back.num.days

The number of days to look back. The look-back days can be specified as a positive or negative number. For instance -5 or 5 sets the look back number of days to 5.

1

airflow.look.back.num.hours

The look-back time window in hours, which can be either a positive or negative integer. If present, it takes precedence over airflow.look.back.num.days to have finer granularity. Suggested value for large clusters: 2

24

com.unraveldata.airflow.task.log.parsing.enabled

Controls whether to parse the Airflow Task logs. These logs are used to populate the Workflow Instance entities in the "Jobs - Workflows" page.

true

com.unraveldata.airflow.task.log.parsing.operators

Controls the Task logs to parse based on the Operator that produced it. Since Unravel only derives insights for Hive, Spark, and MR applications, it is set to only analyze operators that can launch those types of jobs.

The values are delimited using a "," and it treats "*" as a wildcard to many any or no characters. Any special characters like "\" or "." will be removed.

BashOperator, PythonOperator, *Hive*, *Spark*

com.unraveldata.airflow.task.thread.pool.size

Controls whether to process the Airflow Task logs sequentially or in parallel. Process the logs in parallel improves performance. This config takes effect only if airflow.task.log.parsing.enabled=true.

Possible values are 1-16. If one, logs are processed sequentially. If greater than one, logs are processed in parallel using a Thread Pool with of the size specified value.

1

Sessions

com.unraveldata.session.dynamicAllocation.enabled

If set to true Spark sessions recommends dynamic allocation. If users don't want that they need to explicitly set it to false.

true

com.unraveldata.session.spark1.submit.command

The command to use for Spark1 applications while applying the session run. This path should be fully qualified.

spark-submit

com.unraveldata.session.spark2.submit.command

The command to use for Spark2 applications while applying the session run. This path should be fully qualified.

spark2-submit

Spark

com.unraveldata.spark.shutdown.delay.ms

Amount of time to delay shutdown so the last messages are processed (allows Btrace sensor to send all the data before the spark driver exits).

Hive on MR job collectors

com.unraveldata.job.collector.running.load.conf

When set to true

  • Running MR jobs are linked to corresponding Hive app if Hive-on-MR app.

  • AutoAction metrics for running hive queries will be sent to AA2 backend.

false

com.unraveldata.job.collector.hive.queries.cache.size

This is used to improve the Hive-MR pipeline by caching data so it can be retrieved from cache instead of external API. You should not have to change this value.

1000

Property/Description

Default

Auto Actions

com.unraveldata.auto.action.metric.discard.ms

The look-back time window in hours, which can be either a positive or negative integer. If present, it takes precedence over airflow.look.back.num.days to have finer granularity. Suggested value for large clusters: 2

3600000

LDAP

com.unraveldata.ldap.mailAttribute

The mail attribute name in the LDAP response that Unravel server will use to extract the ldap user's email address.

If not configured, Unravel server use use the attribute name "mail".

-

com.unraveldata.ldap.customLDAPQuery

replaced hive.server2.authentication.ldap.customLDAPQuery

-

com.unraveldata.ldap.groupFilter

replaced hive.server2.authentication.ldap.groupFilter

-

com.unraveldata.ldap.groupDNPattern

replaced hive.server2.authentication.ldap.groupDNPattern

-

com.unraveldata.ldap.guidKey

replaced hive.server2.authentication.ldap.guidKey=uid

-

com.unraveldata.ldap.userDNPattern

replaced hive.server2.authentication.ldap.userDNPattern

-

com.unraveldata.ldap.userFilter

replaced hive.server2.authentication.ldap.userFilter

-

com.unraveldata.ldap.groupMembershipKey

replaced hive.server2.authentication.ldap.groupMembershipKey

-

com.unraveldata.ldap.groupClassKey

replaced hive.server2.authentication.ldap.groupClassKey

-

Custom Banner

com.unraveldata.custom.banner.display

Displays a banner at the top of the Unravel UI.

true: banner displays text until end.date

false: no change to UI

false

com.unraveldata.custom.banner.text

Text to display when display is true

-

com.unraveldata.custom.banner.end.date

Date and Time to stop displaying the custom banner

Format: YYYYMMDDTHHMMSSZ-000000

-

HBase

(See here for more information on all relevant properties.)

com.unraveldata.hbase. clustername .node.http.apis

HBase node web UI.

Format: http[s]://host:port,http[s]://host:port,... *

Example: http://your.master.server:16010,http://your.region.server:16030

-

Hive-hook SSL

(See here for properties needed to use Hive-hook.)

com.unraveldata.live.logreceiver.port.https

HTTPS server port (negative value means disabled HTTPS server)

-1

com.unraveldata.server.ssl.cert_path

KeyStore file path. For example, /usr/local/unravel/cert.jks

-

com.unraveldata.server.ssl.cert_password

KeyStore password

-

com.unraveldata.server.ssl.trust_store_path

TrustStore file path

-

com.unraveldata.server.ssl.trust_store_password

TrustStore password

-

Small Files and File Reports

(See here for all relevant properties.)

unravel.python.files_use_avg_file_size_flag

true: average of all the files is used against the threshold criteria and either all the files are accepted/counted or rejected/not counted as per the criteria.

false: absolute file size is used against the threshold criteria and a file is accepted/counted or rejected/not counter as per the criteria.

-

unravel.python.min_parent_dir_depth

Directory depth to start search at.

-

unravel.python.max_parent_dir_depth

Directory depth to end search at. Maximum is 50.

-

unravel.python.drill_down_subdirs_flag

When set a file is accounted (listed) for all its ancestors.

false: a file is accounted in only its immediate parent. This allows Unravel to find a specific directory with maximum number of files matching the size criteria.

-

Small Files

(See here for more information on all relevant properties.)

unravel.python.reports.files.small_files_use_avg_file_size_flag

true

unravel.python.reports.files.small_files_min_parent_dir_depth

0

unravel.python.reports.files.small_files_max_parent_dir_depth

10

reports.files.small_files_drill_down_subdirs_flag

true

File Reports

(See here for more information on all relevant properties.)

The following four properties are defined per file Size: huge, medium, tiny, small, empty

unravel.python.reports.files. Size _files_use_avg_file_size_flag

false

unravel.python.reports.files. Size _file_min_parent_dir_depth

0

unravel.python.reports.files. Size _file_max_parent_dir_depth

10

unravel.python.reports.files. Size _file_drill_down_subdirs_flag

false

Forecasting & Cloud Reports

(See here for more information for all relevant properties.)

com.unraveldata.ambari.manager.url

URL of Cloud Manager. For example, http://$clouderaserver:7180, http://$ambariserver:8080

For Cloudera, if the URL does not contain a port you must define manager.port below

-

com.unraveldata.ambari.manager.username

Username to log into the manager

-

com.unraveldata.ambari.manager.password

Password for the username.

-

Spark

com.unraveldata.spark.master

Default Spark master mode to be used if not available from the sensor.

-

com.unraveldata.spark.eventlog.appDuration

If application duration is more than configured value load the event log is not loaded.

1440 mins

Application Liveness

com.unraveldata.appstatus.refresh.mins

Time interval in minutes to scan for running applications and marking the stale ones.

5 min

com.unraveldata.appstatus.stale_limit.mins

Maximum number of minutes since the latest app update from RM before it is marked as stale.

Unit: minutes

10

Experimental

These properties should not to be used in production.

com.unraveldata.cluster_access.host

Cluster Access Service host (where the service is bound).

0.0.0.0

com.unraveldata.cluster_access.port

Cluster Access Service port.

4020

com.unraveldata.sregistry.hostport

Service Registry host:port.

${com.unraveldata.zk.quorum}

com.unraveldata.sensor.polling.secs

The base polling period of Unravel reactive sensors in seconds.

30s

com.unraveldata.appevents.emitters.exclude.list

Comma separated list of the application event emitter IDs which will be disabled/excluded.

-

com.unraveldata.multicluster.enabled

Allow Unravel to operate in multi-cluster mode.

In this mode a service registry will be used to discover and access all registered (local and remote) clusters.

false

Renamed/Replaced properties

Hive

(See LDAP above)

Property

Replaced with

Hive authentication

hive.server2.authentication.ldap.customLDAPQuery

com.unraveldata.ldap.customLDAPQuery

hive.server2.authentication.ldap.groupFilter

com.unraveldata.ldap.groupFilter

hive.server2.authentication.ldap.groupDNPattern

com.unraveldata.ldap.groupDNPattern

hive.server2.authentication.ldap.guidKey=uid

com.unraveldata.ldap.guidKey

hive.server2.authentication.ldap.userDNPattern

com.unraveldata.ldap.userDNPattern

hive.server2.authentication.ldap.userFilter

com.unraveldata.ldap.userFilter

hive.server2.authentication.ldap.groupMembershipKey

com.unraveldata.ldap.groupMembershipKey

hive.server2.authentication.ldap.groupClassKey

com.unraveldata.ldap.groupClassKey

RBAC

You must update these properties manually.

Property

Replaced with

com.unraveldata.rbac.mode

com.unraveldata.login.mode

com.unraveldata.rbac.user.operations.enabled

com.unraveldata.ngui.user.mode