Log receiver

logreceiver.msg_retention.hours

The retention period, in hours, for messages stored in the DocStorage by the log receiver.

This value should align with the Kafka retention time for AWS, Spark, and JC topics.

1 hour

Spark

com.unraveldata.spark.live.skip.small.jobs

When set to true, skips the processing of small insignificant Spark jobs to improve Spark worker throughput only for a live pipeline.

false

com.unraveldata.spark.live.min.jobs.stored.count

Indicates the count of a minimum number of Spark jobs to persist for an application (irrespective of the duration) in a live pipeline. Considered only when com.unraveldata.spark.live.skip.small.jobs is set to true.

500

com.unraveldata.spark.live.small.jobs.duration.threshold

Indicates the minimum duration threshold of Spark jobs to skip insignificant jobs. Specify the duration in milliseconds.

1000

Impala

com.unraveldata.impala.polling.look.back.secs

A maximum number of seconds to look back when polling the Cloudera Manager API for subsequent polls.

0

Workflow

com.unraveldata.workflow.fetch.size

This property defines a maximum number of workflows to be retrieved from the cluster and displayed on the Unravel UI.

100

Data page

com.unraveldata.bigquery.tableinfo.enabled

Enable/ disable BigQuery for the Data page. By default, this is enabled.

TRUE

com.unraveldata.datapage.cluster.names

The Data page supported valid cluster names. If this property is not set then all clusters are shown.

-

com.unraveldata.metastore.types

Support metastore type for Data page.

Based on the cluster type:

com.unraveldata.datapage.workspace.names

The Data page supported valid workspace names. If this property is not set then all workspaces are shown.

-

Databricks

com.unraveldata.databricks.keys.toRedact

Redacts spark property values if the property key is in the list.

password

Migration

com.unraveldata.migrationplanning.clusterdiscovery.diagnostics

Set to true to enable diagnostics for Cluster Discovery report.

false

com.unraveldata.migrationplanning.cmph.diagnostics

Set to true to enable diagnostics for Cloud Mapping Per Host report.

false

com.unraveldata.migrationplanning.slicedice.diagnostics

Set to true to enable diagnostics for Workload Fit > Cloud Topology report.

false

com.unraveldata.migrationplanning.workload_fit_heatmap.diagnostics

Set to true to enable diagnostics for Workload Fit > Heatmap report.

false

com.unraveldata.migrationplanning.services_compatibility.diagnostics

Set to true to enable diagnostics for Services and Versions Compatibility report.

false

Spark

com.unraveldata.spark.query.size.max

200000

Pipelines

General pipelines

com.unraveldata.min.failed.job.duration.for.attempt.log

Minimum failed application duration to attempt yarn log loading for an application.

0

com.unraveldata.min.failed.job.duration.for.attempt.log

Minimum failed application duration to attempt yarn log loading for an application.

0

com.unraveldata.logs.max.containers.persisted

The maximum number of container logs to be persisted to the DB for an application.

4

com.unraveldata.logs.max.containers.processed

The maximum number of yarn containers to be processed from the yarn log of an application.

100

Impala pipelines

com.unraveldata.impala.scheduled.thread.pool.size

Minimum application duration to attempt yarn log loading for an application.

5

com.unraveldata.impala.thread.initial.delay.secs

Time to delay the first execution of the thread.

5

com.unraveldata.impala.cm.queries.polling.interval.secs

The period between successive executions for polling queries from CM.

60

com.unraveldata.impala.cm.metrics.polling.interval.secs

The period between successive executions for polling metrics from CM

60

com.unraveldata.impala.kafka.retry.millis

Delay (in milliseconds) between successive Impala Kafka message retries.

1000

App store

com.unravel.appstore.apps.enabled

To enable/disable Appstore UI. Set to true to enable and false to disable.

true

unravel.appstore.flask.port

The port on which the Appstore’s flask process will listen. This can be any free port on Unravel’s core node.

5001

unravel.appstore.apps.port.range.start

Defines the start value of a range of ports on which Individual apps will run. This can be any valid port on Unravel's core node.

8190

unravel.appstore.apps.port.range.end

Defines end value of a range of ports on which Individual apps will run. This can be any valid port on Unravel's core node. This value should be greater than the value set for unravel.appstore.apps.port.range.start.

8290

unravel.appstore.apps.gunicorn.worker.count

The default number of gUnicorn workers to be started for each app.

Default is calculated based on the number of cores of Unravel core node.

unravel.appstore.apps.<app_id>.gunicorn.worker.count

The default number of gUnicorn workers to be started for a specific app.

Defaults to value set for unravel.appstore.apps.gunicorn.worker.count

unravel.appstore.apps.gunicorn.thread.count

The default number of gUnicorn threads to be started per gUnicorn worker for each app.

Default is based on the number of cores of the Unravel core node.

unravel.appstore.apps.<app_id>.gunicorn.thread.count

The default number of gUnicorn threads to be started per gUnicorn worker for a specific app.

Defaults to value set for unravel.appstore.apps.gunicorn.thread.count

unravel.appstore.resource_limit.rss.gb

Total RSS memory in GB that all running apps can consume.

The default value is set to half of the system memory of the Unravel core node.

unravel.appstore.apps.resource_limit.process_count

The default maximum limit on the number of processes that can be running for each app.

5 x value set forunravel.appstore.apps.gunicorn.worker.count

unravel.appstore.apps.<app_id>.resource_limit.process_count

The default maximum limit on the number of processes that can be running for a specific app.

Same as set forunravel.appstore.apps.resource_limit.process_count

unravel.appstore.apps.resource_limit.sum.thread_count

Limit on the sum of the number of threads across processes belonging to an app.

No limit

unravel.appstore.apps.<app_id>.resource_limit.sum.thread_count

Limit on the sum of the number of threads across processes belonging to a specific app.

The same value set for unravel.appstore.apps.resource_limit.sum.thread_count

If this property is not set, then the default is:

5 x (unravel.appstore.apps.<app_id>.gunicorn.worker.count x unravel.appstore.apps.<app_id>.gunicorn.thread.count)

unravel.appstore.apps.resource_limit.sum.rss

Limit on the sum of RSS in bytes across processes belonging to an app.

1/10 x Total system memory on Unravel core node

unravel.appstore.apps.<app_id>.resource_limit.sum.rss

Limit on the sum of RSS in bytes across processes belonging to a specific app.

Same value set for unravel.appstore.apps.resource_limit.sum.rss

unravel.appstore.apps.resource_limit.sum.vms

Limit on the sum of VMS in bytes across processes belonging to an app.

No limit

>>unravel.appstore.apps.<app_id}.resource_limit.sum.vms

Limit on the sum of VMS in bytes across processes belonging to a specific app.

Same value set for unravel.appstore.apps.resource_limit.sum.vms

unravel.appstore.apps.resource_limit.sum.memory_percent

Limit on the sum of memory percentage across processes belonging to an app.

No limit

unravel.appstore.apps.<app_id>.resource_limit.sum.memory_percent

Limit on the sum of memory percentage across processes belonging to a specific app.

Same value set forunravel.appstore.apps.resource_limit.sum.memory_percent

unravel.appstore.apps.resource_limit.sum.cpu_percent

Limit on the sum of CPU percentage across processes belonging to an app.

No limit

unravel.appstore.apps.<app_id>.resource_limit.sum.cpu_percent

Limit on the sum of memory percentage across processes belonging to a specific app.

Same value set for unravel.appstore.apps.resource_limit.sum.cpu_percent

unravel.appstore.apps.resource_limit.sum.cpu_time

Limit on the sum of CPU time in seconds across processes belonging to an app.

No limit

unravel.appstore.apps.{app_id}.resource_limit.sum.cpu_time

Limit on the sum of cpu_time in seconds across processes belonging to a specific app.

Same value set for unravel.appstore.apps.resource_limit.sum.cpu_time

unravel.appstore.apps.resource_limit.<aggregation>.<resource_name}>

Property to set resource limits and aggregation methods for all the apps.

No limit

unravel.appstore.apps.<app_id>.resource_limit.<aggregation>.<resource_name>

Property to set resource limits and aggregation methods for a specific app.

No limit

unravel.appstore.apps.healthcheck.interval.seconds

The interval at which the Health check process is run for the apps.

60 seconds

unravel.appstore.apps.resource.usage.refresh.rate.seconds

The interval at which resource usages will be updated.

60 seconds

BigQuery

com.unraveldata.bigquery.poll.interval

Sets the frequency to poll Project data such as project name, state, etc. By default, it is one day that is 1440 minutes.

1440 mins

com.unraveldata.bigquery.projects

Specify project IDs for BigQuery monitoring.

com.unraveldata.bigquery.pubsub.subscription_id.<project-ID>

Specify the subscription ID of a project for BigQuery monitoring.

com.unraveldata.bigquery.credentials.<project-id>

Specify service key file for Google API authentication.

Multi-cluster Hadoop

unravel.hadoop_module.platform

Set this property when you install the core node on a server where there is no Hadoop configuration.

unravel

Impala

com.unraveldata.impala.http.conn.timeout.ms

HTTP connection time out for impala connector.

2000 ms

com.unraveldata.impala.http.read.timeout.ms

HTTP read time out for impala connector.

6000 ms

com.unraveldata.impala.http.backoff.ms

HTTP client back-off time for impala connector. Time for which HTTP client sleeps before re-attempting after unsuccessful read attempt.

1000 ms

com.unraveldata.impala.query.max.retries

Specifies the number of retries that will be done to fetch the profile tree for any impala query.

5

LDAP

com.unraveldata.ldap.domain

Contains a real domain name.

-

Live metrics

com.unraveldata.metrics.merge.cache

Specify the count of messages to be stored in the cache or memory bthem efore storing in the database. For example, if com.unraveldata.metrics.merge.cache = 30 and unravel.metrics.factor = 5 (seconds), then 30 * 5 = 150 seconds. The data is pushed to the database every 150 seconds. For 150 seconds, the data is kept in memory.

6

com.unraveldata.metrics.sample.count.max

Specify the maximum count of data points for sampling purposes.

16

com.unraveldata.metrics.data.points

The number of data points to be stored in a single database row.

44

Role-based Access Control (RBAC)

com.unraveldata.rbac.roles

A comma-separated list of custom roles.

-

com.unraveldata.rbac.role.<role>.users

A comma-separated list of users that can be assigned to a role. The role can be admin, readonlyAdmin, or custom_role, which is set using com.unraveldata.rbac.roles property.

com.unraveldata.rbac.role.<role>.groups

A comma-separated list of groups that can be assigned to a role. The role can be admin, readonlyAdmin, or custom_role, which is set using com.unraveldata.rbac.roles property.

com.unraveldata.rbac.role.<role>.data.user.filter

User data filter, if enabled data is filtered on the username. The role can be admin or custom_role, which is set using com.unraveldata.rbac.roles property.

FALSE

com.unraveldata.rbac.role.<role>.data.tags.filter

Tags data filter, if enabled data is filtered on the user tags. The role can be admin or custom_role, which is set using com.unraveldata.rbac.roles property.

FALSE

com.unraveldata.rbac.role.<role>.data.es.query

Query data filter, if enabled data is filtered on the given valid ElasticSearch query. The role can be admin or custom_role, which is set using com.unraveldata.rbac.roles property.

com.unraveldata.rbac.role.<role>.data.field.<field>

Fields data filter, if enabled data is filtered on the supported ElasticSearch fields for the given values. The <field> should be in the supported field list. The role can be user or custom_role, which is set using com.unraveldata.rbac.roles property.

com.unraveldata.rbac.role.<role>.views

The views that are shown for a specific role. This is configured based on the View ID list. The role can be user or custom_role, which is set using com.unraveldata.rbac.roles property.

clusters.resources, clusters.chargeback, compute, jobs

Role-based Access Control (RBAC)

com.unraveldata.saml.groupFilter

Restricts access to a few selected SAML groups.

com.unraveldata.login.groupFilter

com.unraveldata.rbac.default

Determines how an end-user's views are filtered when no specific tags are set for them.

com.unraveldata.rbac.user.field

com.unraveldata.rbac.ldap.tags / com.unraveldata.rbac.saml.tags

A comma-separated list of the tags. For example, proj, dept.

com.unraveldata.login.tags

com.unraveldata.rbac.ldap.tag.<TAG_KEY>.regex.find / com.unraveldata.rbac.saml.tag.<TAG_KEY>.regex.find

Defines regular expression used to extract a tag_value for a given tag_key.

com.unraveldata.login.tag.<tag_name>.regex.find

com.unraveldata.login.admins

Defines admin users.

com.unraveldata.rbac.role.admin.users

com.unraveldata.login.admins.readonly

Defines read-only admin. users

com.unraveldata.rbac.role.readonlyAdmin.users

com.unraveldata.login.admins.groups / com.unraveldata.login.admins.ldap.groups / com.unraveldata.login.admins.saml.groups

Defines admin groups.

com.unraveldata.rbac.role.admin.groups

com.unraveldata.login.admins.readonly.groups / com.unraveldata.login.admins.readonly.groups / com.unraveldata.login.admins.readonly.saml.groups

Defines read-only admin groups.

com.unraveldata.rbac.role.readonlyAdmin.groups

LDAP

com.unraveldata.ldap.ids

A colon-separated list of LDAP servers (domains). In case this property is not defined then the default value is considered.

default_ldap_id

com.unraveldata.ldap.default.id

The default LDAP server from the com.unraveldata.ldap.ids list. If this property is not defined, the first LDAP server from com.unraveldata.ldap.ids is used

-

Generic usage

LDAP specific usage

com.unraveldata.ldap.baseDN

com.unraveldata.ldap.base.dn

com.unraveldata.ldap.<ldap_id>.base.dn

com.unraveldata.ldap.bind_dn

com.unraveldata.ldap.bind.dn

com.unraveldata.ldap.<ldap_id>.bind.dn

com.unraveldata.ldap.bind_pw

com.unraveldata.ldap.bind.pw

com.unraveldata.ldap.<ldap_id>.bind.pw

com.unraveldata.ldap.groupClassKey

com.unraveldata.ldap.group.class

com.unraveldata.ldap.<ldap_id>.group.class

com.unraveldata.ldap.groupDNPattern

com.unraveldata.ldap.group.dn.pattern

com.unraveldata.ldap.<ldap_id>.group.dn.pattern

com.unraveldata.ldap.groupMembershipKey

com.unraveldata.ldap.group.member.attr

com.unraveldata.ldap.<ldap_id>.group.member.attr

com.unraveldata.ldap.groupSearchMethods

com.unraveldata.ldap.group.search.methods

com.unraveldata.ldap.<ldap_id>.group.search.methods

com.unraveldata.ldap.guidKey

com.unraveldata.ldap.uid.attr

com.unraveldata.ldap.<ldap_id>.uid.attr

com.unraveldata.ldap.mailAttribute

com.unraveldata.ldap.mail.attr

com.unraveldata.ldap.<ldap_id>.mail.attr

com.unraveldata.ldap.realUserAttribute

com.unraveldata.ldap.real.uid.attr

com.unraveldata.ldap.<ldap_id>.real.uid.attr

com.unraveldata.ldap.userDNPattern

com.unraveldata.ldap.user.dn.pattern

com.unraveldata.ldap.<ldap_id>.user.dn.pattern

com.unraveldata.ldap.groupFilter

com.unraveldata.login.groupFilter

NA

com.unraveldata.ldap.userFilter

com.unraveldata.login.userFilter

NA

com.unraveldata.ldap.SAMAccountNameFilter

com.unraveldata.ldap.sAMAccountName.enabled

com.unraveldata.ldap.<ldap_id>.sAMAccountName.enabled

com.unraveldata.ldap.groupQueryFilter

com.unraveldata.ldap.group.query.filter

com.unraveldata.ldap.<ldap_id>.group.query.filter

Azure Active Directory (AAD)

Properties to be updated in aad.json file for integrating Unravel with AAD authentication. For more details refer to AAD Authentication.

tenantID

ID of the AAD directory.

-

Migration

com.unraveldata.migrationplanning.vmprices.<cloud provider>

Set the cloud provider and the path to the CSV file which contains the custom prices for instances.

The following values can be used for the cloud provider:

-

Insights Overview

com.unraveldata.valuedashboard.enabled

Enables the Insights Overview tab on Unravel UI.

false

Databricks

com.unraveldata.agent.metrics.ganglia_enabled

Enables Ganglia metrics, which is supported by the Btrace agent. To enable Ganglia metrics, Dcom.unraveldata.agent.metrics.ganglia_enabled is set to true in extraJavaOptions.

Datastore

com.unraveldata.datastore.rest.api.timeout.millis

Specifies the timeout period for different APIs used by UI to fetch data.

120000

Migration

com.unraveldata.migrationplanning.supported.cloudproviders

Controls the cloud providers that must be displayed for selection for the migration reports. Values are as follows:

You can provide a comma-separated list of cloud providers when you specify this property.

For example: com.unraveldata.migrationplanning.supported.cloudproviders EMR,EC2,HDI

-

com.unraveldata.migrationplanning.blacklist

Provides the path to a CSV or TXT file that can be used to blacklist specific VM instances for all cloud providers. The CSV or TXT file contains a comma-separated list of VM instances that must not be displayed for the migration reports.

For example: com.unraveldata.migrationplanning.blacklist /opt/unravel/data/migration/blacklist.txt.

-

com.unraveldata.migrationplanning.blacklist.<cloud_provider>

Provides the path to a CSV or TXT file that can be used to blacklist specific VM instances for specific cloud providers. The CSV or TXT file contains a comma-separated list of VM instances that must not be displayed for the migration reports.

The values for <cloud_provider> are as follows:

For example: com.unraveldata.migrationplanning.blacklist.EMR /opt/unravel/data/migration/blacklist.txt.

-

Azure Active Directory (AAD) integration

Properties to be updated in aad.json file for integrating Unravel with AAD authentication. For more details refer to Enable AAD authentication

clientID

Client ID from Overview > Essentials > Application (client) ID.

-

clientSecret

Client secret.

-

callbackURL

Redirect URL to consume login response from AAD.

-

logoutCallbackURL

Redirect URL to consume logout response from AAD.

-

logoutEnabled

Enable/disable logout on AAD when the user logs out from Unravel.

-

unravel_mapping

Maps the AAD attributes to Unravel attributes.

-