- Home
- Release Notes
- v4.7.x Release notes
- v4.7.x - Updates to Unravel properties
v4.7.x - Updates to Unravel properties
Note
This page lists Unravel configuration properties added, renamed, removed, and deprecated in 4.7.x. For a complete list of all current properties, see Unravel Properties.
4.7.9.1
Property/Description | Default |
|---|---|
com.unraveldata.metrics.factor Property to skip metrics updates. If this property is specified, lesser messages are processed by the insights worker. Specify any of the following to skip metrics updates:
| 1 |
unravel.python.fsimage.java.home Specify the JAVA home for FSImage processing. | - |
com.unraveldata.table.polling.limit Modify the batch size for polling the table metadata based on the number of tables available in your Hive metastore. | 1000000 |
com.unraveldata.partition.polling.limit Modify the batch size for polling partitions metadata based on the number of partitions available in your Hive metastore. | 1000000 |
NA
NA
NA
NA
NA
4.7.9.0
Property/Description | Default |
|---|---|
logreceiver.verticles_count To increase the capacity of LR, you can vertically scale it by adding more verticles. Use the logreceiver.verticles_count property to set the number of verticles. |
The value specifies that the number of LR verticles equals MIN (CPU cores, 12). |
unravel.appstore.apps.gunicorn.timeout Set the timeout for apps in the appstore. | 300 seconds |
com.unraveldata.config.redaction.regex Use this common unravel property to redact configuration values. |
NA
NA
NA
Property Name | Description |
|---|---|
| Removed the property from the Hive hook to remove HDFS dependency. If encountering difficulties accessing the Log Receiver (LR), logging the information is adequate instead of storing the data in HDFS. |
4.7.8.0
Property/Description | Default |
|---|---|
AutoActions support for EMR Cluster | |
com.unraveldata.auto.action.emr.cluster.enabled Enables ElasticSearch polling for collecting EMR cluster metrics. | false |
com.unraveldata.auto.action.emr.cluster.idle.lookback.millis Set the minimum look-back period for calculating the idleness of an EMR cluster. The EMR cluster idle duration is specific to the EMR Cluster Idle AutoAction policy. You must set this property if configured idle threshold for an EMR cluster is more than an hour. Ensure to specify the value in milliseconds. | 3600000 (1 hour) |
Insights consumers | |
com.unraveldata.insights.consumers Specify this property to set the number of threads in the insights topic worker. This property sets the number of insights consumers running within an insights worker to increase processing speed. | 2 |
Spark consumers | |
com.unraveldata.spark.consumers Specify this property to run multiple Kafka consumers in a single Spark worker. This property sets the number of spark consumers running within a spark worker. You must configure this property in every environment depending on the desired number of consumers. Additionally, set the number of Kafka partitions to the same number (or higher). | 1 |
NA
NA
NA
NA
4.7.7.0
Property/Description | Default |
|---|---|
License | |
com.unraveldata.license.file When you run the | - |
GCP - BigQuery | |
com.unraveldata.bigquery.ondemand.analysis.pricing.per.tb.usd.{region} Specify the on-demand analysis price per TB in USD based on the region to process queries. The The sample Examples:
You can find the possible values for | 5 ($5) |
com.unraveldata.bigquery.live.metrics.polling.interval.mins Determines the frequency at which the live metrics should be fetched for a long running query in minutes. | 5 mins |
com.unraveldata.bigquery.projects.polling.interval.secs The frequency of polling interval of audit log messages for the project in seconds. | 300 secs |
com.unraveldata.bigquery.pubsub.total.message.{projectId} Determines the maximum no of messages that must be fetched in each polling interval for a project. | 100 |
The default value for the spark.unravel.shutdown.delay.ms is changed from 300 ms to 1500 ms (1.5 seconds).
NA
NA
NA
4.7.6.0
Property/Description | Default |
|---|---|
Persistent API Token | |
com.unraveldata.api.access.tokens.store Provide the path to the JSON file that is created in the location accessible only to the Unravel user | - |
com.unraveldata.auto.action.use.cli.enabled Enables use of Command Line Interface (CLI) for actions associated with AutoActions. | true |
NA
NA
NA
NA
4.7.5.0
Property/Description | Default |
|---|---|
Log receiver | |
logreceiver.msg_retention.hours The retention period, in hours, for messages stored in the DocStorage by the log receiver. This value should align with the Kafka retention time for AWS, Spark, and JC topics. | 1 hour |
Spark | |
com.unraveldata.spark.live.skip.small.jobs When set to | false |
com.unraveldata.spark.live.min.jobs.stored.count Indicates the count of a minimum number of Spark jobs to persist for an application (irrespective of the duration) in a live pipeline. Considered only when com.unraveldata.spark.live.skip.small.jobs is set to | 500 |
com.unraveldata.spark.live.small.jobs.duration.threshold Indicates the minimum duration threshold of Spark jobs to skip insignificant jobs. Specify the duration in milliseconds. | 1000 |
Impala | |
com.unraveldata.impala.polling.look.back.secs A maximum number of seconds to look back when polling the Cloudera Manager API for subsequent polls. | 0 |
Workflow | |
com.unraveldata.workflow.fetch.size This property defines a maximum number of workflows to be retrieved from the cluster and displayed on the Unravel UI. | 100 |
Live Metrics
The default value of the com.unraveldata.metrics.merge.cache property has been changed from 6 to 30.
None
None
com.unraveldata.cloudera.manager.impala.poll.interval.millis
com.unraveldata.databricks.dbu.discount.percentage
4.7.4.0
Property/Description | Default |
|---|---|
Data page | |
com.unraveldata.bigquery.tableinfo.enabled Enable/ disable BigQuery for the Data page. By default, this is enabled. | TRUE |
com.unraveldata.datapage.cluster.names The Data page supported valid cluster names. If this property is not set then all clusters are shown. | - |
com.unraveldata.metastore.types Support metastore type for Data page. | Based on the cluster type:
|
com.unraveldata.datapage.workspace.names The Data page supported valid workspace names. If this property is not set then all workspaces are shown. | - |
Databricks | |
com.unraveldata.databricks.keys.toRedact Redacts spark property values if the property key is in the list. | password |
Migration | |
com.unraveldata.migrationplanning.clusterdiscovery.diagnostics Set to true to enable diagnostics for Cluster Discovery report. | false |
com.unraveldata.migrationplanning.cmph.diagnostics Set to true to enable diagnostics for Cloud Mapping Per Host report. | false |
com.unraveldata.migrationplanning.slicedice.diagnostics Set to true to enable diagnostics for Workload Fit > Cloud Topology report. | false |
com.unraveldata.migrationplanning.workload_fit_heatmap.diagnostics Set to true to enable diagnostics for Workload Fit > Heatmap report. | false |
com.unraveldata.migrationplanning.services_compatibility.diagnostics Set to true to enable diagnostics for Services and Versions Compatibility report. | false |
Spark | |
com.unraveldata.spark.query.size.max | 200000 |
Pipelines | |
General pipelines | |
com.unraveldata.min.failed.job.duration.for.attempt.log Minimum failed application duration to attempt yarn log loading for an application. | 0 |
com.unraveldata.min.failed.job.duration.for.attempt.log Minimum failed application duration to attempt yarn log loading for an application. | 0 |
com.unraveldata.logs.max.containers.persisted The maximum number of container logs to be persisted to the DB for an application. | 4 |
com.unraveldata.logs.max.containers.processed The maximum number of yarn containers to be processed from the yarn log of an application. | 100 |
Impala pipelines | |
com.unraveldata.impala.scheduled.thread.pool.size Minimum application duration to attempt yarn log loading for an application. | 5 |
com.unraveldata.impala.thread.initial.delay.secs Time to delay the first execution of the thread. | 5 |
com.unraveldata.impala.cm.queries.polling.interval.secs The period between successive executions for polling queries from CM. | 60 |
com.unraveldata.impala.cm.metrics.polling.interval.secs The period between successive executions for polling metrics from CM | 60 |
com.unraveldata.impala.kafka.retry.millis Delay (in milliseconds) between successive Impala Kafka message retries. | 1000 |
None
None
None
com.unraveldata.cloudera.manager.impala.poll.interval.millis
4.7.3.0
Property/Description | Default |
|---|---|
App store | |
com.unravel.appstore.apps.enabled To enable/disable Appstore UI. Set to true to enable and false to disable. | true |
unravel.appstore.flask.port The port on which the Appstore’s flask process will listen. This can be any free port on Unravel’s core node. | 5001 |
unravel.appstore.apps.port.range.start Defines the start value of a range of ports on which Individual apps will run. This can be any valid port on Unravel's core node. | 8190 |
unravel.appstore.apps.port.range.end Defines end value of a range of ports on which Individual apps will run. This can be any valid port on Unravel's core node. This value should be greater than the value set for unravel.appstore.apps.port.range.start. | 8290 |
unravel.appstore.apps.gunicorn.worker.count The default number of gUnicorn workers to be started for each app. | Default is calculated based on the number of cores of Unravel core node. |
unravel.appstore.apps.<app_id>.gunicorn.worker.count The default number of gUnicorn workers to be started for a specific app. | Defaults to value set for unravel.appstore.apps.gunicorn.worker.count |
unravel.appstore.apps.gunicorn.thread.count The default number of gUnicorn threads to be started per gUnicorn worker for each app. | Default is based on the number of cores of the Unravel core node. |
unravel.appstore.apps.<app_id>.gunicorn.thread.count The default number of gUnicorn threads to be started per gUnicorn worker for a specific app. | Defaults to value set for unravel.appstore.apps.gunicorn.thread.count |
unravel.appstore.resource_limit.rss.gb Total RSS memory in GB that all running apps can consume. | The default value is set to half of the system memory of the Unravel core node. |
unravel.appstore.apps.resource_limit.process_count The default maximum limit on the number of processes that can be running for each app. | 5 x value set forunravel.appstore.apps.gunicorn.worker.count |
unravel.appstore.apps.<app_id>.resource_limit.process_count The default maximum limit on the number of processes that can be running for a specific app. | Same as set forunravel.appstore.apps.resource_limit.process_count |
unravel.appstore.apps.resource_limit.sum.thread_count Limit on the sum of the number of threads across processes belonging to an app. | No limit |
unravel.appstore.apps.<app_id>.resource_limit.sum.thread_count Limit on the sum of the number of threads across processes belonging to a specific app. | The same value set for unravel.appstore.apps.resource_limit.sum.thread_count If this property is not set, then the default is: 5 x (unravel.appstore.apps.<app_id>.gunicorn.worker.count x unravel.appstore.apps.<app_id>.gunicorn.thread.count) |
unravel.appstore.apps.resource_limit.sum.rss Limit on the sum of RSS in bytes across processes belonging to an app. | 1/10 x Total system memory on Unravel core node |
unravel.appstore.apps.<app_id>.resource_limit.sum.rss Limit on the sum of RSS in bytes across processes belonging to a specific app. | Same value set for unravel.appstore.apps.resource_limit.sum.rss |
unravel.appstore.apps.resource_limit.sum.vms Limit on the sum of VMS in bytes across processes belonging to an app. | No limit |
>>unravel.appstore.apps.<app_id}.resource_limit.sum.vms Limit on the sum of VMS in bytes across processes belonging to a specific app. | Same value set for unravel.appstore.apps.resource_limit.sum.vms |
unravel.appstore.apps.resource_limit.sum.memory_percent Limit on the sum of memory percentage across processes belonging to an app. | No limit |
unravel.appstore.apps.<app_id>.resource_limit.sum.memory_percent Limit on the sum of memory percentage across processes belonging to a specific app. | Same value set forunravel.appstore.apps.resource_limit.sum.memory_percent |
unravel.appstore.apps.resource_limit.sum.cpu_percent Limit on the sum of CPU percentage across processes belonging to an app. | No limit |
unravel.appstore.apps.<app_id>.resource_limit.sum.cpu_percent Limit on the sum of memory percentage across processes belonging to a specific app. | Same value set for unravel.appstore.apps.resource_limit.sum.cpu_percent |
unravel.appstore.apps.resource_limit.sum.cpu_time Limit on the sum of CPU time in seconds across processes belonging to an app. | No limit |
unravel.appstore.apps.{app_id}.resource_limit.sum.cpu_time Limit on the sum of cpu_time in seconds across processes belonging to a specific app. | Same value set for unravel.appstore.apps.resource_limit.sum.cpu_time |
unravel.appstore.apps.resource_limit.<aggregation>.<resource_name}> Property to set resource limits and aggregation methods for all the apps. | No limit |
unravel.appstore.apps.<app_id>.resource_limit.<aggregation>.<resource_name> Property to set resource limits and aggregation methods for a specific app. | No limit |
unravel.appstore.apps.healthcheck.interval.seconds The interval at which the Health check process is run for the apps. | 60 seconds |
unravel.appstore.apps.resource.usage.refresh.rate.seconds The interval at which resource usages will be updated. | 60 seconds |
BigQuery | |
com.unraveldata.bigquery.project.details.poll.delay.mins Sets the frequency to poll Project data such as project name, state, etc. By default, it is one day that is 1440 minutes. | 1440 mins |
com.unraveldata.bigquery.projects Specify project IDs for BigQuery monitoring. | |
com.unraveldata.bigquery.pubsub.subscription_id.<project-ID> Specify the subscription ID of a project for BigQuery monitoring. | |
com.unraveldata.bigquery.credentials.<project-id> Specify service key file for Google API authentication. | |
Multi-cluster Hadoop | |
unravel.hadoop_module.platform Set this property when you install the core node on a server where there is no Hadoop configuration. | unravel |
Impala | |
com.unraveldata.impala.http.conn.timeout.ms HTTP connection time out for impala connector. | 2000 ms |
com.unraveldata.impala.http.read.timeout.ms HTTP read time out for impala connector. | 6000 ms |
com.unraveldata.impala.http.backoff.ms HTTP client back-off time for impala connector. Time for which HTTP client sleeps before re-attempting after unsuccessful read attempt. | 1000 ms |
com.unraveldata.impala.query.max.retries Specifies the number of retries that will be done to fetch the profile tree for any impala query. | 5 |
LDAP | |
com.unraveldata.ldap.domain Contains a real domain name. | - |
Live metrics | |
com.unraveldata.metrics.merge.cache Specify the count of messages to be stored in the cache or memory before storing in the database. For example, if com.unraveldata.metrics.merge.cache = 30 and unravel.metrics.factor = 5 (seconds), then 30 * 5 = 150 seconds. The data is pushed to the database every 150 seconds. For 150 seconds, the data is kept in memory. | 6 |
com.unraveldata.metrics.sample.count.max Specify the maximum count of data points for sampling purposes. | 16 |
com.unraveldata.metrics.data.points The number of data points to be stored in a single database row. | 44 |
Role-based Access Control (RBAC) | |
com.unraveldata.rbac.roles A comma-separated list of custom roles. | - |
com.unraveldata.rbac.role.<role>.users A comma-separated list of users that can be assigned to a role. The role can be admin, readonlyAdmin, or custom_role, which is set using com.unraveldata.rbac.roles property. | |
com.unraveldata.rbac.role.<role>.groups A comma-separated list of groups that can be assigned to a role. The role can be admin, readonlyAdmin, or custom_role, which is set using com.unraveldata.rbac.roles property. | |
com.unraveldata.rbac.role.<role>.data.user.filter User data filter, if enabled data is filtered on the username. The role can be admin or custom_role, which is set using com.unraveldata.rbac.roles property. | FALSE |
com.unraveldata.rbac.role.<role>.data.tags.filter Tags data filter, if enabled data is filtered on the user tags. The role can be admin or custom_role, which is set using com.unraveldata.rbac.roles property. | FALSE |
com.unraveldata.rbac.role.<role>.data.es.query Query data filter, if enabled data is filtered on the given valid ElasticSearch query. The role can be admin or custom_role, which is set using com.unraveldata.rbac.roles property. | |
com.unraveldata.rbac.role.<role>.data.field.<field> Fields data filter, if enabled data is filtered on the supported ElasticSearch fields for the given values. The <field> should be in the supported field list. The role can be user or custom_role, which is set using com.unraveldata.rbac.roles property. | |
com.unraveldata.rbac.role.<role>.views The views that are shown for a specific role. This is configured based on the View ID list. The role can be user or custom_role, which is set using com.unraveldata.rbac.roles property. | clusters.resources, clusters.chargeback, compute, jobs |
The property com.unraveldata.ldap.customLDAPQuery is changed to a new format. This can be specified in two formats as follows:
Generic format (Same value across all LDAP servers): com.unraveldata.ldap.customLDAPQuery:
LDAP specific format (Different value for a specific LDAP server): com.unraveldata.ldap.<ldap_id>.custom.query.filter
None
Old property name | New property name |
|---|---|
Role-based Access Control (RBAC) | |
com.unraveldata.saml.groupFilter Restricts access to a few selected SAML groups. | com.unraveldata.login.groupFilter |
com.unraveldata.rbac.default Determines how an end-user's views are filtered when no specific tags are set for them. | com.unraveldata.rbac.user.field |
com.unraveldata.rbac.ldap.tags / com.unraveldata.rbac.saml.tags A comma-separated list of the tags. For example, proj, dept. | com.unraveldata.login.tags |
com.unraveldata.rbac.ldap.tag.<TAG_KEY>.regex.find / com.unraveldata.rbac.saml.tag.<TAG_KEY>.regex.find Defines regular expression used to extract a tag_value for a given tag_key. | com.unraveldata.login.tag.<tag_name>.regex.find |
com.unraveldata.login.admins Defines admin users. | com.unraveldata.rbac.role.admin.users |
com.unraveldata.login.admins.readonly Defines read-only admin. users | com.unraveldata.rbac.role.readonlyAdmin.users |
com.unraveldata.login.admins.groups / com.unraveldata.login.admins.ldap.groups / com.unraveldata.login.admins.saml.groups Defines admin groups. | com.unraveldata.rbac.role.admin.groups |
com.unraveldata.login.admins.readonly.groups / com.unraveldata.login.admins.readonly.groups / com.unraveldata.login.admins.readonly.saml.groups Defines read-only admin groups. | com.unraveldata.rbac.role.readonlyAdmin.groups |
com.unraveldata.ldap.verbose
com.unraveldata.login.hbase.groups
com.unraveldata.login.clusters
com.unraveldata.login.clusters.<clusterid>.groups
com.unraveldata.ngui.user.mode
4.7.2.2
Property/Description | Default |
|---|---|
LDAP | |
com.unraveldata.ldap.ids A colon-separated list of LDAP servers (domains). In case this property is not defined then the default value is considered. | default_ldap_id |
com.unraveldata.ldap.default.id The default LDAP server from the com.unraveldata.ldap.ids list. If this property is not defined, the first LDAP server from com.unraveldata.ldap.ids is used | - |
The following existing properties are updated for consistency and replaced. However backward compatibility is provided for all the existing properties.
LDAP
Existing properties | New properties | |
|---|---|---|
Generic usage | LDAP specific usage | |
com.unraveldata.ldap.baseDN | com.unraveldata.ldap.base.dn | com.unraveldata.ldap.<ldap_id>.base.dn |
com.unraveldata.ldap.bind_dn | com.unraveldata.ldap.bind.dn | com.unraveldata.ldap.<ldap_id>.bind.dn |
com.unraveldata.ldap.bind_pw | com.unraveldata.ldap.bind.pw | com.unraveldata.ldap.<ldap_id>.bind.pw |
com.unraveldata.ldap.groupClassKey | com.unraveldata.ldap.group.class | com.unraveldata.ldap.<ldap_id>.group.class |
com.unraveldata.ldap.groupDNPattern | com.unraveldata.ldap.group.dn.pattern | com.unraveldata.ldap.<ldap_id>.group.dn.pattern |
com.unraveldata.ldap.groupMembershipKey | com.unraveldata.ldap.group.member.attr | com.unraveldata.ldap.<ldap_id>.group.member.attr |
com.unraveldata.ldap.groupSearchMethods | com.unraveldata.ldap.group.search.methods | com.unraveldata.ldap.<ldap_id>.group.search.methods |
com.unraveldata.ldap.guidKey | com.unraveldata.ldap.uid.attr | com.unraveldata.ldap.<ldap_id>.uid.attr |
com.unraveldata.ldap.mailAttribute | com.unraveldata.ldap.mail.attr | com.unraveldata.ldap.<ldap_id>.mail.attr |
com.unraveldata.ldap.realUserAttribute | com.unraveldata.ldap.real.uid.attr | com.unraveldata.ldap.<ldap_id>.real.uid.attr |
com.unraveldata.ldap.userDNPattern | com.unraveldata.ldap.user.dn.pattern | com.unraveldata.ldap.<ldap_id>.user.dn.pattern |
com.unraveldata.ldap.groupFilter | com.unraveldata.login.groupFilter | NA |
com.unraveldata.ldap.userFilter | com.unraveldata.login.userFilter | NA |
com.unraveldata.ldap.SAMAccountNameFilter | com.unraveldata.ldap.sAMAccountName.enabled | com.unraveldata.ldap.<ldap_id>.sAMAccountName.enabled |
com.unraveldata.ldap.groupQueryFilter | com.unraveldata.ldap.group.query.filter | com.unraveldata.ldap.<ldap_id>.group.query.filter |
None
None
com.unraveldata.ldap.Domain
This property is deprecated, however, backward compatibility is provided. com.unraveldata.ldap.default.id is used in place of the deprecated property.
4.7.2.0
Property/Description | Default |
|---|---|
Azure Active Directory (AAD) Properties to be updated in | |
tenantID ID of the AAD directory. | - |
Migration | |
com.unraveldata.migrationplanning.vmprices.<cloud provider> Set the cloud provider and the path to the CSV file which contains the custom prices for instances. The following values can be used for the cloud provider:
| - |
Insights Overview | |
com.unraveldata.valuedashboard.enabled Enables the Insights Overview tab on Unravel UI. | false |
Databricks | |
com.unraveldata.agent.metrics.ganglia_enabled Enables Ganglia metrics, which is supported by the Btrace agent. To enable Ganglia metrics, Dcom.unraveldata.agent.metrics.ganglia_enabled is set to true in extraJavaOptions. | |
Datastore | |
com.unraveldata.datastore.rest.api.timeout.millis Specifies the timeout period for different APIs used by UI to fetch data. | 120000 |
AAD Integration
unravel_mapping property includes the following:
username: Any username attribute to use.
group: Group name.
For example:
"unravel_mapping": { "username": "upn", "group": "name" }For more details refer to AAD Authentication.
None
None
None
4.7.1.0
Property/Description | Default |
|---|---|
Migration | |
com.unraveldata.migrationplanning.supported.cloudproviders Controls the cloud providers that must be displayed for selection for the migration reports. Values are as follows:
You can provide a comma-separated list of cloud providers when you specify this property. For example: com.unraveldata.migrationplanning.supported.cloudproviders EMR,EC2,HDI | - |
com.unraveldata.migrationplanning.blacklist Provides the path to a CSV or TXT file that can be used to blacklist specific VM instances for all cloud providers. The CSV or TXT file contains a comma-separated list of VM instances that must not be displayed for the migration reports. For example: com.unraveldata.migrationplanning.blacklist /opt/unravel/data/migration/blacklist.txt. | - |
com.unraveldata.migrationplanning.blacklist. Provides the path to a CSV or TXT file that can be used to blacklist specific VM instances for specific cloud providers. The CSV or TXT file contains a comma-separated list of VM instances that must not be displayed for the migration reports. The values for
For example: com.unraveldata.migrationplanning.blacklist.EMR /opt/unravel/data/migration/blacklist.txt. | - |
Azure Active Directory (AAD) integration Properties to be updated in | |
clientID Client ID from Overview > Essentials > Application (client) ID. | - |
clientSecret Client secret. | - |
callbackURL Redirect URL to consume login response from AAD. | - |
logoutCallbackURL Redirect URL to consume logout response from AAD. | - |
logoutEnabled Enable/disable logout on AAD when the user logs out from Unravel. | - |
unravel_mapping Maps the AAD attributes to Unravel attributes. | - |
Yarn log processing for Tez apps | |
com.unraveldata.yarn.log.delayForRetry Delay in milliseconds for consecutive retries when loading the yarn log files. | 2000 ms |
com.unraveldata.yarn.log.maxAttempts A maximum number of attempts for loading the yarn log files. | 3 |
None
None
None
com.unraveldata.migrationplanning.workloadfit.max.apps
Configures the limit on the maximum number of apps shown in a single slice of workload fit report and heatmap generation.