OnDemand configurations

These configurations control the features which use OnDemand:

Required

Property/Description	Set by user	Unit	Default
unravel.server.ip FQDN or IP-Address of Unravel Server	Required	string	-
com.unraveldata.python.enabled Enable/disable all ondemand reports and Sessions features in UI. (This property is configured during the OnDemand installation.)		boolean	true

Property/Description

Set by user

Unit

Default

unravel.server.ip

FQDN or IP-Address of Unravel Server

Required

string

com.unraveldata.python.enabled

Enable/disable all ondemand reports and Sessions features in UI. (This property is configured during the OnDemand installation.)

boolean

true

Property/Description	Set by user	Unit	Default
unravel.hive.server2.host FQDN or IP-Address of the HiveServer2 instance.		string	-
unravel.hive.server2.port Port for the HiveServer2 instance.		number	10000

Property/Description	Set by user	Unit	Default
com.unraveldata.ngui.proxy.celery		string (URL)	http://localhost:5000
unravel.celery.broker.url	Optional	string (URL)	-
unravel.celery.result.backend	Optional		-

Property/Description	Set by user	Unit	Default
javax.jdo.option.ConnectionDriverName JDBC Driver class name for the hive metastore containing the metadata Examples: MySQL:`com.mysql.jdbc.Driver` Oracle:`oracle.jdbc.driver.OracleDriver` Microsoft:`com.microsoft.sqlserver.jdbc.SQLServerDriver`	Required	string	-
javax.jdo.option.ConnectionPassword Password used to access the metastore.	Required	string	-
javax.jdo.option.ConnectionUserName Username used to access the metastore.	Required	string	-
javax.jdo.option.ConnectionURL JDBC connection string for the metastore containing the metadata of the form: jdbc:`DB_Driver`://`HOST`:`PORT`/hive Example: Oracle:`jdbc:oracle:thin:@//prodHost:1521/ORCL` Microsoft:`jdbc:sqlserver://jdbc_url`	Required	string (URL)	-

If KERBEROS (unravel.hive.server2.authentication=KERBEROS)

set unravel.hive.server2.kerberos.service.name=hive
define Kerberos properties.

See Configuring Forecasting and Migration Planning Reports for required cluster configurations.

See Configuring FSImage for complete configuration requirements.Configuring FSImage

Configuration properties for reports

Migration planning

Note

Note: As of 29 July 2019, the default is false for new installations for Unravel 4.5.3.0 and later.

Property/Description	Set by user	Unit	Default
unravel.python.reporting.cloudreport.enable Controls the generation of Migration Reports. `true`: Reports can be created. `false`: Reports cannot be created.		boolean	true (See note.)

Property/Description

Set by user

Unit

Default

unravel.python.reporting.cloudreport.enable

Controls the generation of Migration Reports.

true: Reports can be created.

false: Reports cannot be created.

boolean

true (See note.)

Queue analysis

Property/Description	Unit	Default
com.unraveldata.report.queue.metrics.sensor.enabled Enables or disables the queue metric sensor.	boolean	true
com.unraveldata.report.queue.http.retries The number YARN Resource Manager HTTP connection retries.	count	3
com.unraveldata.report.queue.http.retry.period.msec Period to wait between attempts to connect to the YARN Resource Manager HTTP.	ms	0
com.unraveldata.report.queue.http.timeout.msec YARN Resource Manager HTTP connection timeout.	ms	10000
com.unraveldata.report.queue.poll.interval.msec How often queue metric sensor polls YARN Resource Manager.	ms	60000
unravel.python.queueanalysis.daterange.span UI report date picker range.	days	30
unravel.python.queueanalysis.metrics.scale UI rendered graph metrics scale factor.	integer	1000
com.unraveldata.report.excluded.queues The queues to exclude from metric collection and the queue analysis report.	CSL	-

Sessions

Property/Description	Unit	Default
com.unraveldata.session.enabled Enables Sessions features tab in the UI. `true`: Sessions enabled. `false`: Sessions disabled. You must restart the `ngui` and `ondemand` daemons when changing the value.	boolean	true
com.unraveldata.session.max.autotune.runs Maximum number of runs allowed in an auto-tune session.	count	8
com.unraveldata.session.dynamicAllocation.enabled If set to true Spark sessions recommends dynamic allocation. If users don't want that they need to explicitly set it to false.	boolean	true
com.unraveldata.session.spark1.submit.command The command to use for Spark1 applications while applying the session run. This path should be fully qualified.	string	spark-submit
com.unraveldata.session.spark2.submit.command The command to use for Spark2 applications while applying the session run. This path should be fully qualified.	string	spark2-submit

Small files and file reports

Property/Description	Set by user	Unit	Default
com.unraveldata.ngui.sfhivetable.schedule.interval Frequency, in days, in which to trigger FSimage extraction, for example, every 3 days The scheduler schedules extraction relative to the 1st of the month and then sets each extraction such that it 1st, 1st + X days, 1st + 2X days until 1st + nX days crosses into the next month, at which point the schedule resets to the 1st. See below for an example. Format:`X`d		day	1d
com.unraveldata.ngui.sfhivetable.schedule.time Specify the time to download in hours (using 24 hour time) the FSimage. Format: two digits between 00 and 23.		two digits (set member)	00

Property/Description

Set by user

Unit

Default

com.unraveldata.ngui.sfhivetable.schedule.interval

Frequency, in days, in which to trigger FSimage extraction, for example, every 3 days

The scheduler schedules extraction relative to the 1st of the month and then sets each extraction such that it 1st, 1st + X days, 1st + 2X days until 1st + nX days crosses into the next month, at which point the schedule resets to the 1st. See below for an example.

Format:Xd

day

com.unraveldata.ngui.sfhivetable.schedule.time

Specify the time to download in hours (using 24 hour time) the FSimage. Format: two digits between 00 and 23.

two digits

(set member)

Note

You must restart the unravel_ondemand and unravel_ngui daemons for any configuration changes to take effect. Execute the following commands:

/etc/init.d/unravel_ngui restart
/etc/init.d/unravel_ondemand restart

Property/Description	Unit	Default
unravel.python.reporting.files.disable Enables or disables Unravel ability to generate Small Files and File Reports. Note `false`: enables the Small Files and Files reports in both the backend and UI. `true`: disables the Small Files and Files reports. in both the backend and UI.	boolean	false
unravel.python.reporting.files.hive.database Hive Database where Ondemand creates five Hive tables (four temporary, one permanent) for Small Files/File Reports. When not set, tables are created in the default Hive Database. In addition, the Hive queries used for this feature run against the default MR queue. It must point to a valid Hive database.	string	default database
unravel.python.reporting.files.hive.mr.queue The Hive queries ran by the Ondemand process run against this MR queue. It must point to a valid MR queue.	string	default

Property/Description

Set by user

Unit

Default

unravel.python.reporting.files.disable

Enables or disables Unravel ability to generate Small Files and File Reports.

Note

false: enables the Small Files and Files reports in both the backend and UI.

true: disables the Small Files and Files reports. in both the backend and UI.

boolean

false

unravel.python.reporting.files.hive.database

Hive Database where Ondemand creates five Hive tables (four temporary, one permanent) for Small Files/File Reports. When not set, tables are created in the default Hive Database. In addition, the Hive queries used for this feature run against the default MR queue. It must point to a valid Hive database.

string

default database

unravel.python.reporting.files.hive.mr.queue

The Hive queries ran by the Ondemand process run against this MR queue. It must point to a valid MR queue.

string

default

Note

These are global properties that apply to both Small Files Report and File Reports. They each have equivalent "local" properties. The "local" property takes precedence over the equivalent global property.

Should you unset/delete any of the below properties and their equivalent properties in Small Files Report or Files Report, Unravel has hard-coded values to ensure your reports are generated.

Property/Description	Set by user	Unit	Default
unravel.python.reporting.files.files.use.avg.file.size_flag `true`: average of all the files is used against the threshold criteria and either all the files are accepted and counted or rejected and not counted as per the criteria. `false`: absolute file size is used against the threshold criteria and a file is accepted/counted or rejected/not counted as per the criteria.	Optional	boolean	-
unravel.python.reporting.files.min.parent.dir.depth Directory depth to end the search at. For instance, if depth=2, a directory having depth more than or equal to 2 can be included in the report. Thus a directory such as`/one/two` or its sub-directories can be included in the report.	Optional	count	-
unravel.python.reporting.files.max.parent.dir.depth Directory depth to end search at. The maximum is 50. For instance, if depth=5, a directory having a depth less than or equal to 5 can be included in the report. Thus a directory, for example, `/one/two/three/four/five` or it's ancestor directories can be included in the report.	Optional	count	-
unravel.python.reporting.drill.down.subdirs.flag Determines to which ancestor directories a filesize is accounted for. If this is True then the file size is accounted for all ancestor directories. If this is False the file size is accounted for only the parent directory. `true` a file is accounted for (listed) in all of its ancestor's file lists. `false`: a file is accounted in only its immediate parent. This allows Unravel to find a specific directory with a maximum number of files matching the size criteria. For example, given a file under a directory /one/two/three. when `true` the filesize is accounted to four ancestor directories, that is, `/`, `/one`, `/one/two` and `/one/two/three`. when `false` the file size is accounted to only the parent directory, that is, `/one/two/three` /one - lists files in one. /one/two - lists files in /one/two. true: / - lists files in /, /one, and /one/two. /one - lists files in /one, and /one/two. /one/two - lists files in /one/two.	Optional	boolean	true

Small Files

Note

You must restart the unravel_ondemand daemon for any changes to take effect.

Property/Description	Unit	Default
unravel.python.reporting.small.files.use.avg.file.size.flag `true`: average of all the files is used against the threshold criteria and either all the files are accepted and counted or rejected and not counted as per the criteria. `false`: absolute file size is used against the threshold criteria and a file is accepted/counted or rejected/not counted as per the criteria.	boolean	true
unravel.python.reporting.files.small.files.min.parent.dir.depth Directory depth to end the search at. For instance, if depth=2, a directory having depth more than or equal to 2 can be included in the report. Thus a directory such as`/one/two` or its sub-directories can be included in the report.	count	0
unravel.python.reporting.files.small.files.max.parent.dir.depth Directory depth to end search at. The maximum is 50. For instance, if depth=5, a directory having a depth less than or equal to 5 can be included in the report. Thus a directory, for example, `/one/two/three/four/five` or it's ancestor directories can be included in the report.	count	10
unravel.python.reporting.files.small.files.drill.down.subdirs.flag Determines to which ancestor directories a filesize is accounted for. If this is True then the file size is accounted for all ancestor directories. If this is False the file size is accounted for only the parent directory. `true` a file is accounted for (listed) in all of its ancestor's file lists. `false`: a file is accounted in only its immediate parent. This allows Unravel to find a specific directory with a maximum number of files matching the size criteria. For example, given a file under a directory /one/two/three. when `true` the filesize is accounted to four ancestor directories, that is, `/`, `/one`, `/one/two` and `/one/two/three`. when `false` the file size is accounted to only the parent directory, that is, `/one/two/three`	boolean	true

File Reports

The following properties are for Huge files report.

Property/Description	Unit	Default
unravel.python.reporting.files.huge.files.drill.down.subdirs.flag Determines to which ancestor directories a filesize is accounted for. If this is True then the file size is accounted for all ancestor directories. If this is False the file size is accounted for only the parent directory. For example, given a file under a directory /one/two/three. when `true` the filesize is accounted to four ancestor directories, that is, `/`, `/one`, `/one/two` and `/one/two/three`. when `false` the file size is accounted to only the parent directory, that is, `/one/two/three`	boolean	true
unravel.python.reporting.files.huge.files.max.parent.dir.depth Maximum depth of a directory to be included in the report. Directory depth to end search at. The maximum is 50. For instance, if depth=5, a directory having a depth less than or equal to 5 can be included in the report. Thus a directory, for example, `/one/two/three/four/five` or it's ancestor directories can be included in the report.	integer	10
unravel.python.reporting.files.huge.files.min.files The minimum number of files which must meet the threshold size for a directory to be included.	integer	1
unravel.python.reporting.files.huge.files.min.parent.dir.depth Minimum depth of a directory to be included in the report. Directory depth to end the search at. For instance, if depth=2, a directory having depth more than or equal to 2 can be included in the report. Thus a directory such as`/one/two` or its sub-directories can be included in the report.	integer	0
unravel.python.reporting.files.huge.files.threshold.size File size must be greater than or equal to the threshold_size for it to be counted.	bytes	100 * 1024 * 1024 * 1024
unravel.python.reporting.files.huge.files.top.n.dirs Number of directories to be included in the report.	integer	200
unravel.python.reporting.files.huge.files.use.avg.file.size.flag `true`: average of all the files is used against the threshold criteria and either all the files are accepted and counted or rejected and not counted as per the criteria. `false`: absolute file size is used against the threshold criteria and a file is accepted/counted or rejected/not counted as per the criteria.	boolean	false

The following properties are for Medium files report.

Property/Description	Unit	Default
unravel.python.reporting.files.medium.files.drill.down.subdirs.flag Determines to which ancestor directories a filesize is accounted for. If this is True then the file size is accounted for all ancestor directories. If this is False the file size is accounted for only the parent directory. `true` a file is accounted for (listed) in all of its ancestor's file lists. `false`: a file is accounted in only its immediate parent. This allows Unravel to find a specific directory with a maximum number of files matching the size criteria. For example, given a file under a directory /one/two/three. when `true` the filesize is accounted to four ancestor directories, that is, `/`, `/one`, `/one/two` and `/one/two/three`. when `false` the file size is accounted to only the parent directory, that is, `/one/two/three`	boolean	true
unravel.python.reporting.files.medium.files.max.parent.dir.depth Maximum depth of a directory to be included in the report. For instance, if depth=5, a directory having a depth less than or equal to 5 can be included in the report. Thus a directory, for example, `/one/two/three/four/five` or it's ancestor directories can be included in the report.	integer	10
unravel.python.reporting.files.medium.files.max.threshold.size The file size must be less than or equal to the max_threshold_size for it to be counted.	bytes	10 * 1024 * 1024 * 1024
unravel.python.reporting.files.medium.files.min.files Minimum depth of a directory to be included in the report.	integer	5
unravel.python.reporting.files.medium.files.min.parent.dir.depth Directory depth to end the search at. For instance, if depth=2, a directory having depth more than or equal to 2 can be included in the report. Thus a directory such as`/one/two` or its sub-directories can be included in the report.	integer	0
unravel.python.reporting.files.medium.files.min.threshold.size The file size must be greater than or equal to the min_threshold_size for it to be counted.	integer	5 * 1024 * 1024 * 1024
unravel.python.reporting.files.medium.files.top.n.dirs Maximum number of directories to display, that is, only display the results for the top N directories.	integer	200
unravel.python.reporting.files.medium.files.use.avg.file.size.flag `true`: average of all the files is used against the threshold criteria and either all the files are accepted and counted or rejected and not counted as per the criteria. `false`: absolute file size is used against the threshold criteria and a file is accepted/counted or rejected/not counted as per the criteria.	boolean	false

The following properties are for Tiny files report.

Property/Description	Unit	Default
unravel.python.reporting.files.tiny.files.drill.down.subdirs.flag Determines to which ancestor directories a filesize is accounted for. If this is True then the file size is accounted for all ancestor directories. If this is False the file size is accounted for only the parent directory. `true` a file is accounted for (listed) in all of its ancestor's file lists. `false`: a file is accounted in only its immediate parent. This allows Unravel to find a specific directory with a maximum number of files matching the size criteria. For example, given a file under a directory /one/two/three. when `true` the filesize is accounted to four ancestor directories, that is, `/`, `/one`, `/one/two` and `/one/two/three`. when `false` the file size is accounted to only the parent directory, that is, `/one/two/three`	boolean	true
unravel.python.reporting.files.tiny.files.max.parent.dir.depth Maximum depth of a directory to be included in the report. For instance, if depth=5, a directory having a depth less than or equal to 5 can be included in the report. Thus a directory, for example, `/one/two/three/four/five` or it's ancestor directories can be included in the report.	integer	10
unravel.python.reporting.files.tiny.files.min.files Minimum depth of a directory to be included in the report.	integer	10
unravel.python.reporting.files.tiny.files.min.parent.dir.depth Directory depth to end the search at. For instance, if depth=2, a directory having depth more than or equal to 2 can be included in the report. Thus a directory such as`/one/two` or its sub-directories can be included in the report.	integer	0
unravel.python.reporting.files.tiny.files.threshold.size The file size must be less than or equal to the max_threshold_size for it to be counted.	bytes	100 * 1024
unravel.python.reporting.files.tiny.files.top.n.dirs Maximum number of directories to display, in other words, only display the results for the top N directories.	integer	200
unravel.python.reporting.files.tiny.files.use.avg.file.size.flag `true`: average of all the files is used against the threshold criteria and either all the files are accepted and counted or rejected and not counted as per the criteria. `false`: absolute file size is used against the threshold criteria and a file is accepted/counted or rejected/not counted as per the criteria.	boolean	false

The following properties are for Empty files report.

Property/Description	Unit	Default
unravel.python.reporting.files.empty.files.drill.down.subdirs.flag Determines to which ancestor directories a filesize is accounted for. If this is True then the file size is accounted for all ancestor directories. If this is False the file size is accounted for only the parent directory. `true` a file is accounted for (listed) in all of its ancestor's file lists. `false`: a file is accounted in only its immediate parent. This allows Unravel to find a specific directory with a maximum number of files matching the size criteria. For example, given a file under a directory /one/two/three. when `true` the filesize is accounted to four ancestor directories, that is, `/`, `/one`, `/one/two` and `/one/two/three`. when `false` the file size is accounted to only the parent directory, that is, `/one/two/three`	boolean	true
unravel.python.reporting.files.empty.files.max.parent.dir.depth Maximum depth of a directory to be included in the report. For instance, if depth=5, a directory having a depth less than or equal to 5 can be included in the report. Thus a directory, for example, `/one/two/three/four/five` or it's ancestor directories can be included in the report.	integer	10
unravel.python.reporting.files.empty.files.min.files Minimum depth of a directory to be included in the report.	integer	10
unravel.python.reporting.files.empty.files.min.parent.dir.depth Directory depth to end the search at. For instance, if depth=2, a directory having depth more than or equal to 2 can be included in the report. Thus a directory such as`/one/two` or its sub-directories can be included in the report.	integer	0
unravel.python.reporting.files.empty.files.top.n.dirs Maximum number of directories to display, in other words, only display the results for the top N directories.	integer	200
unravel.python.reporting.files.empty.files.use.avg.file.size.flag `true`: average of all the files is used against the threshold criteria and either all the files are accepted and counted or rejected and not counted as per the criteria. `false`: absolute file size is used against the threshold criteria and a file is accepted/counted or rejected/not counted as per the criteria.	boolean	false

Top X

Property/Description	Set by user	Unit	Default
com.unraveldata.ngui.topx.enabled Controls the generation of Top X reports. `true`: report enabled. `false`: report are disabled.		boolean	true

In this section:

Home

OnDemand configurations

Required

Configuration properties for reports

Migration planning

Note

Queue analysis

Sessions

Small files and file reports

Note

Note

Note

Small Files

Note

File Reports

Top X

Search results