Small files and file reports

Home

Small files and file reports

Note

You must restart the unravel_ondemand daemon for any changes to take effect

Property/Description	Unit	Default
com.unraveldata.ngui.sfhivetable.schedule.interval Frequency, in days, in which to trigger FSimage extraction, for example, every 3 days The scheduler schedules extraction relative to the 1st of the month and then sets each extraction such that it 1st, 1st + X days, 1st + 2X days until 1st + nX days crosses into the next month, at which point the schedule resets to the 1st. See below for an example. Format:`X`d	day	1d
com.unraveldata.ngui.sfhivetable.schedule.time Specify the time to download in hours (using 24 hour time) the FSimage. Format: two digits between 00 and 23.	two digits (member of set)	00
unravel.python.reporting.files.external_fsimage_dir Directory for fsimage when skip_fetch_fsimage=true. The fsimage externally fetched is expected to be in this directory. Unravel uses the latest file in this directory which starts with " fsimage_". This directory must be different than the Unravel's internal directory, i.e., /srv/unravel/tmp/reports/fsimage.	string	-
unravel.python.reporting.files.skip_fetch_fsimage If hdfs admin privileges can not be granted, set this to true to allow Unravel's Ondemand process to use an externally fetched FSimage. `true`: Ondemand etl_fsimage process does not fetch FSimage from name node. Instead, the FSimage is expected to be available in directory specified by unravel.python.reporting.files.external_fsimage_dir.	boolean	false
unravel.python.reporting.fsimage.run_mode Unravel uses Hive or Spark to process the FSImage. Spark is the recommended mode for Unravel 4.6.0.0 or later. Member of set: spark, hive	string (member of set)	spark

Note

In order for the configuration changes to take effect, unravel_ondemand and unravel_ngui daemons need to be restarted.

/etc/init.d/unravel_ngui restart
/etc/init.d/unravel_ondemand restart

Property/Description	Unit	Default
unravel.python.reporting.files.disable Enables or disables Unravel ability to generate Small Files and File Reports. Note `false`: enables the Small Files and Files reports in both the backend and UI. `true`: disables the Small Files and Files reports. in both the backend and UI.	boolean	false
unravel.python.reporting.files.hive_database Hive Database where Ondemand creates five Hive tables (four temporary, one permanent) for Small Files/File Reports. When not set, tables are created in the default Hive Database. In addition, the Hive queries used for this feature run against default MR queue. It must point to a valid Hive database.	string	default database
unravel.python.reporting.files.hive_mr_queue The Hcive queries ran by Ondemand process run against this MR queue. It must point to a valid MR queue.	string	default

Property/Description

Set by user

Unit

Default

unravel.python.reporting.files.disable

Enables or disables Unravel ability to generate Small Files and File Reports.

Note

false: enables the Small Files and Files reports in both the backend and UI.

true: disables the Small Files and Files reports. in both the backend and UI.

boolean

false

unravel.python.reporting.files.hive_database

Hive Database where Ondemand creates five Hive tables (four temporary, one permanent) for Small Files/File Reports. When not set, tables are created in the default Hive Database. In addition, the Hive queries used for this feature run against default MR queue. It must point to a valid Hive database.

string

default database

unravel.python.reporting.files.hive_mr_queue

The Hcive queries ran by Ondemand process run against this MR queue. It must point to a valid MR queue.

string

default

Note

These are global properties which apply to both Small Files Report and File Reports. They each have equivalent "local" properties. The "local" property takes precedence over the equivalent global property.

Should you unset/delete any of the below properties and their equivalent properties in Small Files Report or Files Report, Unravel has hard-coded values to ensure your reports are generated.

Property/Description	Set by user	Unit	Default
unravel.python.reporting.files.files_use_avg_file_size_flag `true`: average of all the files is used against the threshold criteria and either all the files are accepted and counted or rejected and not counted as per the criteria. `false`: absolute file size is used against the threshold criteria and a file is accepted/counted or rejected/not counted as per the criteria.	Optional	boolean	-
unravel.python.reporting.files.min_parent_dir_depth Directory depth to end the search at. For instance, if depth=2, search begins below two levels. Give given HDFS_root/one/two the search starts in the directory two	Optional	count	-
unravel.python.reporting.files.max_parent_dir_depth Directory depth to end search at. Maximum is 50. For instance, if depth=5 given HDFS_root/one/two/three/four/five/six/seven the search ends at five.	Optional	count	-
unravel.python.reporting.drill_down_subdirs_flag When true a file is accounted for (listed) in all of its ancestors. `false`: a file is accounted in only its immediate parent. This allows Unravel to find a specific directory with maximum number of files matching the size criteria. `true`: lists each file with its ancestors. For example given the directory structure is /one/two false: / - lists files in /. /one - lists files in one. /one/two - lists files in /one/two. true: / - lists files in /, /one, and /one/two. /one - lists files in /one, and /one/two. /one/two - lists files in /one/two.	Optional	boolean	false

In this section:

Was this helpful?

Would you like to provide feedback? Just click here to suggest edits.

Home

Small files and file reports

Note

Note

Note

Note

Search results