Skip to main content

Home

Configuring FSImage

This topic explains how to configure FSimage which is triggered by default. See Toggling FSimage status for how to disable it.

FSImage requires configuration for some Data Page features and content, specifically to:

Important

This feature runs as hdfs user.

You must restart the unravel_ondemand and unravel_ngui daemons for any configuration changes to take effect. Execute the following commands:

/etc/init.d/unravel_ngui restart
/etc/init.d/unravel_ondemand restart
Toggling FSimage status

Set the following property in /usr/local/unravel/etc/unravel.properties.

Property/Description

Default

unravel.python.reporting.files.disable

Enables or disables Unravel ability to generate Small Files and File Reports.

Note

false: enables the Small Files and Files reports in both the backend and UI.

true: disables the Small Files and Files reports. in both the backend and UI.

false

1. Determine how Unravel will access FSimage

  • Unravel can run as HDFS admin.

    This allows Unravel to access the FSimage on the Namenode using the hdfs dfsadmin command.

  • Unravel can't run as HDFS admin

    FSimage must be downloaded for Unravel to use it. Create a cron to download it to the Unravel server. The old image must be deleted before running the cron job.

    The download can take up to 10 hours depending on the size of FSimage. The time can determine how often cron is/should be run.

    Edit /usr/local/unravel/etc/unravel.properties and set the following properties.

    Property/Description

    Default

    unravel.python.reporting.files.skip_fetch_fsimage

    If HDFS admin privileges can not be granted, set this to true to allow Unravel's OnDemand process to use an externally fetched FSimage.

    true: OnDemand etl_fsimage process does not fetch FSimage from name node. Instead, the FSimage is expected to be available in directory specified by unravel.python.reporting.files.external_fsimage_dir.

    false

    unravel.python.reporting.files.external_fsimage_dir

    Directory for FSimage when skip_fetch_fsimage=true. The FSimage externally fetched is expected to be in this directory. Unravel uses the latest file in this directory which starts with " fsimage_".

    This directory must be different than the Unravel's internal directory, i.e., /srv/unravel/tmp/reports/fsimage.

    -

    For example,

    unravel.python.reporting.files.skip_fetch_fsimage=true;
unravel.python.reporting.files.external_fsimage_dir=/srv/unravel/tmp/fsimages/reports;
2. Define the following

Edit /usr/local/unravel/etc/unravel.properties and set the following properties.

  • Unravel uses Spark to process the FSimage. See the FSimage properties for a complete list of properties. The following properties define the resources used to process FSimage.

  • If you are using a secured CDH or HDP cluster verify that you have configured Unravel correctly.

Important

As part of configuring the Data page Unravel connects to the Hive Metastore using the JDO properties. To load the specified JDBC driver, the jar file containing the JDBC driver class must be available in <unravel-installation-dir>/share/java.

3. Importing etl_fsimage

The etl_fsimage task imports the latest FSimage from Namenode. The etl_fsimage run time is proportional to the image size, for example

FSimage Size

etl_fsimage

19 GB

24 hours

9 GB

14 hours

4 GB

7 hours

The following two properties determine when the FSimage is imported (interval) and at what time. The default is once a day at 00:00 (midnight).

Important

FSimage is a snapshot that becomes outdated with the passage of time, in other words, the older the image the more it diverges from the real-time structure.

There may be times when you need to import FSimage immediately, such as after Unravel Server is installed or upgraded. Run time following command to trigger the import.

curl -v http://localhost:5000/small-files-etl
4. Unravel UI features

Once FSimage has been successfully fetched you can go the UI to verify

Tips

  1. The relevant log file is /usr/local/unravel/ondemand/logs/unravel_ondemand.out

  2. Run one of the following commands to display the progress of the etl_fsimage task.

    egrep 'ETL_FSIMAGE|FSIMAGE_REPORTS_UTILS' unravel_ondemand.out
    grep etl_fsimage\(\) unravel_ondemand.out

  3. Run one of the following commands to display the progress of the run_small_files which is started whenever Small Files Report is triggered from UI.

    egrep 'SMALL_FILES_REPORT|FSIMAGE_REPORTS_UTILS' unravel_ondemand.out
    grep run_small_files\(\) unravel_ondemand.out
In this section: