Home

Elasticsearch storage requirements on the Unravel Node

When you are using HBase you must ensure you have enough disk space to accommodate the Elasticsearch Index on the Unravel node. The volume (size) of metrics that Unravel collects is dependent on the following:

  • Number of

    • Master servers

    • Region servers

    • Tables

    • Table regions

  • com.unraveldata.hbase.metric.poll.interval

The total disk size you need for one week's metrics is equal to

  • (total size per poll) * (number of polls per week)

    where

    • total size per poll = (total documents per poll) * (avg doc size)

      • total documents per poll = (12 * #_of_Master_Servers) + 3 + (19 * #_of_Region_Servers)

      • avg doc size = ±100 bytes

    • number of polls per week = (7 * 24 * 60) / com.unraveldata.hbase.metric.poll.interval

Important

avg doc size is a rough estimate and is an average across master, region server, table, table region, and OS metrics. It varies depending on the content, for example, namespace, host name, table name, etc.

The polling interval determines both the granularity of the collected data and the storage size needed. The more frequent the collection the larger the storage space required. For instance, if you need 100 GB using a five-minute polling, you need 50 GB for a 10-minute interval.