Home

v4.6.2.0 RC1 Release Notes

Release Date: 06/03/2020

Notice

The v4.6.2.0 RC1 is a preview release. This release note includes information about supported and unsupported functionalities for Elasticsearch, Logstash, Kibana (ELK), and PostgreSQL. The new configuration properties for v4.6.2.0 RC1 release and the known issues are also included.

Download information

CDH, HDP, and MapR
curl -v https://preview.unraveldata.com/unravel/RPM/4.6.2/unravel-4.6.2.0-onpremise-RC1.rpm -o unravel-4.6.2.0-onpremise-RC1.rpm -u username:password

MD5SUM: eaecd70edbf3ccd8d745e6a0530431e1

Cloud
curl -v https://preview.unraveldata.com/unravel/RPM/4.6.2/unravel-4.6.2.0-cloud-RC1.rpm -o unravel-4.6.2.0-cloud-RC1.rpm -u username:password

MD5SUM: fd04cadb8225d8af3ed7a2b7dd7cc8ec

OnDemand - RHEL 7.6
curl -v https://preview.unraveldata.com/unravel/RPM/4.6.2/ondemand-4.6.2.0-rhel7-RC1.tar.gz -o ondemand-4.6.2.0-rhel7-RC1.tar.gz -u username:password

MD5SUM: ae783cd1e45e131eec4196a4b19cc48c

Elasticsearch, Logstash, Kibana (ELK)

The KPIs of the ELK stack, which comprises of Elasticsearch (search engine), Logstash (ingest pipeline), and Kibana (visualization tool), can be monitored from Unravel. This section lists the supported platform, versions, capabilities, configuration properties, and APIs available for monitoring ELK metrics in Unravel.

Supported platforms and versions
  • CDH 6.3.3 (CentOS 7.6)

  • ELK version 6.8 and 5.5

Supported capabilities

Unravel supports metrics and graphically represents the KPI information for Elasticsearch, Logstash, and Kibana.

The following cluster level, node level, as well as the indices level, details, metrics, and graphs are supported for Elasticsearch:

Cluster metrics

Metrics

Description

Status

Status of the cluster.

Nodes

Total number of nodes in the cluster.

Indices

Total count of indices in the cluster.

Total Shards

Total number of shards in the cluster.

Unassigned Shards

Total number of unassigned shards in the cluster.

Documents

Total count of indexed documents in the cluster.

Successful Shards

Total number of successful shards in the cluster.

Active Shards

Total number of active shards in the cluster.

Initializing Shards

Number of initializing shards in the cluster.

Relocating Shards

Number of relocating shards in the cluster.

Cluster graphs

  • Search Rate: This graph plots the rate at which documents are queried in Elasticsearch for the selected cluster.

  • Search Latency: This graph plots the time taken to execute the search request for the selected cluster.

  • Indexing Rate: This graph plots the rate at which documents are indexed in the selected cluster.

  • Indexing Latency: This graph plots the time taken to index the documents in the selected cluster.

Node details

Column

Description

Name

Name of the node.

CPU usage

Total CPU usage for the node.

Disk Total

Total disk space available for the Elasticsearch node.

Disk Used

Total disk space used by the Elasticsearch node.

Load Average

The load average of a CPU in an operating system.

JVM Memory

JVM heap memory used.

Node metrics

Metrics

Description

Documents

The total number of indexed documents in the node.

Free Disk Space

The available disk space in the node.

Heap Percent

Percentage of heap memory used.

Flush Latency

Time taken to flush the indices in a node.

Flush Rate

The rate at which the indices are flushed.

CPU

The CPU utilization by the operating system in percentage.

RAM

RAM utilization in percentage.

Total Read Operations

Total amount of read operations processed.

Total write Operations

Total amount of write operations processed.

Uptime

Time since the node is up and active.

Node graphs

  • JVM Heap: This graph plots the total heap used by Elasticsearch in the JVM for a selected node.

  • Index Memory: This graph plots the heap memory used for indexing in a selected node.

  • CPU Utilization: This graph plots the percentage of CPU usage for the Elasticsearch process in the selected node.

  • System Load: The graph plots the load average per minute for the selected node.

  • Segment Count: This graph plots the maximum segment count for the selected node.

  • Search Latency: This graph plots the average search latency for the selected node.

  • Indexing Latency: This graph plots the average latency for indexing documents in the selected node.

  • Documents: This graph plots the count of indexed documents for the selected node.

Index details

Column

Description

Name

Name of the index.

Docs

Number of indexed documents.

Store Size

Storage size of the indexed documents.

Search Rate

Number of search requests being executed.

Active Shards

Total number of active shards in an index.

Unassigned Shards

Total number of unassigned shards in an index.

Index metrics

Metrics

Description

Flush Total

Total number of flushed indices.

Flushed Latency

Time taken to flush an index.

Flush Rate

The rate at which indices are flushed.

Segments count

Count of segments.

Index graphs

  • Search Rate: This graph plots the number of search requests that are being executed in Elasticsearch.

  • Indexing Latency: This graph plots the time taken to index the documents in Elasticsearch.

  • Indexing Rate: This graph plots the rate at which documents are indexed in Elasticsearch

  • Segments Count: This graph plots the count of indexing segments operation.

  • Fetch Latency: This graph plots the fetch latency for the indices.

  • Fetch Rate: This graph plots the time taken to fetch the query results.

  • Flush Latency: This graph plots the time taken to flush indices.

  • Flush Rate: This graph plots the rate at which indices are flushed.

The following cluster level, node level, as well as the indices level, details, metrics, and graphs are supported for Logstash:

Cluster metrics

Metrics

Description

Nodes

Total number of nodes in the cluster.

Events Received

Average of the number of events flowing into all the nodes, in a selected time period.

Events Emitted

Average of the number of events flowing out of all the nodes, in a selected time period.

Memory

The total heap memory used versus total heap memory available (Heap memory used/Total heap memory available).

Pipelines

Total number of logical pipelines.

Node details

Column

Description

Name

Name of the node.

Jvm Heap Used

JVM heap memory used.

Events Received

Current number of events flowing into the selected node.

Events Emitted

Current number of events flowing out of the selected node.

Events Filtered

Current number of events that are filtered for the selected node.

Node metrics

Metrics

Description

Version

Logstash version.

CPU Usage

Average CPU usage for the selected period of time.

Uptime

Time since the node has been up and active.

Reloads

Successful reloads versus the total number of reloads.

Node graphs

  • Events Received: This graph plots the average of the total events flowing into the selected node, for the specified time period.

  • Events Received Rate: This graph plots the average rate at which events flow into the selected node, for the specified time period.

  • Events Emitted: This graph plots the average of the total events flowing out of the selected node, for the specified time period.

  • Events Emitted Rate: This graph plots the average rate at which events flow out of the selected node, for the specified time period.

  • Cpu Percent: This graph plots the percentage of CPU usage for the Logstash process in the selected node.

  • Jvm Heap: This graph plots the total heap used by Elasticsearch in the JVM for a selected node.

  • System Load: The graph plots the load average per minute for the selected node.

Pipeline details

Column

Description

Name

Name of the node.

Events Received

Current number of events flowing into the selected pipeline.

Events Emitted

Current number of events flowing out of the selected pipeline.

Events Filtered

Current number of events that are filtered for the selected pipeline.

Number of Nodes

Total number of nodes in the pipeline.

Pipeline metrics

Pipelines Metrics

Description

Reload Interval

Time interval that specifies how often Logstash checks the config files for changes in the pipeline (in seconds).

Batch size

Maximum number of events that an individual worker thread collects before executing filters and outputs.

Reloads

Successful reloads versus total number of reloads.

Workers

Number of threads to run for filter and output processing.

Pipeline graphs

  • Events Received: This graph plots the average of the total events flowing into the selected pipeline, for the specified time period.

  • Events Received Rate: This graph plots the average rate at which events flow into the selected pipeline, for the specified time period.

  • Events Emitted: This graph plots the average of the total events flowing out of the selected pipeline, for the specified time period.

  • Events Emitted Rate: This graph plots the average rate at which events flow out of the selected pipeline, for the specified time period.

The following details, metrics, and graphs are supported for Kibana:

Metrics

Metric

Description

Status

Status of the cluster.

Heap Total

Total heap memory available.

Heap Used

Total heap memory used.

Requests Disconnects

Number of disconnected client requests.

Request Total

Total number of client requests received by the Kibana instance.

Response Time Max

Maximum time taken to respond to the client requests received by the Kibana instance.

Concurrent Connections

Total number of concurrent connections to the Kibana instance.

Collection Interval

Time period between data sampling for the metrics.

Kibana graphs

  • Heap Used: This graph plots the total heap memory used by Kibana, in a cluster, for the specified period.

  • Request Total: This graph plots the total number of client requests received by the Kibana instance in the specified period.

  • Request Disconnects: This graph plots the totals no of disconnected requests in a specified period.

  • Concurrent Connections: This graph plots the concurrent connections to the Kibana instance in a specified period.

Supported/Unsupported metrics - ELK version 5.5 and 6.8

The following table list the supported/unsupported metrics in Elasticsearch v5.5 and v6.8:

Metrics

v5.5

v6.8

Nodes - Disk Total

No

Yes

Nodes - Disk Used

No

Yes

Nodes - FREE DISK SPACE

Yes

Yes

The following table listed the supported/unsupported metrics in Logstash v5.5 and v6.8:

Metrics

v5.5

v6.8

All metrics are supported for v5.5 and v6.8.

The following table listed the supported/unsupported metrics in Kibana v5.5 and v6.8:

Metrics

v5.5

v6.8

Heap Total

Time series

Yes

Heap Used

Time series

Yes

Requests Disconnects

No

Yes

Requests Tests

No

Yes

Response Times Max

Time series

Yes

Concurrent Connections

No

Yes

Collection Interval

No

Yes

Graph - Metrics - Heap Used

No

Yes

Requests Total

Time series

Yes

Requests Disconnected

No

Yes

Concurrent Connections

No

Yes

Configuration properties
ELK APIs

PostgreSQL

Supported platforms and versions
  • HDI 4.0 (RHEL 7.8)

  • PostgreSQL version 11.4 and 12.3

Platforms
  • On HDI secure clusters, where Kerberos is enabled, unravel_es daemon does not refresh the Kerberos token after it expires.

    Workaround: Navigate to /usr/local/unravel_es/etc/ and add the following property in the unravel.properties file:

    com.unraveldata.kerberos.kinit_scheduler.enabled=false

For support issues, contact Unravel Support