Home

Kafka APM

The Kafka APM provides multi-cluster support for monitoring:

  • Multi-cluster metrics monitoring.

  • Multi-cluster consumer offset/lag monitoring.

See Kafka Insights for a Use Case example of drilling down into lagging or stalled Consumer Groups.

Clusters > Kafka displays the List of Configured Kafka Clusters.

Click the Cluster Name to bring up the Cluster View.

clusterlist.png
Cluster view

This window has three sections:

  • Key performance indicators

  • Metric graphs

  • Table of Kafka topics

Key performance indicators

  • # Under Replicated Partitions

  • # Offline Partitions

  • # Controller

  • Bytes In/sec

  • Bytes Out/sec

  • Messages in/sec

Kafa-Cluster-WithConsumer-KPIsOnly.png
Metrics

This section has three tabs:

  • Metrics: Graphs the metrics for the data across the entire cluster.

  • Broker: Graphs the metric for the data of the broker you select from the broker list.

  • Topic: Graphs the metric data for a specific topic you select from the topic list.

Metrics

The Metrics and Broker tab graphs all the following Metrics. The Topic tab displays a subset of the Metrics. In addition, subsets of these Metrics are used in the Topic and Consumer Group pages.

While the definition of the metric is the same throughout, the breadth of the collected data is not. Therefore, the implication/interpretation of the metric varies by context. For each metric, we explain:

  • Its definition.

  • Whether it is used in Topic tab or the Topic and Consumer pages' Topic details tab.

  • The data collected for the metric at each point.

  • For some Metrics, the implication or interpretation of data at the point of collection is explained; along with how you can use it to monitor your cluster or improve its performance.

In addition to these Metrics the Topic and Consumer pages Partition tab graphs the partition's Consumer lag and offset.

  • Bytes In Per Second

    The one minute rate of all the bytes flowing into the Kafka network, specifically the Kafka Producers publishing messages. It indicates how much traffic the brokers are receiving from the Producer clients.

    This metric is also displayed in the Topic tab and the Topic and Consumer group views. For any given data point, for that point in time, the graph displays:

    • Metrics tab: The total number of bytes flowing into the cluster across all topics and brokers.

    • Broker tab: The number of bytes flowing into the selected broker.

    • Topic tab: The number of bytes flowing into a particular topic across all brokers.

    • Topic view: The same as the Topic tab, but each broker is represented by a trend line so you can see the data flowing into individual topics.

    • Consumer group view: Similar to the Topic page.

    Analysis:

    • The Brokers tab can help you determine if you need to expand your cluster.

    • There might be situations where it could be possible to identify a broker that is receiving more traffic than others which indicates that there is a need for a partition rebalance.

  • Bytes Out Per Second

    The one minute rate of all the bytes flowing out of the Kafka network, specifically Kafka Consumers consuming messages from the topics. This metric scales differently than Bytes In Per Second, because of the great job Kafka does handling multiple consumers.

    It possible for the outgoing rate to be two to three times the incoming rate for topics being consumed. Before Kafka 0.11.0.0, this metric included the internal replica traffic. Therefore, it showed an outbound rate for a topic that was an actively producing client, even when no consumers were consuming from the same.

    This metric is also displayed in the Topic tab and the Topic and Consumer group pages.

    These graphs are exactly like Bytes In Per Second except they display the bytes out.

  • Messages In Per Second

    The number of messages produced per second. While Bytes In Per Second displays the broker traffic in absolute terms of bytes, this is the number (count) of individual messages that flow in, regardless of their size.

    This metric is also displayed in the Topic tab and the Topic and Consumer group views. For any given data point, the graph displays, for that point in time:

    • Metrics tab: The total number of messages produced in the cluster across all topics and brokers.

    • Broker tab: The number of messages being written into the selected broker.

    • Topic tab: The number of messages flowing into the selected topic.

    • Topic view: The same as the Topic tab. Each Topic is represented by a trend line so you can see the number of messages flowing into each broker.

    • Consumer group view: Similar to the Topic page.

    Analysis:

    • Useful for end-users who are more interested in the message count rather than the network throughput.

    • When used in conjunction with Bytes in Per Second you can estimate the size of a single message.

    • Like Bytes in Per Second it can help identify broker imbalance so you can rebalance your partitions.

  • Total Fetch Requests Per Second

    The number of consumer fetch requests sent per second. Like Bytes Out Per Second this is an indicator of the rate consumers are uptaking/requesting messages from the Kafka cluster.

    This metric is also displayed in the Topic tab and the Topic view. For any given data point in the graph, for that point in time, it displays:

    • Metrics tab: The total number of fetch requests within the cluster across all topics and brokers.

    • Broker tab: The number of fetch requests received by the selected broker across all topics.

    • Topic tab: The number of fetch requests received for a particular topic across all brokers.

    • Topic view: The total number of fetch requests received for that topic. Each Topic is represented by a trend line so you can see the number of fetch requests flowing into each broker.

    Analysis:

    • Care must be taken when interpreting this metric because the value also includes replica traffic, that is, the fetch requests that get raised by the brokers to keep the topic partitions in sync.

  • Under Replicated Partitions Per Second

    The count of under replicated partitions existing in a cluster.

    • Metrics tab: The trend line for under replicated partitions for the entire cluster.

    • Broker tab: The trend line for under replicated partitions for the selected broker.

    Analysis:

    • This is the metric to monitor. It can provide insight into a number of problems on a Kafka cluster ranging from a broker being down to resource exhaustion.

      For example, a constant number of under replicated partitions for many brokers might indicate that one of the brokers is down. The count across the cluster is equal to the number of partitions on the faulty broker.

  • # Active Controller Trend

    The active controller trend line.

    • Metrics tab: The trend line indicates the presence of an active controller for the entire cluster.

    • Broker tab: The trend line indicates the duration for which the selected broker was an active controller.

    Analysis:

    • On the cluster level the value for this metric should always be one, that is, at all times only one broker should be the active controller. Any other value could indicate the cluster might be susceptible to administrative issues such as partition reassignment, etc.

    • On a broker level, except for the active controller, the brokers should return zero.

  • Request Handler Idle Ratio Average Per Minute

    The request handler idle ratio trend over a time period. It indicates the percentage of the time the request handlers aren't in use.

    • Metrics tab: The trend line is the metric value across the entire cluster averaged over all brokers.

    • Broker tab: The trend line is the metric value for the selected broker.

    Analysis:

    • Lower numbers indicate increased load on the broker. Anything under 0.2 (20 percent) is a cause for concern. Values under 0.1 (10 percent) usually indicate an active performance problem. Typically, there are two reasons for high thread utilization:

    • There aren't enough threads.

    • The threads are devoting some of their resources to overhead tasks and not servicing the client requests itself.

  • # Partition Count

    The number of partitions. The count includes both leader and follower replica. It should be basically constant over time.

    • Metrics tab: The trend line indicates the metric's value across the entire cluster averaged over all brokers.

    • Broker tab: The trend line indicates the metric's value for the selected broker.

    Analysis:

    • Expect this to be basically constant over time. This count includes both leader and follower replica.

  • # Leader Partition Count

    The number of partition leaders.

    • Metrics tab: The trend line indicates the metric's value across the entire cluster averaged over all brokers.

    • Broker tab: The trend line indicates the metric's value for the selected broker. It represents the number of partitions for which the broker is currently a leader.

    Analysis:

    • Consider monitoring this metric because it gives you a qualitative distinction between the replicas on a broker in regard to leaders and followers. Even if the replicas are balanced in size or number across the cluster, there might be a broker who has an indiscriminately large/small number of leaders. This is an indication of imbalance.

    • You can use this metric in conjunction with # Partition Count to calculate the percentage of partitions on the Broker for the partition the broker is a leader. In well-balanced clusters, this should be uniform. For example, if you have a replication factor of two all brokers should be leaders of approximately 50% of their partitions.

  • # Offline Partition Count

    The number of offline partitions in the cluster indicating the number of partitions in the cluster that have no leader.

    • Metrics tab: The trend line indicates the metric's value across the entire cluster averaged over all brokers.

    • Broker tab: The trend line indicates the metric's value for the selected broker.

    Analysis:

    • Along with Under Replicated Partitions, this is a critical metric to monitor. Such partitions are inaccessible to clients because produce and fetch requests are sent only to leaders. This could result in issues like producer clients losing messages.

      One potential reason for the presence of offline partitions is the broker/brokers hosting the leader replicas are down.

  • Fetch Total Time, 99th Percentile

    The 99th percentile of the total turnaround time (time from receiving the request to sending a response back) the broker spends processing the fetch request. 99% of all values in a group of fetch request timing are less than this metric's value.

    • Metrics tab: Shows the time value for the entire cluster averaged across all brokers.

    • Broker tab: Shows the time value for the selected broker.

    Analysis:

    • You can gain insight into how an average request performs and what are the outliers by correlating this metric value with the average value of the metric.

  • Produce Total Time, 99th Percentile

    The 99th percentile of the total turnaround time (time from receiving the request to sending a response back) the broker spends processing the produce request. 99% of all values in a group of fetch request timing are less than this metric's value.

    • Metrics tab: Shows the time value for the entire cluster averaged across all brokers.

    • Broker tab: Shows the time value for the selected broker.

    Analysis:

    • You can gain insight into how an average request performs and what are the outliers by correlating this metric value with the average value of the metric.

    • You can use this to set a baseline of sorts. Much like Under Replicated Partitions, a spike in the 99 percentile for Produce requests can alert you to a wide range of performance issues.

  • # Fetch Requests Per Sec

    Displays the one minute rate for all the fetch requests being received and processed. Like Bytes Out Per Sec, it is an indicator of the rate at which consumers are uptaking/requesting messages from the Kafka cluster.

    • Metrics tab: Shows the summation of the fetch requests being processed across all brokers per second.

    • Broker tab: Shows the fetch requests being processed by a specific broker per second.

    Analysis:

    • You must take care when interpreting this metric. In addition to the consumer client fetch request, the value also includes replica traffic, that is, the fetch requests that get raised by the brokers to keep the topic partitions in sync.

  • # Produce Requests Per Sec

    Displays the one minute rate of all the Produce Requests being received and processed. Like Bytes In Per Sec, it indicates how much traffic the brokers are receiving from the Producer clients.

    • Metrics tab: Shows the summation of the produce requests being processed across all brokers per second.

    • Broker tab: Shows the produce requests being processed by a specific broker per second.

    Analysis:

    • This metric can be useful in deciding whether to expand your cluster.

    • Additionally, it can be helpful when tracked at the broker level. On a normal Kafka cluster the value across all brokers should be basically the same. It helps you identify a broker that is receiving more traffic than others indicating there is a need for a partition rebalance.

  • Produce Purgatory Size

    The number of Produce requests sitting in the Producer Request Purgatory. It is a holding pen for requests waiting to be satisfied (Delayed). Of all Kafka request types, it is used only for Produce requests. It tracks the number of requests sitting in purgatory (including both watchers map and expiration queue).

    • Metrics tab: The count for the entire cluster averaged across all brokers.

    • Broker tab: The count for the selected broker.

    Analysis:

    • You can use this metric as a rough gauge of memory usage.

  • Fetch Purgatory Size

    The number of Fetch requests sitting in the Fetch Request Purgatory. It is a holding pen for requests waiting to be satisfied (Delayed). Of all Kafka request types, it is used only for Fetch requests. It tracks the number of requests sitting in purgatory (including both watchers map and expiration.

    • Metrics tab: The count for the entire cluster averaged across all brokers.

    • Broker tab: The count for the selected broker.

    Analysis:

    • You can use this metric as a rough gauge of memory usage.

  • Log Flush Latency, 99th Percentile

    The 99th percentile value of the latency incurred by a log flush, that is, write to disk in milliseconds. 99% of all values in the group are less than the value of the metric.

    • Metrics tab: The time value for the entire cluster averaged across all brokers

    • Broker tab: The time value for the selected broker.

    Analysis:

    • Log flushes are generally one of the most expensive operations and can affect the durability, latency, and throughput of the Kafka cluster. This trend line depicts what's the log latency like for the cluster and can help in configuring the log flush interval.

Metrics tab
ClusterView-MetricsTab.png
Broker tab
ClusterView-BrokerTab.png
TOPIC tab
ClusterView-TopicTab.png
Kafka topics list consumed by a consumer group (CG) with relevant KPIs.

Click the topic to bring up the Topic view and the consumer to bring up the Consumer group view. By default it is sorted on Topic name, this table is sorted on Consumer Groups in order to see the active consumer groups. The Consumer Groups status is clearly indicated below it. See Kafka Insights for a Use Case example of locating of lagging or stalled Consumer Groups.

ClusterView-TopicTable.png
Consumer group page
6-CGView-TopicDetail.png
Key Performance Indicators

  • Number of Topics

  • Number of Partitions

The Topic list displays the KPIs; when details are available a more info icon is displayed. Click it to bring up the Kafka view for the topic. Below the list are two tabs that display graphs of the Topic and Partition details. By default, the window opens with the Topic Detail graph displayed.

Partition Tab

You can choose both the Partition and the Metric for the display. By default, the 0th partition is displayed using the metric offset. The Partition Details' list is populated if the details are available.

541098741.png
Topic page

The Kafka View has two tabs, Topic Detail and Partition Detail. Each view has a Consumer Details' list which is populated if the details are available.

Kafka Topic Detail

By default, the Kafka Topic Detail opens in the Topic Detail view which graphs the KPIs. The KPIs in the title bar, present

541098728.png

Kafka Partition Detail

You can choose both the Partition and the Metric for the display. By default, the 0th partition will be displayed on using the metric offset.

541197137.png
Unravel insights for Kafka

Unravel provides auto-detection of lagging/stalled Consumer Groups. It lets you drill down into your cluster and determine which consumers, topics, partitions are lagging or stalled. See Kafka Insights for a use case example of drilling down into lagging or stalled Consumer Groups.

Unravel determines Consumer status by evaluating the consumer's behavior over a sliding window. For example, we use average lag trend for 10 intervals (of 5 minutes duration each), covering a 50-minute period. Consumer Status is evaluated on several factors during the window for each partition it is consuming.

For a topic partition Consumer status is:

  • Stalled: If the Consumer commit offset for the topic partition is not increasing and lag is greater than zero.

  • Lagging: If the Consumer lag for the topic partition is increasing consistently, and an increase in lag from the start of the window to the last value is greater than lag threshold (e.g., 250).

The information is distilled down into a status for each partition, and then into a single status for the consumer. A consumer is either in one of the following states:

  • OK: The consumer is working and is current.

  • Warning: The consumer is working, but falling behind.

  • Error: The consumer has stopped or stalled.

In this section: