Kafka 2.0 Queue Monitor Configuration Guide on the Elven Platform

The Kafka Monitor from the Elven Platform enables checks based on key queries in Kafka. This feature helps configure continuous checks, set alerts, and define thresholds for automatic incident creation, ensuring you are quickly informed about any irregularities in connectivity or service performance.

Apache Kafka is an open-source distributed streaming platform designed for real-time data stream processing and integration. It operates both as a pub/sub messaging system and a distributed log storage system, enabling the transport of large volumes of data with high resilience and low latency. Kafka organizes data into topics, which can be consumed by multiple clients simultaneously, ensuring horizontal scalability.

Its ability to handle millions of messages per second makes it ideal for applications such as real-time event analysis, data pipelines, system integrations, and microservices-based solutions.

Accessing Kafka 2.0 Monitoring

• Navigate to the main menu and click on Services Hub.

• In Queue, select the item Kafka 2.0.

Monitoring Configuration

Monitoring your service availability has never been easier. Start by giving a clear name to the resource you want to monitor (Resource Name) to make it easier to identify. Then, adjust the Interval between checks and the Timeout for receiving a response.

Select where the monitoring agent is located by choosing the appropriate Checkpoint Cloud. If no environment exists, you can create one by clicking + Checkpoint. After this setup, in Form Type, you can choose between User and Password or Healthcheck URL.

If you select User and Password, you must provide the server address in the Host, Port, Topic (the name of the Kafka topic you want to monitor or interact with), Consumer Group (the consumer group to be analyzed), and Lag Tolerance (the maximum delay tolerated before triggering an alert or identifying a system issue).

If you choose Healthcheck URL, simply enter the URL of the resource to be monitored. Remember that the Host and Healthcheck URL fields accept only URLs. If you need to use an IP address, it must be stored in a Secret to ensure security and organization of the information.

If authentication is required in Kafka, check the Authentication option and fill in the fields according to the security mechanism implemented by the cluster. First, enter the Username and corresponding Password. These values are provided by the Kafka administrator and are used to authenticate the client.

Next, in the Mechanism field, select the authentication mechanism used by the Kafka broker. If the cluster uses a simple method like PLAIN, only the username and password are required. For more secure configurations like SCRAM, Kafka performs authentication based on hashed passwords, offering greater protection. If the cluster uses OAuth, authentication is done using OAuth 2.0 tokens, allowing secure integration with identity providers. In the case of GSSAPI, the mechanism uses the Kerberos protocol for authentication, providing a robust and widely adopted solution in enterprise environments.

Additionally, check whether Kafka is configured to use TLS or SASL. If the cluster implements encryption to ensure secure communication, enable the TLS option. If Kafka uses an additional authentication layer such as SASL, also check the corresponding box.

Configuration with User and Password

Configuration with User and Password and Authentication

Configuration with Healthcheck URL

Automatic Incident Opening

You can configure automatic incident opening to ensure a quick response to critical issues. To begin, define the incident severity, allowing you to prioritize according to urgency. Next, adjust the Check Interval, specifying the check frequency in seconds to continuously monitor the resource. This helps ensure you're always one step ahead, detecting problems as soon as they arise.

Additionally, select the team to be notified whenever an incident occurs and enable the "Enable to set up automatic incidents opening" option to ensure the configuration is active. With this setup, the platform automates incident management, making the response process faster and more efficient, without the need for manual intervention. This ensures your team is always ready to resolve any issue with speed and precision.

Maintenance Window

We also have the Maintenance Window, an essential feature for managing planned maintenance periods in your application. During this interval, checks are temporarily paused, preventing monitoring, alerts, and notifications from being triggered while you perform updates or adjustments. This allows maintenance to proceed smoothly, without generating unnecessary notifications or false alarms, ensuring your operations continue in an orderly fashion without unexpected interruptions in performance reports.

For example, imagine you need to update the payment system of an e-commerce platform, making backend adjustments such as installing new security certificates. To do this, you can configure the Maintenance Window for a specific time, such as 12/13/2024, from 2:00 PM to 2:30 PM. During this period, the Elven Platform suspends checks, preventing the monitoring system from registering temporary failures or triggering false alerts. This way, you can make the necessary changes calmly, knowing that the monitoring system will not be impacted during maintenance.

This approach ensures that the update is carried out in an organized manner, without affecting the user experience or generating unwanted notifications.

Application Opening Hours

You can also rely on the Application Opening Hours feature, which allows you to configure your application's operating hours. This functionality is essential for customizing monitoring based on the periods when your application is actually active, avoiding alerts and notifications outside of business hours. This way, monitoring becomes more aligned with your business’s real needs, ensuring more accurate reports and efficient management.

For example, imagine your application operates only from Monday to Friday, between 9:00 AM and 6:00 PM. You can configure the Application Opening Hours to reflect this schedule by specifying the days and operating periods. With this setup, the Elven Platform automatically disables checks outside of these hours, preventing the logging of failures that don’t affect end users and avoiding unnecessary alerts.

This approach optimizes performance analysis, focusing only on relevant periods and providing a clearer view of your application's health during its operating hours.

Glossary of Technical Terms

Kafka: An open-source event streaming platform designed to process, store, and transmit large volumes of data in real time. With support for publish-subscribe patterns, distributed queues, and durable storage, it offers high scalability, low latency, and high availability. Ideal for applications requiring continuous data processing, such as system integration, real-time analytics, and monitoring. Compatible with various programming languages, it is widely used in microservices architectures and modern data pipelines.

Interval: Time interval between automatic checks performed during monitoring.

Timeout: Maximum time allowed for monitoring to receive a response from the monitored resource before registering a failure.

Checkpoint Cloud: Location where the monitoring agent is hosted, which can be a preexisting environment or one created by the user.

Host: URL address of the monitored resource. If an IP address is needed, it must be stored in a Secret for enhanced security.

Port: Access port for Kafka.

Consumer Group: A group of consumers that share the load of processing messages from one or more Kafka topics. Ensures that each partition is processed by only one consumer within the group.

Lag Tolerance: Acceptable limit of message consumption delay, measured by the number of unprocessed messages. Used to identify bottlenecks and maintain system performance.

Secret: Resource used to store sensitive information, such as IP addresses or credentials, ensuring security and organization.

Username: Unique identifier configured in the Kafka broker to authenticate a client’s access to the cluster.

Password: Credential associated with the username, used for authentication and authorization in Kafka.

Mechanism: Authentication method used by Kafka, selected in the form. Examples include:

• PLAIN: Simple method using username and password for authentication.

• SCRAM (Salted Challenge Response Authentication Mechanism): More secure mechanism using hashed passwords for authentication.

• OAUTHBEARER: Uses OAuth 2.0 tokens for authentication, allowing integration with identity providers.

• GSSAPI (Generic Security Service Application Program Interface): Kerberos-based authentication, widely used in enterprise environments.

TLS (Transport Layer Security): Encryption protocol that protects communication between the Kafka client and server, ensuring data confidentiality.

SASL (Simple Authentication and Security Layer): Authentication layer supporting various mechanisms (such as PLAIN, SCRAM, OAUTHBEARER, and GSSAPI) for secure authentication in Kafka.

Kafka Broker: A component of a Kafka cluster responsible for storing and distributing messages between producers and consumers.

Kafka Cluster: A group of Kafka brokers working together to manage and process messages in a distributed and scalable manner.

Enable to set up automatic incidents opening: Option that, when enabled, activates automatic incident creation upon detection of critical issues.

Severity: Level of criticality assigned to an incident, allowing it to be prioritized according to urgency.

Check Interval: Time interval, in seconds, for performing continuous checks on the monitored resource.

Maintenance Window: Feature that temporarily pauses monitoring, alerts, and notifications during planned maintenance periods.

Application Opening Hours: Configuration that defines the operating hours of the application, aligning monitoring with active periods and avoiding alerts outside those hours.