Kafka Queue Monitor Configuration Guide on the Elven Platform

The Kafka Monitor from the Elven Platform enables checks based on key queries in Kafka. This feature helps configure continuous checks, set alerts, and define thresholds for automatic incident creation, ensuring you are promptly informed of any irregularities in connectivity or service performance.

Apache Kafka is an open-source distributed streaming platform designed for real-time data stream processing and integration. It operates both as a pub/sub messaging system and a distributed log storage system, allowing the transport of large volumes of data with high resilience and low latency. Kafka organizes data into topics, which can be consumed by multiple clients simultaneously, ensuring horizontal scalability. Its ability to handle millions of messages per second makes it ideal for applications such as real-time event analysis, data pipelines, system integrations, and microservices-based solutions.

Accessing Kafka Monitoring

• Navigate to the main menu and click on Services Hub.

• In Queue, select the Kafka item.

Monitor Configuration

Monitoring the availability of your services has never been easier. Start by giving a clear name to the resource you want to monitor (Resource Name) to make identification easier. Then, adjust the interval between checks (Interval) and the response timeout (Timeout).

Select where the monitoring agent is located (Checkpoint Cloud) by choosing the Environment. If it doesn’t exist, you can create one using + Checkpoint. After this setup, in Form Type, you can choose between User and Password or Healthcheck URL.

In User and Password, you must provide the server address in the Host field, the Port, and the Topic (the name of the Kafka topic you want to monitor or interact with). In Healthcheck URL, you can simply enter the URL of the resource to be monitored.

Remember that the Host and Healthcheck URL fields only accept URLs; if you need to use an IP address, it must be created as a secret to ensure the security and organization of the information.

If authentication is required in Kafka, check the Authentication option and fill in the fields according to the security mechanism implemented by the cluster. The first step is to enter the username in the Username field and the corresponding password in the Password field. These values are provided by the Kafka administrator and are used to authenticate the client. Then, in the Mechanism field, select the authentication mechanism used by the Kafka broker.

If the cluster uses a simple method like PLAIN, you only need to provide the username and password. For more secure configurations like SCRAM, Kafka performs authentication based on hashed passwords, offering greater protection. If the cluster uses OAuth, authentication is done using OAuth 2.0 tokens, allowing secure integration with identity providers. In the case of GSSAPI, the mechanism uses the Kerberos protocol for authentication, providing a robust solution widely used in enterprise environments.

Additionally, it’s important to check whether Kafka is configured to use TLS or SASL. If the cluster implements encryption to ensure secure communication, check the TLS option to enable this protection layer. If Kafka uses an additional authentication layer like SASL, also check the corresponding box.

Configuration with User and Password

Configuration with User and Password and Authentication

Configuration with Healthcheck URL

Automatic Incident Opening

You can configure automatic incident opening to ensure a quick response to critical issues. To begin, define the incident severity, allowing you to prioritize according to urgency. Next, adjust the Check Interval, specifying the check frequency in seconds to continuously monitor the resource. This helps ensure you're always one step ahead, detecting problems as soon as they arise.

Additionally, select the team to be notified whenever an incident occurs and enable the "Enable to set up automatic incidents opening" option to ensure the configuration is active. With this setup, the platform automates incident management, making the response process faster and more efficient, without the need for manual intervention. This ensures your team is always ready to resolve any issue with speed and precision.

Maintenance Window

Maintenance Window is an essential feature for managing planned maintenance periods in your application. During this time frame, checks are temporarily paused, preventing monitoring, alerts, and notifications from being triggered while you perform adjustments or updates. This allows maintenance to proceed smoothly, without generating unnecessary notifications or false alarms, ensuring your operations continue in an orderly manner without unexpected interruptions in performance reports.

For example, imagine you need to update the payment system of an e-commerce platform, making backend adjustments such as installing new security certificates. To do this, you can configure a Maintenance Window for a specific time, such as 12/13/2024, from 2:00 PM to 2:30 PM. During this period, the Elven Platform suspends checks, preventing the monitoring system from logging temporary failures or triggering false alerts. This way, you can make the necessary changes calmly, knowing that the monitoring system will not be affected during maintenance.

This approach ensures that updates are carried out in an organized manner, without affecting the user experience or generating unwanted notifications.

Application Opening Hours

You can also rely on the Application Opening Hours feature, which allows you to configure your application's operating hours. This functionality is essential for customizing monitoring based on the periods when your application is actually active, avoiding alerts and notifications outside of business hours. This makes monitoring more aligned with your business's real needs, ensuring more accurate reports and efficient management.

For example, imagine your application operates only from Monday to Friday, between 9:00 AM and 6:00 PM. You can configure Application Opening Hours to reflect this schedule by specifying the days and operating periods. With this setup, the Elven Platform automatically disables checks outside these hours, preventing the logging of failures that don’t affect end users and avoiding unnecessary alerts.

This approach optimizes performance analysis, focusing only on relevant periods and providing a clearer view of your application's health during its operational hours.

Glossary of Technical Terms

Kafka: An open-source event streaming platform designed to process, store, and transmit large volumes of data in real time. With support for publish-subscribe patterns, distributed queues, and durable storage, it offers high scalability, low latency, and high availability. Ideal for applications requiring continuous data processing, such as system integration, real-time analytics, and monitoring. Compatible with various programming languages, it is widely used in microservices architectures and modern data pipelines.

Interval: The time interval between automatic checks performed during monitoring.

Timeout: The maximum time allowed for the monitoring system to receive a response from the monitored resource before registering a failure.

Checkpoint Cloud: The location of the monitoring agent, which can be a preexisting environment or one created by the user.

Host: The URL address of the monitored resource. If an IP address is required, it must be stored in a Secret for enhanced security.

Port: The access port for Kafka.

Secret: A resource used to store sensitive information, such as IP addresses or credentials, ensuring security and organization.

Username: A unique identifier configured in the Kafka broker to authenticate a client's access to the cluster.

Password: The credential associated with the username, used for authentication and authorization in Kafka.

Mechanism: The authentication method used by Kafka, selected in the form. Examples include:

• PLAIN: A simple method using username and password for authentication.

• SCRAM (Salted Challenge Response Authentication Mechanism): A more secure mechanism using hashed passwords for authentication.

• OAUTHBEARER: Uses OAuth 2.0 tokens for authentication, allowing integration with identity providers.

• GSSAPI (Generic Security Service Application Program Interface): Kerberos-based authentication, widely used in enterprise environments.

TLS (Transport Layer Security): An encryption protocol that secures communication between the Kafka client and server, ensuring data confidentiality.

SASL (Simple Authentication and Security Layer): An authentication layer supporting various mechanisms (such as PLAIN, SCRAM, OAUTHBEARER, and GSSAPI) for secure authentication in Kafka.

Kafka Broker: A component of a Kafka cluster responsible for storing and distributing messages between producers and consumers.

Kafka Cluster: A group of Kafka brokers working together to manage and process messages in a distributed and scalable manner.

Enable to set up automatic incidents opening: An option that, when enabled, activates automatic incident creation upon detection of critical issues.

Severity: The level of criticality assigned to an incident, allowing it to be prioritized based on urgency.

Check Interval: The time interval, in seconds, for performing continuous checks on the monitored resource.

Maintenance Window: A feature that temporarily pauses monitoring, alerts, and notifications during planned maintenance periods.

Application Opening Hours: A configuration that defines the operating hours of the application, aligning monitoring with active periods and avoiding alerts outside those hours.