How-to · 11 Nov
How-to · 3 min read
How to get a Slack alert when Kafka lag gets too high
Your consumers fall behind on a Kafka topic, and nobody notices until a downstream team asks why their data is an hour late. You do not want a full observability stack just to watch one number. You want a message in Slack when partition lag crosses a line you choose, and another one when it recovers. And you want it without wiring up exporters, a metrics database and an alert manager first.
01
Why it's hard
Kafka itself does not alert anyone, so lag is something you have to go and look at. The standard command line tools answer the question for one moment, which means someone has to remember to run them. The usual answer is a metrics pipeline with an exporter, a time series store, dashboards and alert rules, which is a lot of moving parts for a small team. Alerting also gets noisy fast, because a lag spike that bounces around the threshold can post the same warning to a channel again and again. After a few repeats, people mute the channel and the real breach goes unseen.
This part describes the usual way. It makes no claims about kafkaesque, so nothing is quoted here.
02
How kafkaesque does it
Kafkaesque is an open source, read-only Kafka monitoring tool with a web UI, an API and a CLI, and it includes Slack lag alerts. You create named alerts with either an average or a maximum partition-lag threshold, and send them to a Slack Incoming Webhook or a Workflow Builder trigger. It posts one notification per breach rather than repeating, and can send a follow-up when the lag is resolved. It never produces messages or changes Kafka resources, so pointing it at a production cluster is low risk.
Named alerts with average or maximum partition-lag thresholds.Quoted from its GitHub page
Slack Incoming Webhook and Workflow Builder trigger support.Quoted from its GitHub page
One firing notification per breach and optional resolution notification.Quoted from its GitHub page
03
Step by step
- 1Run the local demo with make demo-up if you want to see it work before touching your own cluster.
- 2Prepare a PostgreSQL 18+ database, a Super Admin login and an encryption key for Slack webhook URLs.
- 3Start the Docker image with your Kafka bootstrap address and config mounted read-only.
- 4Create a Slack Incoming Webhook or a Workflow Builder trigger in your workspace.
- 5Add a named alert in Kafkaesque, choose average or maximum partition lag, and set the threshold.
- 6Turn on the resolution notification so the channel also hears when things are back to normal.
04
Tips
Choose between average and maximum on purpose. Maximum catches one stuck partition that an average would hide, while average is calmer when lag is spread evenly and short bursts are normal. If you are unsure, start with maximum and loosen the threshold if it fires too often.
General advice. Not a claim about the product.
Set the threshold from what normal looks like. Watch the lag views in the UI for a few days first, then put the line above the usual peaks so the alert means something when it fires. A threshold set too low trains people to ignore the channel, which defeats the point.
General advice. Not a claim about the product.
Generate the encryption key once and keep it stable. Kafkaesque uses it to protect the stored Slack webhook URLs, so changing it later can break the alerts you already set up.
A stable encryption key for Slack webhook URLs. Generate one with:Quoted from its GitHub page
Do not expect lag history from the database. PostgreSQL holds accounts, roles and alert subscriptions only, so if you need long-term lag charts, keep a separate metrics store for that.
not Kafka messages or historical metrics.Quoted from its GitHub page
05
Questions
Can it change anything in my Kafka cluster?
No. It is read-only and never produces messages or modifies Kafka resources.
Kafkaesque does not produce messages or modify Kafka resources.Quoted from its GitHub page
Does it work with secured Kafka clusters?
Yes. It supports TLS and mutual TLS, plus SASL PLAIN, SCRAM-SHA-256 and SCRAM-SHA-512.
SASL PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 authentication.Quoted from its GitHub page
What do I need to run it?
A single Go binary or the Docker image, plus a PostgreSQL 18+ database for accounts and alerts. There is no Node.js runtime.
One Go binary with embedded UI assets and no Node.js runtime.Quoted from its GitHub page