Integration

Apache Kafka

Consumer group lag, offsets, partitions and replicas in sync, and the broker log.

Category: Message brokers
How data is sent: Logthread Agent
Runs on: Linux
Status: Beta

How it works#

Answer a few questions. They only fill in the setup steps. Nothing on this page connects to your servers: Logthread never reaches into your network.
Run one command on the Apache Kafka machine. The next page shows the command with this connector's ingest key filled in. It downloads the Logthread Agent from downloads.logthread.com and installs it as a service (Linux, as root with sudo). No other download.
Data arrives. Everything is collected inside your environment. It is sent over outbound HTTPS (port 443) to ingest.logthread.com. Nothing is opened to the internet: no inbound ports, VPN or public access to your servers. The connection page shows what has arrived and turns Active with the first record.

What you answer#

Question	Answer	Notes
This host reads the cluster metrics	Yes or no	Turn it on for one host per cluster, and off on the others: they send their logs only, so nothing is counted twice. Default: `Yes`.
Bootstrap broker (optional)	Text	host:port; default localhost:9092.
Cluster name (optional)	Text	Shown on the dashboard; default kafka.
Sign in	`none` (No SASL), `SCRAM-SHA-512` (SCRAM-SHA-512), `SCRAM-SHA-256` (SCRAM-SHA-256), `PLAIN` (PLAIN)	With SASL the command asks for the password; TLS is used. Default: `none`.
SASL user (optional)	Text	Default: logthread_agent.
Log file (optional)	Text	Default: /opt/kafka/logs/server.log.
Labels (optional)	Text	For Fleet and filtering, for example env=prod.
Manage this agent from Fleet	Yes or no	Change its configuration from Logthread later without logging in to the host. Default: `Yes`.

Setup#

These are the steps for the default answers. In the app, they follow your answers, and the connector's ingest key replaces YOUR_INGEST_KEY.

Step 1. Install the Logthread Agent. On a Kafka broker: it installs the agent with the Kafka and Linux host profiles. It reads brokers, topics, partitions and consumer group offsets every 30 seconds (no message contents), and the broker log.

curl -sSL https://downloads.logthread.com/agent/install.sh | sudo sh -s -- \
  --key YOUR_INGEST_KEY --profiles linux-host,kafka --fleet
sudo tee -a /etc/logthread-agent/agent.env >/dev/null <<'EOF'
LOGTHREAD_KAFKA_BROKERS=localhost:9092
EOF
sudo systemctl restart logthread-agent

Step 2. Wait for data. Metrics arrive within a minute; log lines as the broker writes them. Stack traces stay with their error line.

Check that it works#

The connection page under Integrations counts what has arrived and turns Active with the first record. In Explore, search service:kafka.

Then add#

Each is one click on the connection page and opens with your data, so you can check it before saving.

Dashboard: Kafka: Consumer group lag, committed and newest offsets, partitions, replicas in sync and broker log warnings and errors.
Alert: Kafka broker errors: service:kafka severity:(error|fatal), at least 5 in 10 minutes

Set it up under Integrations in the app, or from an AI agent with MCP (add_integration). All integrations: Integrations guide. Next: find what it sends in Explore and live tail.