Spark stays in charge
Komiter does not replace the checkpoint. Spark keeps full control of its offsets; a failed commit never affects the query.
Komiter
Your Spark streaming jobs, leaving a trail Kafka can follow.
Spark Structured Streaming never tells Kafka where it got to. Komiter does — so consumer lag shows up in every tool your team already runs.
- 1 listener
- 1 option
- 0 changes to your query
The problem
Your streaming job is invisible to Kafka.
Spark keeps its offsets in its own checkpoint. There is no consumer group, no lag, and no alert when the job falls behind — until someone notices the data is hours old.
Without Komiter
kafka-consumer-groupshas nothing to describe- The job never appears in Kafka UI
- Lag alerting needs a separate metrics pipeline
- A bespoke exporter for every job
With Komiter
- Full per-partition lag, from the CLI you already use
- Shows up as a consumer group in Kafka UI and Redpanda Console
- Burrow, Cruise Control and Grafana alerts work out of the box
- Spark's fault tolerance, unchanged
See it
Real lag, in the tools you already have.
Captured from a live run against a real broker. Nothing mocked up.
$ kafka-consumer-groups --bootstrap-server localhost:9092 \
--describe --group komiter-demo
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
komiter-demo orders 0 220241 257046 36805
komiter-demo orders 1 221664 260848 39184
komiter-demo orders 2 221786 258621 36835
komiter-demo orders 3 216392 255768 39376
komiter-demo orders 4 220246 257984 37738
komiter-demo orders 5 223661 259733 36072
Get started
Three lines. That is the whole integration.
-
1
Add the dependency
-
2
Register the listener in your Spark config
spark.sql.streaming.streamingQueryListeners=com.komiter.Komiter
-
3
Name the consumer group on your Kafka source
spark.readStream .format("kafka") .option("kafka.bootstrap.servers", "localhost:9092") .option("subscribe", "orders") .option("kafka.consumer.commit.groupid", "my-group") // the only new line .load()
Sources without that option are left alone, so you can adopt it one query at a time.
How it works
A copy of the progress, written where Kafka looks.
Spark micro-batchonQueryProgress
Komiterend offsets per source
__consumer_offsetsKafka's own topic
Your lag toolsCLI · UI · Burrow · Grafana
Nothing new to run
It writes into the same place any consumer would, so existing tooling picks it up with no configuration.
Many topics, many sources
Every source commits to its own group, even when one source spans several topics and others sit idle.
Built for your exact Spark
Scala, the Kafka client and the Java target are copied from Spark's own pom for each release. Nothing to collide with.
24 Spark releases supported · 3.5.0 to 4.3.0 · Scala 2.12 & 2.13 · Apache 2.0