Apache Kafka[1] has moved from being "the messaging system for big enterprises" to being the event backbone of modern architectures. It has crossed maturity thresholds, including the complete ZooKeeper exit via KRaft[2], that simplify operations and unlock new usage patterns.

Key takeaways

  • KRaft (Kafka Raft) integrates consensus inside Kafka and removes the ZooKeeper dependency: one system to operate, faster startup, and support for millions of partitions per cluster.

  • The three most consolidated patterns are CDC with Debezium, event sourcing, and stream processing with Kafka Streams, ksqlDB, or Flink.

  • Redpanda (C++, no JVM or ZooKeeper) and Pulsar (separated storage/compute) are real alternatives with different use cases.

  • Schema evolution, exactly-once cross-topic, and consumer rebalancing remain the most problematic areas.

  • For new projects the question isn’t "Kafka or not?" but "Kafka, Pulsar, Redpanda, or managed cloud?".

KRaft: goodbye ZooKeeper

For over a decade, Kafka depended on ZooKeeper[3] for cluster coordination, metadata management, and leader election. This meant two clusters to operate, two failure modes to understand, and two backup systems.

KRaft (Kafka Raft) integrates consensus inside Kafka. The advantages:

  • One system to operate. A Kafka cluster is, finally, a Kafka cluster.

  • Faster startup. No ZooKeeper synchronisation at boot.

  • Scalable metadata. Kafka in KRaft mode supports millions of partitions per cluster, vs ~200k with ZooKeeper.

  • Smaller footprint. One fewer role means less memory, fewer nodes, and less configuration.

In Kafka 3.5 (June), KRaft is GA for new clusters. For existing ZooKeeper clusters, in-place migration is in beta: not yet recommended for critical production, but the direction is clear.

Mature usage patterns

Kafka is used well in distinct usage patterns. The most consolidated are:

Change Data Capture (CDC)

Capture database changes and publish them as events. Debezium[4] is the de facto standard, with connectors for PostgreSQL, MySQL, MongoDB, Oracle, and SQL Server.

A typical pattern: replicate monolith tables to new services (dual-write pattern), enabling incremental migration without changing the monolith. The database remains source of truth; Kafka distributes its changes.

Event sourcing

Store the complete change history as immutable events. Any consumer can replicate state by replaying events. Powerful pattern but demanding: requires discipline in event design and schema evolution handling.

Works well in domains with mandatory traceability, such as financial audits or compliance. Less suited for general CRUD where the cognitive cost isn’t recovered.

Stream processing

Continuous processing over flows with Kafka Streams[5], ksqlDB[6], or Apache Flink[7]. Typical use cases: real-time fraud detection, enriching events with reference data, continuous aggregations for dashboards.

Flink wins with complex state or sophisticated time windows. Kafka Streams fits better when the pipeline lives "inside" Kafka and you don’t want an additional cluster.

CDC pipeline architecture diagram with Debezium, Kafka, and multiple microservice consumersCDC pipeline architecture diagram with Debezium, Kafka, and multiple microservice consumers (Image: Apache Software Foundation, Apache License 2.0, via Wikimedia Commons)

Competing alternatives

Three projects worth knowing:

  • Apache Pulsar[8]: separated storage/compute architecture, native multi-tenancy, geo-replication. Wins on large-scale operations, but smaller ecosystem.

  • Redpanda[9]: C++ rewrite of the Kafka protocol, no JVM or ZooKeeper. Claims ~10x lower latency with ~1/6 the hardware. Compatible with existing Kafka clients.

  • Confluent Cloud[10] / AWS MSK[11]: managed Kafka for those willing to pay for not operating.

Choice depends on context: for greenfield with latency requirements, Redpanda is attractive. For mature ecosystem and broad tooling, Apache Kafka still wins. For teams without infrastructure operational culture, managed cloud.

What’s still hard

Four areas Kafka still doesn’t handle cleanly:

  • Schema evolution. With Avro + Schema Registry it works, but incompatible changes still need careful human coordination.

  • Exactly-once semantics cross-topic. Transactional producers work, but at a cost in performance. The default choice is at-least-once + consumer idempotency.

  • Consumer rebalancing. The more partitions and dynamic consumers a topic has, the longer rebalances take: they can run from seconds to tens of seconds.

  • Fine-grained retention. Retaining data per-tenant or per-event is complex with Kafka’s retention policies.

Architecture comparison: Kafka (monolithic), Pulsar (separated storage/compute), and Redpanda (native C++)Architecture comparison: Kafka (monolithic), Pulsar (separated storage/compute), and Redpanda (native C++) (Image: ™/®Apache Software Foundation, Apache License 2.0, via Wikimedia Commons)

Also see the RabbitMQ vs Kafka analysis: deciding which broker fits each case remains one of the most important architectural decisions. For the observability layer over Kafka, OpenTelemetry and the Grafana stack are the natural complements.

Conclusion

Kafka is mature infrastructure for enterprise-scale event streaming. With KRaft it’s operationally simpler; with the stream-processing ecosystem consolidated, usage patterns are well documented.

For new projects, the question is no longer "Kafka or not?" but "Kafka, Pulsar, Redpanda, or managed cloud?". Each has its moment.

Spanish version: Kafka en 2023: streaming de eventos en la empresa.

Sources:

  1. Apache Kafka official documentation: KRaft[2]
  2. Debezium: Change Data Capture platform[4]
  3. Apache Pulsar official documentation[8]
  4. Redpanda product documentation[9]

Frequently asked questions

Can I migrate an existing Kafka cluster from ZooKeeper to KRaft in production?

Carefully: in Kafka 3.5 KRaft is GA only for new clusters, and in-place migration from ZooKeeper is in beta, not yet recommended for critical production. The direction is clear and the gains are tangible: one system to operate, faster startup without ZooKeeper synchronisation, a smaller footprint, and support for millions of partitions per cluster versus roughly 200k with ZooKeeper.

Does Kafka guarantee exactly-once semantics across topics?

Transactional producers make it possible, but at a cost in performance, so the default choice is at-least-once plus consumer idempotency. It is one of four areas that remain hard. Another is schema evolution: Avro plus Schema Registry works, but incompatible changes need careful human coordination. The remaining two are consumer rebalancing, which can take tens of seconds as partition counts grow, and fine-grained per-tenant or per-event retention.

Flink wins with complex state or sophisticated time windows. Kafka Streams fits better when the pipeline lives inside Kafka and you don't want to run an additional cluster; ksqlDB is the ecosystem's third option. Typical use cases are real-time fraud detection, enriching events with reference data, and continuous aggregations for dashboards.

Sources

  1. Apache Kafka
  2. KRaft
  3. ZooKeeper
  4. Debezium
  5. Kafka Streams
  6. ksqlDB
  7. Apache Flink
  8. Apache Pulsar
  9. Redpanda
  10. Confluent Cloud
  11. AWS MSK