What is a Kafka Topic? Definition, Examples & Uses

If you’ve ever wondered how millions of events like website clicks or payment transactions get organized in real time, the answer often starts with a Kafka topic. It’s a simple concept with powerful implications for building scalable, real-time data pipelines. In this article, you’ll learn exactly what a Kafka topic is, see concrete examples, and understand how partitions make it all work.

Typical retention: 7 days (configurable) ·
Default partitions per topic: 1 (configurable) ·
Max message size: 1 MB (default)

Kafka topics bundle several configuration properties. Here’s a glance at the key parameters that govern topic behavior.

What is a Kafka topic?

Kafka topic definition

A Kafka topic is a named, durable, append-only log that producers write events to and consumers read from. Think of it like a folder in a filesystem: events are the files, and the topic provides the organizational category.

• Each topic has a unique name across the entire cluster (Conduktor).
• Producers send records to topics; consumers subscribe and process those records independently (Confluent Developer).
• Topics are immutable – once written, records are never updated in place (Conduktor).

The implication: a topic acts as a single source of truth for a stream of events, enabling multiple consumers to replay history without interfering with each other.

Kafka topic as a log

At the storage level, a Kafka topic is a log – an ordered sequence of records. Each record is assigned an offset (a sequential ID) within its partition. This log structure is what gives Kafka its ability to replay events and preserve historical context.

• Records are appended, never modified (Apache Kafka introduction).
• Offsets uniquely identify messages within a partition (Apache Kafka introduction).
• The log is stored on disk and replicated across brokers for fault tolerance (Conduktor).

What is a Kafka topic example?

Example topic names

In practice, topics are named after the business event they represent. Common naming conventions use present-tense verb phrases like ‘order-placed’ or ‘user-clicks’.

• order-placed – stores every completed order event (Baeldung).
• user-clicks – captures website clickstream data.
• payment-transactions – records payment processing events.
• temperature-readings – logs IoT sensor measurements (Baeldung).

Real-world use cases

Topics are the backbone of event sourcing and stream processing. Companies use them to decouple microservices, process real-time analytics, and maintain an audit trail.

• An e-commerce site might stream all customer interactions into a ‘user-actions’ topic. Downstream services independently read that topic to send recommendation emails, update inventory, and produce dashboards (Redpanda guide).
• A financial institution uses a ‘fraud-alerts’ topic to process transactions in real time while preserving the full event history for regulatory audits (Confluent Developer).

The pattern: topics organize events into logical categories, allowing multiple independent consumers to work with the same data without tight coupling.

What is the purpose of a Kafka topic?

Data organization

Topics provide the primary categorization mechanism in Kafka. Without topics, events would be a single chaotic stream. Topics let you segment data by business domain or event type.

• Topics enable multiple consumers to read the same data independently (Confluent Developer).
• Consumers can be added or removed without affecting producers or other consumers – true decoupling (Redpanda guide).
• Topics retain events for a configurable time, enabling replay of historical data (Apache Kafka docs).

Stream processing

Kafka’s topic model is the foundation for stream processing applications like Apache Kafka Streams or ksqlDB. Topics serve as both input and output for processing pipelines.

• Streaming applications consume from one topic, process data, and produce results to another topic (Confluent Developer).
• This allows building real-time dashboards, alerting systems, and machine learning feature pipelines (Redpanda guide).

The trade-off: while topics enable rich processing, managing retention and compaction policies requires careful planning to avoid unbounded storage growth.

What is a Kafka topic partition?

Partition concept

A partition is the physical unit of Apache Kafka. While a topic is logical, a partition is the immutable log that lives on a broker.

• Each topic is divided into one or more partitions (Apache Kafka introduction).
• Partitions are ordered, immutable logs with sequential offsets (Apache Kafka introduction).
• Partitions are the unit of parallelism: producers and consumers can work on different partitions simultaneously (Confluent docs).

Within a partition, message order is guaranteed. Across partitions, no global order exists. This trade-off is central to Kafka’s scalability.

Partition count

Choosing the right partition count is a balancing act. More partitions increase parallelism but also add overhead.

• Within a consumer group, up to one consumer per partition can be active; extra consumers sit idle (Confluent docs).
• Writes to different partitions can happen in parallel, boosting throughput (Confluent docs).
• Recommended starting point: 3–10 partitions per broker, adjusted based on actual throughput (Confluent docs).

The implication: partition count is a design decision that directly impacts both throughput and ordering constraints. Start low and scale up with monitoring.

Kafka topic vs queue: What is the difference?

One of the most frequent questions is how Kafka topics compare to traditional message queues like RabbitMQ or Amazon SQS. The differences are fundamental and influence architectural choices.

Message delivery semantics

• Kafka topic: Multiple consumers can read the same message independently; each consumer maintains its own offset (Confluent Developer).
• Queue: A message is consumed by one consumer and then typically deleted – competing consumers pattern.

Persistence

• Kafka topic: Messages are retained for a configurable time (default 7 days) regardless of consumption. Old messages can be replayed (Apache Kafka docs).
• Queue: Messages are removed once acknowledged. No replay of historical messages.

Ordering guarantees

• Kafka topic: Order is preserved within a partition, not across partitions (Baeldung).
• Queue: Order may not be guaranteed, especially with multiple consumers.

The pattern: Kafka topics shine when you need multiple consumers, event replay, and ordered processing per key. Traditional queues are simpler for point-to-point messaging with once-and-only-once delivery.

What we know and what remains unclear

Confirmed facts

• Kafka topics are immutable logs – records are never updated in place (Conduktor).
• Topics are the primary data organization unit in Kafka (Redpanda).
• Each topic can have multiple partitions for parallel processing (Apache Kafka).

What’s unclear

• The exact partition count that optimizes performance depends on hardware, workload, and consumer behavior (Confluent).
• The maximum retention period is unbounded in theory, but cluster storage limits apply – no universal ceiling (Apache Kafka docs).

Expert perspectives

Two authoritative sources capture the essence of Kafka topics. The official Apache Kafka documentation uses a filesystem analogy to explain the concept:

“A topic is like a folder in a filesystem, events are files.”– Apache Kafka documentation (the original source)

The Confluent developer course offers a more functional definition:

“Topics are powerful, immutable logs that allow you to store, process, and replay streams of events.”– Confluent Developer course (the official training arm)

These quotes reinforce the two core attributes: organization and immutability.

The implication: whether you’re a DevOps engineer or a data architect, thinking of topics as folders with replay capability helps avoid common design mistakes like treating topics as transient queues.

Summary

A Kafka topic is more than just a category – it’s a durable, scalable log that decouples producers from consumers and enables replay. For engineers building event-driven architectures, the choice is clear: leverage Kafka topics as the backbone of your data pipeline, or risk building on outdated message queue patterns that lack replayability and independent consumer groups.

Related reading: How Word Finders Work · How to Disable Location Services

For a deeper dive into how events are organized, see this comprehensive guide on Apache Kafka topics that explains partitions and replication.

Frequently asked questions