Explained with a diagram · Data engineering

Kappa architecture: one stream-processing path, explained with a diagram

In short

Kappa architecture is a data-processing architecture in which every event is appended to an immutable log and all processing, live and historical, runs through a single stream-processing path. There is no separate batch layer: to recompute results you replay the log through a new version of the same streaming job and switch readers to its output. Jay Kreps proposed it in 2014 as a simpler alternative to the Lambda architecture, which implements the same logic twice, once in a batch system and once in a stream system.

Updated · by the ChartCraft team · one of the patterns and models explained with a diagram

What does a Kappa architecture look like as a diagram?

Events flow from top to bottom: sources append to a log, a stream-processing job reads the log and writes an output table, and applications query that table. The only extra part is the reprocessing path, a second copy of the job that replays the log from the beginning into a new table.

Kappa architecture diagram: event sources append to an immutable log, a stream job writes an output table, and a second job replays the log into a new tableStream processing (one code base)Serving layerEvent sources (apps,devices, databases)Immutable event log (long retention)Stream job v1 (live)Stream job v2 (replaysfrom the start)Output table v1Output table v2Applications andqueriesappend eventslive streamreplay historywriteswritesread todayswitch when caught up
Kappa architecture: one log, one processing framework. Job v2 exists only while reprocessing; the dashed arrow is the cut-over. Rendered by ChartCraft's layout engine.

Mermaid code for this diagram

flowchart TD
    subgraph stream["Stream processing (one code base)"]
        jobV1["Stream job v1 (live)"]
        jobV2["Stream job v2 (replays from the start)"]
    end
    subgraph serving["Serving layer"]
        tableV1[("Output table v1")]
        tableV2[("Output table v2")]
    end
    sources["Event sources (apps, devices, databases)"]
    log[("Immutable event log (long retention)")]
    app["Applications and queries"]
    sources -->|"append events"| log
    log -->|"live stream"| jobV1
    log -->|"replay history"| jobV2
    jobV1 -->|"writes"| tableV1
    jobV2 -->|"writes"| tableV2
    tableV1 -->|"read today"| app
    tableV2 -.->|"switch when caught up"| app

Paste it into any Mermaid renderer, or check and export it with the Mermaid syntax checker and Mermaid to PNG.

What are the components of a Kappa architecture?

A Kappa architecture has an append-only log as its canonical data store, a stream-processing system that computes results from it, and a serving layer that applications query. Batch processing is replaced by replaying the log.

Part of the diagramWhat it isWhat it does
Event sources (apps, devices, databases)Anything that produces eventsAppends each event to the log once. Events are immutable: state changes only when a new event is appended.
Immutable event log (long retention)A distributed, ordered, append-only logKeeps the full history you want to be able to reprocess and allows several subscribers. Apache Kafka is the usual example, but any system with long retention of ordered data works.
Stream job v1 (live)The current stream-processing jobConsumes the live stream and continuously updates the output table.
Stream job v2 (replays from the start)A second instance of the job with new codeStarted only when the logic changes. It reads from the beginning of the retained log, with higher parallelism, and writes to a new table.
Output table v1The serving table in useHolds the results applications read today. It is deleted after the switch.
Output table v2The serving table being rebuiltFills up as job v2 replays history, and takes over once that job has caught up.
Applications and queriesReaders of the resultsQuery the serving layer and are redirected from the old table to the new one at cut-over.

How does reprocessing work in a Kappa architecture?

You reprocess by replaying the log through a new version of the same streaming job, and only when the processing code changes. Jay Kreps described it as four steps.

  1. Use Kafka, or another system that retains the full log and allows multiple subscribers. To be able to reprocess 30 days of data, set retention to 30 days.
  2. Start a second instance of the stream-processing job that reads from the beginning of the retained data and writes to a new output table.
  3. When the second job has caught up, switch the application to read from the new table.
  4. Stop the old version of the job and delete the old output table.

Keeping both tables for a short time has a benefit: you can revert instantly by pointing the application back at the old table.

What is the difference between Kappa and Lambda architecture?

Lambda architecture feeds the same data into a batch system and a stream system in parallel and merges their results at query time; Kappa architecture sends all data through one stream-processing path. Kappa has the same goals but only one code base to write, test and operate.

Lambda architectureKappa architecture
Processing pathsTwo: a batch layer and a speed (stream) layerOne: stream processing
Transformation logicImplemented twice, once per systemImplemented once
Recomputing historyThe batch layer recomputes continuouslyReplay the log through a new job, only when the code changes
Query timeStitches batch and real-time results togetherReads one output table
Main costRunning, debugging and keeping two systems in syncTemporarily double storage in the output database, which must also take high-volume writes during the reload

When should you use a Kappa architecture?

Use it when you need low-latency results from event data and want to be able to recompute them without maintaining a second, batch implementation. Its real advantage is simplicity: people develop, test, debug and operate one processing framework.

Use it when

Skip it when

What are common mistakes with a Kappa architecture?

The mistakes are almost all about replay: not being able to do it, or doing it in a way that cannot be undone.

How do you draw a Kappa architecture diagram?

Draw a single pipeline of sources, log, stream job, output table and applications, then add the reprocessing path as a second job and table fed from the same log, with a dashed arrow for the cut-over.

Opens the studio with your text · or start from the Architecture diagram generator

What do real data architecture diagrams look like?

Live diagrams from the public ChartCraft gallery, each generated from a plain-English description. Open one to see the prompt that produced it.

Questions

What is Kappa architecture?

Kappa architecture is a data-processing architecture in which all data flows through a single stream-processing path. Events are stored in an immutable, append-only log, results are computed from that log by a streaming job, and history is recomputed by replaying the log rather than by a separate batch system.

Who invented the Kappa architecture?

Jay Kreps described it in the article Questioning the Lambda Architecture, published on 2 July 2014, drawing on his work building real-time data infrastructure with Kafka and Samza at LinkedIn. The name Kappa is his own suggestion in that article.

What is the difference between Kappa and Lambda architecture?

Lambda architecture runs a batch layer and a stream layer side by side and implements the processing logic in both. Kappa architecture removes the batch layer: the logic exists once, in the stream-processing job, and historical data is reprocessed by replaying the log.

How do you reprocess data in a Kappa architecture?

Start a second instance of the streaming job that reads the log from the beginning and writes to a new output table. When it has caught up, switch the application to the new table, then stop the old job and delete the old table.

Does Kappa architecture require Kafka?

No. Kafka is the common choice because it retains ordered data for a long time and supports many consumers, but nothing ties the idea to Kafka. Any system that supports long retention of ordered data can serve as the log.

More patterns and models explained with a diagram

Describe the system in plain English and get the diagram.

Architecture diagram generator →