Change Data Capture: JSON Events with Debezium & Kafka

Turning a database into a stream of JSON events

Change Data Capture (CDC) reads a database's internal transaction log — the same log used for replication and crash recovery — and turns every insert, update, and delete into a JSON event on a message stream. Instead of polling "what changed since I last checked?", downstream systems (search indexes, caches, data warehouses, other microservices) simply subscribe to a Kafka topic and react to each change as JSON, in near real time, with no extra load on the source database beyond reading a log it already maintains.

Debezium is the dominant open-source CDC platform, and its event shape has become a de facto standard that most modern data pipelines are built around.

The anatomy of a Debezium change event

Every CDC event carries the before state, the after state, and metadata about the source and the operation:

{
  "before": {
    "id": 1042,
    "status": "pending",
    "total": 149.97
  },
  "after": {
    "id": 1042,
    "status": "shipped",
    "total": 149.97
  },
  "source": {
    "connector": "postgresql",
    "db": "orders_db",
    "table": "orders",
    "ts_ms": 1751800000000,
    "lsn": 289471200
  },
  "op": "u",
  "ts_ms": 1751800000123
}

The before/after pair is what makes CDC events genuinely useful beyond "something changed" — a consumer can see *exactly* which fields changed by diffing the two objects, without a separate query back to the source database.

Why op matters: three very different events, one shape

The same JSON structure represents inserts, updates, and deletes — only before, after, and op change:

// Insert (op: "c") — before is null
{ "before": null, "after": { "id": 1043, "status": "pending" }, "op": "c" }

// Delete (op: "d") — after is null
{ "before": { "id": 1043, "status": "pending" }, "after": null, "op": "d" }

A naive consumer that only looks at after will crash on a delete event, since after is null — always branch on op first.

Why CDC beats polling

Polling ("run this query every 30 seconds and diff the results") has three real costs: it adds recurring load on the database for data that usually hasn't changed, it introduces up-to-30-second staleness by design, and it requires the source table to have a reliable "updated_at" column to poll against — which not every table has. CDC instead taps the transaction log the database is *already writing* for its own durability guarantees, so capturing changes adds negligible extra load and delivers events within milliseconds of the commit.

A consumer processing the stream

// Node.js — KafkaJS consumer
import { Kafka } from "kafkajs";

const kafka = new Kafka({ brokers: ["kafka:9092"] });
const consumer = kafka.consumer({ groupId: "search-indexer" });

await consumer.subscribe({ topic: "orders_db.public.orders" });
await consumer.run({
  eachMessage: async ({ message }) => {
    const event = JSON.parse(message.value.toString());
    if (event.op === "d") {
      await searchIndex.delete(event.before.id);
    } else {
      await searchIndex.upsert(event.after); // covers both create and update
    }
  },
});

Common real-world uses

• Keeping a search index in sync — Elasticsearch or Algolia updated the moment a product row changes, with no batch job or cron delay.
• Cache invalidation — evict or refresh a Redis-cached record the instant its source row changes, instead of relying on a TTL alone.
• Feeding a data warehouse — stream operational database changes into Snowflake or BigQuery for analytics without a nightly batch ETL job.
• Microservice data replication — one service owns a table; other services subscribe to its CDC stream to maintain their own read-optimized copy, avoiding a synchronous cross-service call on every read.
• Audit logging — the before/after pair is a ready-made audit trail of exactly what changed and when.

Gotchas worth knowing

• Schema changes ripple through. Adding a column to the source table changes the shape of every future event — consumers need to tolerate new, unexpected fields gracefully (don't use strict/additionalProperties: false schemas for CDC consumers).
• The initial snapshot is a special case. When Debezium first connects to an existing table, it emits a read (op: "r") event per existing row before switching to live c/u/d events — consumers need to handle the snapshot phase distinctly if ordering matters.
• Numeric precision. Database-specific numeric types (like PostgreSQL's numeric) sometimes serialize as a string or a special encoded structure in Debezium's default JSON converter rather than a plain JSON number, to avoid floating-point precision loss — check your connector's numeric handling mode.