The problem: producers and consumers drift apart
In a Kafka pipeline, producers write messages and consumers read them — often deployed independently, by different teams, at different times. The moment a producer changes the shape of its messages, every consumer that expected the old shape can break. A Schema Registry is the centralized service that prevents this: it stores a versioned history of the schemas used on each topic and *refuses* a schema change that would break existing readers. It's the governance layer that makes event streaming safe to evolve.
Schema Registry works with three serialization formats — Avro (the classic choice), JSON Schema, and Protobuf. The concepts below apply to all three; the details differ per format.
How it works on the wire
The clever part is that the full schema is not sent with every message — that would be enormous overhead. Instead:
- A producer registers its schema with the registry, which returns a unique schema ID.
- Each serialized message carries a tiny prefix: a magic byte, then the 4-byte schema ID, then the payload (Avro binary, or JSON).
- The consumer reads the schema ID from the prefix, fetches that exact schema from the registry (and caches it), and uses it to deserialize.
message on the wire:
┌────────┬──────────────┬─────────────────────────┐
│ 0x00 │ schema ID (4)│ serialized payload … │
│ magic │ e.g. 42 │ Avro binary / JSON │
└────────┴──────────────┴─────────────────────────┘So a 4-byte ID replaces a kilobyte of schema on every message, and the registry is the single source of truth for what ID 42 actually means.
Compatibility modes: the safety rules
The registry's real value is that it checks a new schema version against the old one before allowing it, according to a configured compatibility mode. The important ones:
- BACKWARD (the default) — a new schema can read data written with the old schema. This lets you upgrade *consumers first*, then producers. It's the default because it lets you replay a topic from the beginning with new consumer code.
- FORWARD — an old schema can read data written with the new schema. This lets you upgrade *producers first*; old consumers keep working against new messages.
- FULL — both backward and forward: new can read old *and* old can read new. The safest and most restrictive.
- NONE — no checks (don't).
Each also has a TRANSITIVE variant (BACKWARD_TRANSITIVE, etc.) that checks against *all* previous versions, not just the immediately preceding one.
What each mode lets you change
The compatibility mode dictates which schema edits are legal, and it comes down to defaults:
- Under BACKWARD compatibility, you can add a field only if it has a default (old data lacks the field, so the new reader fills the default), and you can delete a field that had a default (new reader ignores the missing old field). You cannot add a required field with no default — old messages couldn't satisfy it.
- Under FORWARD compatibility, the reverse: you can add a field freely (old readers ignore it) and delete a field that had a default.
- Under FULL, only changes that satisfy both — effectively, add or remove fields that have defaults.
The practical rule that keeps you out of trouble: always give new fields a default, and never remove a field that lacks one. That single habit satisfies BACKWARD (the common default) and most of FULL.
Avro vs JSON Schema vs Protobuf in the registry
- Avro — schemas are themselves written in JSON, the payload is compact binary, and Avro's resolution rules make schema evolution clean. The traditional Kafka default.
- JSON Schema — human-readable, best when consumers want plain JSON and simplicity over compactness; ties into your existing JSON Schema tooling.
- Protobuf — best when you're already invested in protobuf or need maximum efficiency for very high volume; see binary formats compared.
All three get the same registry benefits: a versioned history and enforced compatibility. Choose based on whether you value Avro's evolution story, JSON's readability, or protobuf's efficiency.
Practical guidance
- Keep the default BACKWARD mode unless you have a specific reason — it matches how most teams deploy (consumers first) and enables topic replay.
- Default every new field. It's the one rule that prevents most compatibility rejections.
- Use TRANSITIVE modes for long-lived topics where consumers may be far behind.
- Validate before you ship — generate a JSON Schema from sample events and validate messages against it in CI, so an incompatible change fails a test, not a consumer.
For the surrounding patterns see change data capture with Kafka, CloudEvents, and generating an Avro schema from JSON.