Schema validation answers "is this the right shape." It doesn't answer "is this data right."
A JSON Schema check confirms age is an integer and email matches a string pattern — necessary, and often where JSON validation stops. It says nothing about whether age: 150 is plausible, whether total_amount should never be negative, or whether the count of rows in today's load is suspiciously different from yesterday's. That gap between *structurally valid* and *actually correct* is what dedicated data quality tools exist to close.
Great Expectations: quality rules as a JSON document
Great Expectations (GX) is the most widely adopted open-source framework for this, and its core unit — an "expectation" — is itself just a JSON object describing one assertion about your data:
{
"expectation_type": "expect_column_values_to_be_between",
"kwargs": { "column": "age", "min_value": 0, "max_value": 120 }
}A collection of these forms an Expectation Suite, saved as JSON, version-controlled alongside your pipeline code the same way you'd version a database migration. Suites cover the ground a JSON Schema can't reach: value ranges, referential checks against another table, statistical distribution checks (a column's mean shouldn't drift more than X% day over day), row-count deltas, and uniqueness constraints across a whole dataset rather than a single record.
Validation results: the JSON that tells you what actually failed
Running a suite against real data produces a Validation Result — also JSON — recording, per expectation, whether it passed and, if not, exactly which values violated it:
{
"success": false,
"results": [
{
"expectation_config": { "expectation_type": "expect_column_values_to_be_between" },
"success": false,
"result": { "unexpected_count": 3, "unexpected_list": [-5, 999, -1] }
}
]
}This is the artifact CI pipelines and alerting actually consume — diffing two validation results from consecutive pipeline runs is a fast way to tell whether a new failure is a one-off anomaly or a persistent regression that started at a specific run.
Choosing the right layer for the right check
The practical rule: use JSON Schema at the boundary where data *enters* your system (an API request body, a webhook payload, a file upload) because it's cheap, fast, and catches malformed structure before anything downstream touches it. Use a statistical/quality framework like Great Expectations *inside* your pipeline, on data that's already structurally valid but whose business correctness needs checking — because "the JSON parsed fine" and "this data is safe to load into the warehouse" are genuinely different guarantees, and conflating them is how bad data quietly reaches a dashboard with a green checkmark on it.
Where this fits with lineage and lake formats
Quality checks are most valuable placed at pipeline stage boundaries you already have visibility into — see OpenLineage for how job/dataset relationships get tracked, and Iceberg and Delta Lake for how a lakehouse table's own JSON metadata already tracks schema evolution; a quality gate sitting at the same boundary (before a new Iceberg snapshot is committed, for instance) is a natural place to fail a pipeline loudly rather than let bad data become the new "current" version silently.