Can I query a Parquet file without Spark or a data warehouse?

Yes — DuckDB runs SQL directly against local Parquet files with no server or cluster, and pandas/pyarrow can read one into a DataFrame in a couple of lines, both shown above. Parquet's advantages hold even at a single-file, single-machine scale, not just at data-warehouse scale.

Do I need to convert JSON to Parquet myself, or do tools do it automatically?

Depends on the pipeline. Managed services like AWS Glue, Databricks Auto Loader, and BigQuery's load jobs can convert JSON/NDJSON to Parquet as part of ingestion automatically; for a smaller or custom pipeline, the pandas/pyarrow snippet above (or Spark's df.write.parquet(...)) is the common manual equivalent.

Is Parquet a good format for a single small file?

Not particularly — its advantages (column pruning, per-column compression, predicate pushdown) only pay off once a file is large enough that skipping unread columns or unread row-groups actually saves meaningful I/O. For a handful of records, plain JSON is simpler and the performance difference is negligible.

Why not just use Parquet for an API response instead of JSON?

Parquet is a file format built for batch reads over many records at once, with per-file metadata and column statistics that only pay off at that scale — encoding a single API response as Parquet would add overhead with none of the benefit. It's the right tool once you're storing or querying a dataset, not for point-to-point request/response traffic.

Parquet vs. JSON — Columnar Storage for Analytics at Scale

A different axis entirely

MessagePack, CBOR, and Avro are "JSON, but smaller" — same row-shaped data model, binary-encoded. Protobuf goes further, trading JSON's flexibility for a compiled, versioned schema between two services. Parquet solves a different problem than either: it's not about making one record smaller or safer to evolve, it's about reading *billions* of records without paying for the ones you don't need — which requires rethinking how data is laid out on disk, not just how it's encoded.

If you're choosing a JSON alternative for service-to-service calls, the Protobuf comparison linked above is the one you want (and JSONKit's JSON to Protobuf generator will infer a starting .proto schema from a real payload). This post is about the other kind of "too much JSON" problem: querying a dataset, not exchanging one message.

Row-oriented vs. column-oriented, concretely

JSON, MessagePack, Avro, and Protobuf are all row-oriented — record one, in full, then record two, in full:

json

{"order_id": "A1", "user_id": 41, "total": 129.99}
{"order_id": "A2", "user_id": 88, "total": 44.50}
{"order_id": "A3", "user_id": 41, "total": 9.99}

Parquet stores the same data column-oriented — every order_id together, then every user_id together, then every total together. That single structural change is the entire reason it exists:

Column pruning. SELECT total FROM orders reads *only* the total column off disk. A row-oriented format has to read and parse every field of every record just to extract the one you asked for, even if your query never touches order_id.
Better compression. A column of nothing but timestamps, or a repeated category string across millions of rows, compresses dramatically better sitting next to similar values than interleaved with unrelated fields the way it is in a JSON array of objects.
Predicate pushdown. Parquet stores per-column min/max statistics per chunk, so a query filtering WHERE total > 1000 can skip whole chunks of the file without decompressing or reading them at all.

Getting JSON into Parquet

python

# pip install pandas pyarrow
import pandas as pd

df = pd.read_json("orders.jsonl", lines=True)
df.to_parquet("orders.parquet")   # columnar, compressed, queryable in place

A nightly job that converts a day's worth of JSON application logs into Parquet is a standard data-engineering pattern — not because JSON is "wrong" for logging, but because ad-hoc analytical queries over Parquet run an order of magnitude faster, and cost less in pay-per-byte-scanned engines like Athena, than the same query run directly over raw JSON or NDJSON files.

Where this shows up in practice

A nightly job that converts a day's worth of JSON application logs into Parquet is a standard data-engineering pattern — not because JSON was "wrong" for logging (it's still the right choice for writing the logs in the first place, one line at a time, from many services), but because ad-hoc analytical queries over the accumulated Parquet run an order of magnitude faster, and cost less in pay-per-byte-scanned engines like Amazon Athena, than the same query run directly over the raw JSON or NDJSON files. This is the format underneath Spark, Athena, BigQuery's external tables, Snowflake's external stages, and most data-lake pipelines — JSON in, Parquet at rest, SQL on top.

Choosing between them

	JSON / NDJSON	Parquet
Layout	Row-oriented, text	Columnar, binary
Schema	None, self-describing	Embedded in the file
Best for	Writing records one at a time (logs, events)	Querying many records at once
Human-readable	Yes	No
Query a single field cheaply	No — must parse every full record	Yes — reads only that column
Typical tooling	Any language, any log shipper	Spark, Athena, BigQuery, DuckDB, pandas

The practical rule: JSON (or NDJSON) is usually still how data gets *written* — one event at a time, from many independent producers, with no coordination needed on a shared schema file. Parquet is how the same data gets *read back* once "how many orders over $100 last month" matters more than "here is one order." Most mature pipelines use both — JSON at the write/ingest edge, Parquet after a batch or streaming conversion step, for exactly the reasons above.