Structured JSON from Local LLMs with Ollama

Reliable JSON without a cloud API

Running models locally with Ollama gives you privacy and zero per-token cost — but a raw local model is just as happy to wrap its answer in prose as any cloud model. Since version 0.3.0, Ollama solves this with the `format` parameter: pass a JSON Schema and Ollama applies constrained decoding during inference, masking any token that would break the schema. The output is guaranteed to parse, and — counterintuitively — it's usually *faster*, because the model never spends tokens deliberating about formatting.

Passing a JSON Schema directly

Send your schema as the format value. Every generated token is constrained to keep the output valid:

import ollama

schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"},
        "skills": {"type": "array", "items": {"type": "string"}},
    },
    "required": ["name", "age", "skills"],
}

resp = ollama.chat(
    model="llama3.1",
    messages=[{"role": "user", "content": "Describe a fictional software engineer."}],
    format=schema,
)
print(resp["message"]["content"])   # always valid JSON matching the schema

Using Pydantic for schema + validation in one step

You rarely want to hand-write JSON Schema. Define a Pydantic model, generate the schema with model_json_schema(), and validate the reply with model_validate_json() — schema and parsing from one source of truth:

from pydantic import BaseModel
import ollama

class Engineer(BaseModel):
    name: str
    age: int
    skills: list[str]

resp = ollama.chat(
    model="llama3.1",
    messages=[{"role": "user", "content": "Describe a fictional software engineer."}],
    format=Engineer.model_json_schema(),
)
engineer = Engineer.model_validate_json(resp["message"]["content"])

Why it's faster, and how to keep it reliable

In direct measurements, the same prompt has run several times faster with format than without — one reported case dropped from ~32s to ~5s — because constrained decoding removes the model's freedom to ramble. Constrained decoding guarantees *shape*, not *correctness*, so a few habits still matter:

• Keep schemas flat. Deeply nested structures are harder for smaller local models; flatten where you can.
• Reach for a bigger model when fields go wrong. A larger or instruction-tuned variant (e.g. a 12B) improves semantic accuracy even though the shape is already guaranteed.
• Still validate. The JSON will match the schema, but business rules (a price that must be positive, an enum you actually accept) belong in a Pydantic validator or a schema validation pass.
• Use the tools parameter for function calling on capable models (Llama 3.1+, Qwen 2.5+, Mistral) when you want the model to *choose* an action rather than fill a fixed shape.

To build the schema you'll pass, prototype a sample and generate a JSON Schema from it, or convert a sample to Python to get the Pydantic model. For the cloud-vs-local trade-offs, compare with LLM JSON Mode & Structured Outputs and TOON vs JSON for trimming prompt tokens on local hardware.