Computer Vision JSON: COCO Format, Bounding Boxes & Annotations

A picture is worth a thousand JSON coordinates

Every object-detection model — a self-checkout camera spotting produce, a self-driving car tracking pedestrians, a photo app finding faces — outputs its predictions as JSON: a class label, a confidence score, and a bounding box. The dominant dataset format tying it all together is COCO (Common Objects in Context), and once you know its shape, you can read the annotation files behind most public vision datasets and the output of most detection models.

Bounding boxes: four numbers, several conventions

A bounding box is just four numbers describing a rectangle around a detected object — but which four numbers, and in what order, varies frustratingly by tool:

{ "bbox": [120, 45, 200, 150] }
// COCO reads this as: x=120, y=45, width=200, height=150
// A Pascal VOC consumer would misread the same array as x_min=120, y_min=45, x_max=200, y_max=150

This mismatch is the single most common bug in computer vision pipelines — a box that renders in completely the wrong place because the consumer assumed the wrong convention. Always check which convention a model or dataset documents before parsing its `bbox` array, since the field name alone never tells you.

The COCO annotation format

A full COCO dataset file is JSON with four top-level sections that reference each other by ID:

{
  "images": [
    { "id": 1, "file_name": "photo001.jpg", "width": 640, "height": 480 }
  ],
  "annotations": [
    {
      "id": 1001,
      "image_id": 1,
      "category_id": 3,
      "bbox": [120, 45, 200, 150],
      "area": 30000,
      "iscrowd": 0
    }
  ],
  "categories": [
    { "id": 3, "name": "dog", "supercategory": "animal" }
  ]
}

The relational structure mirrors a normalized database: annotations[].image_id points into images, and annotations[].category_id points into categories, so a category name or image filename is never duplicated across every box that references it. iscrowd: 1 marks a region containing an indistinguishable group of the same object (a crowd of people) rather than one clean instance — a detail that changes how evaluation metrics score that annotation.

Model prediction output: similar shape, added confidence

A trained detection model's raw output looks like a COCO annotation with one addition — a confidence score, since the model isn't certain the way ground-truth human annotations are:

{
  "predictions": [
    { "category": "dog", "bbox": [118, 47, 198, 148], "score": 0.94 },
    { "category": "person", "bbox": [300, 20, 150, 400], "score": 0.87 },
    { "category": "cat", "bbox": [50, 60, 80, 70], "score": 0.31 }
  ]
}

Application code typically filters this by a confidence threshold (say, score > 0.5) before acting on a detection — that low-confidence 0.31 cat is exactly the kind of prediction a real pipeline discards.

Segmentation: beyond boxes, to exact shapes

A bounding box is a rectangle; segmentation describes the object's actual outline, either as a polygon (a list of points) or a pixel mask, both still expressed as JSON in COCO:

{
  "segmentation": [[120, 45, 320, 45, 320, 195, 120, 195]],
  "bbox": [120, 45, 200, 150]
}

The polygon here happens to be a rectangle for simplicity — real segmentation polygons trace irregular object outlines with many more points, useful when you need the exact silhouette (background removal, precise area measurement) rather than just an approximate region.

Annotation tools export this exact shape

Labeling tools like Label Studio, CVAT, and Roboflow all support exporting directly to COCO JSON, which is what makes it the practical lingua franca between "a team labeled some images" and "a model trained on them" — you rarely hand-write this JSON; you export it from a labeling tool and feed it straight into a training pipeline (YOLO, Detectron2, etc.) that expects the format.

Working with vision JSON in practice

1. Inspect a large COCO file's structure. A real dataset's annotations array can have tens of thousands of entries — paste a sample into JSONKit's JSON Formatter to check the shape before writing loader code against it.
2. Validate before training. A malformed category_id that doesn't exist in categories fails silently in many training scripts — spot-check a sample with the JSON Schema Validator against a schema you define once.
3. Diff two annotation versions. When a labeling team re-annotates a batch, JSON Diff surfaces exactly which boxes moved or which labels changed between versions.