A benchmark for vision models

Jigsaw Bench

A model gets scans of a jigsaw puzzle's loose pieces and has to work out the picture they make. No tools, no code.

Why jigsaws

Jigsaws and layered visual understanding

Jigsaws are relaxing: find the next piece and see the image being revealed. But imagine you're not allowed to move any of the pieces. Then a jigsaw turns into a challenging visual processing task involving two steps:

Put it together
Work out how the pieces fit: which ones form the edges, which colours and shapes continue across a join, which way up each piece goes.
Look inside the result
Describe the assembled picture, including where everything sits relative to everything else: the barn left of the pond, the dog in front of the tractor.

Humans can't solve this task well. How well do LLMs do?

A scanned page of loose, rotated jigsaw pieces from the farmyard puzzle — **What the model gets**8 scans like this one, holding all 54 pieces, loose and rotated, up to 8 per scan.

The complete farmyard picture: red barn, farmer on a green tractor, cows, a horse, sheep, pigs in mud and a duck pond — **What it has to work out**The finished picture: what's in it, where each thing is, and how the 54 pieces fit together.

Looking only. No tools, no code.

Models get no code interpreter, no image editing and no web search, so they can't crop, rotate or stitch the scans. They have to do it in their head.

not available# what a model with a code tool might try
page = Image.open("scan_03.jpg")
piece = page.crop((412, 96, 688, 371))
piece = piece.rotate(-31, expand=True)
edges = find_straight_edges(piece)
canvas.paste(piece, grid_cell(0, 4))

Results

Models can describe the picture. They can barely put it together.

8 scored pictures, each cut at about 50, 100 and 200 pieces. The benchmark is far from solved: even the best model loses a quarter of the description score, and at about 200 pieces no model assembles anything.

0.76best description score (f1, out of 1), GPT-6 Astra

35%of neighbouring pieces placed correctly at ~50 pieces, best model (GPT-6 Astra)

0%placed correctly at ~200 pieces, by any model

Opus 5.5 medium effortSonnet 5.5 high effortHaiku 5.5 medium effortGPT-6 Astra default effortGPT-5.6 Sol medium effortGPT-5.6 Terra medium effortGPT-5.6 Luna medium effortRandom arrangement

Describing the picture

How well the description matches the rubric (1 = perfect)

Assembling the puzzle

Share of neighbouring pieces placed side by side (1 = solved). The axis stops at 0.40 so the lines can be told apart.

Show the numbers

	Description f1				Neighbour				Proximity
Model	~50	~100	~200	all	~50	~100	~200	all	~50	~100	~200	all
Opus 5.5	0.75	0.73	0.66	0.71	0.114	0.067	0.000	0.060	0.26	0.19	0.00	0.15
Sonnet 5.5	0.71	0.72	0.63	0.69	0.078	0.046	0.000	0.041	0.21	0.16	0.00	0.12
Haiku 5.5	0.58	0.54	0.44	0.52	0.027	0.001	0.000	0.009	0.08	0.00	0.00	0.02
GPT-6 Astra	0.83	0.78	0.66	0.76	0.352	0.073	0.002	0.142	0.48	0.16	0.00	0.21
GPT-5.6 Sol	0.70	0.63	0.64	0.66	0.075	0.020	0.000	0.032	0.13	0.08	0.00	0.07
GPT-5.6 Terra	0.66	0.59	0.48	0.58	0.038	0.028	0.000	0.022	0.10	0.11	0.00	0.07
GPT-5.6 Luna	0.60	0.50	0.41	0.50	0.062	0.022	0.000	0.028	0.14	0.07	0.00	0.07
Random	–	–	–	–	0.019	0.010	0.005	0.011	0.00	0.00	0.00	0.00

Assembly, one example

What Opus built

The farmyard puzzle (54 pieces), rebuilt from Opus's answer: each piece is moved to the cell where Opus placed it. Yellow outlines mark the 12 of 54 pieces in their true place. Its neighbour score is 0.23, against 0.02 for a random arrangement.

Sky goes roughly to the top and mud to the bottom, but little else lines up.

The farmyard picture divided into its piece grid — **The puzzle**

The farmyard picture rearranged according to Opus's answer — **Opus's arrangement**

Reasoning effort

More thinking helps only a little

Opus 5.5 assembling the 8 scored puzzles of about 50 pieces at each effort level. High effort did no better than the default. Extra-high effort did somewhat better, but used about 40 times as many tokens, some 50,000 per puzzle. At max effort Opus reasoned until it hit the 64,000-token output limit and never gave an answer. A retest with 128,000 tokens, the most the API allows, ended the same way: 21 minutes of reasoning on one puzzle, then cut off before writing a grid.

medium0.111,142 tokens

high0.101,505 tokens

xhigh0.1552,008 tokens

max0.0064,000 tokens · no answers

Under each bar: median output tokens per puzzle. Dashed line: random arrangement ≈ 0.02.

Orientation

Portrait pictures come out sideways

Every answer states whether the picture is landscape or portrait. Most models call nearly every portrait picture landscape, so their whole description is turned 90°. Opus got none of the 9 portrait cuts right. The smaller GPT models lean the other way: GPT-5.6 Luna calls most pictures portrait, so it gets the portrait ones right and the landscape ones wrong.

Model	Landscape pictures	Portrait pictures
Opus 5.5	15/15	0/9
Sonnet 5.5	15/15	3/9
Haiku 5.5	8/15	2/9
GPT-6 Astra	15/15	1/9
GPT-5.6 Sol	14/15	3/9
GPT-5.6 Terra	8/15	5/9
GPT-5.6 Luna	4/15	7/9

Piece ID tags

Numbering the pieces changes nothing

For the assembly task each piece gets a random three-digit tag. We ran the description task with and without the tags on otherwise identical scans: the scores barely move. The tags only helped the models that miscounted the pieces, Haiku and GPT-5.6 Luna, count them.

Part of a scan page without tags — Plain scan

The same part of the scan page with an ID tag below each piece — Tagged scan

	f1		Count error
Model	Plain	Tagged	Plain	Tagged
Opus 5.5	0.71	0.70	0.0%	0.0%
Sonnet 5.5	0.69	0.68	0.0%	0.0%
Haiku 5.5	0.52	0.54	6.7%	0.7%
GPT-6 Astra	0.76	0.75	0.0%	0.0%
GPT-5.6 Sol	0.66	0.63	0.9%	1.1%
GPT-5.6 Terra	0.58	0.57	1.9%	0.9%
GPT-5.6 Luna	0.50	0.50	4.5%	1.5%

Method

How it works

Pieces in
The model gets the scanned pages and one prompt. The pieces are never assembled for it.
Answer out
Either a description of the finished picture, ending with its orientation, grid and piece count, or the assembled grid of piece IDs.
Scored
A judge model grades descriptions against a hand-reviewed rubric and the real picture. Assemblies are checked directly against the true layout.

The puzzles

AI-generated pictures, cut with classic tab shapes, rotated at random and rendered as scans. The puzzle size is fixed, so more pieces means smaller pieces. The 8 scored pictures are each cut at about 50, 100 and 200 pieces (24 puzzles). To keep them out of training data, they're shown here only pixelated, with one small part left sharp.

Scored picture A, pixelated — Picture A54 / 96 / 187

Scored picture B, pixelated — Picture B54 / 96 / 216

Scored picture C, pixelated — Picture C54 / 96 / 187

Scored picture D, pixelated — Picture D54 / 96 / 187

Scored picture E, pixelated — Picture E54 / 96 / 187

Scored picture F, pixelated — Picture F48 / 108 / 192

Scored picture G, pixelated — Picture G54 / 96 / 216

Scored picture H, pixelated — Picture H54 / 96 / 216

The farmyard puzzle used as the example on this page is not one of them. It isn't scored, so it can be shown in full.

Scoring a description

Each picture has a rubric of 34–49 weighted items. A model drafted each rubric from the picture, then a person checked every item against the picture, correcting or removing items where needed. The judge marks each one and also checks every claim in the answer against the picture. Here is part of Opus's grade for the farmyard puzzle.

Farmyard · Opus 5.5

44 rubric items, 62 claims in the answer

Rubric item	Answer said	Verdict	Points
A red barn with white trim and open doors.	A large red barn with white trim, X-braced doors	correct	2 / 2
Three pink pigs wallowing in mud.	Several pink pigs wallow in a large muddy puddle	partial	1 / 2
Five cattle in total (three adult cows and two calves).	(not mentioned)	missing	0 / 1
About seven adult chickens (five clear hens and roosters at lower left, one half-hidden pale bird behind them, and the rooster on the cow's back in the barn) plus two small yellow chicks.	(not mentioned)	missing	0 / 0.5
Barn on the left, pasture cows and horse on the right behind a fence across the middle with the sheep just in front of it, mud with pigs and a duck pond across the foreground, and an open countryside view at the top centre.	sky and countryside at the top, a band of farm buildings and animals in the middle, and mu…	partial	1 / 2

completeness 0.72andcorrectness 0.90→f10.80

Completeness

How much of the rubric the answer got: correct earns an item's full weight, partial half.

Correctness

The share of the answer's own claims that are true. Inventing things lowers it.

f1

The harmonic mean of the two, so a model can't score well by saying little, or by saying everything.

Orientation, grid size and piece count are scored separately and don't change f1.

Scoring an assembly

The model answers with a grid of piece IDs, which is checked against the true layout without a judge. It is told the grid size. The neighbour score is the share of truly adjacent pairs that the answer puts side by side in the right order, so a correct patch still counts if it's shifted. The direct score is the share of pieces in exactly the right cell. A random arrangement scores about 1 divided by the number of pieces on both.

Caveats

Claude and GPT models only so far. Other vision models are next.
One run per puzzle. 24 puzzles per model and task, so small differences between models are within noise.
The judge is Opus 5.5, which is also one of the models tested.
Effort levels: each model ran at its default effort (shown in the legend) unless stated.