This benchmark tests whether large language models can infer a specific latent theme from a few examples, use anti-examples to reject the broader but wrong pattern, and then identify the one true match among close distractors.

Each item gives the model:

• 3 positive examples
• 3 anti-examples that fit a broader or adjacent pattern but not the exact one
• 8 candidates, with exactly 1 hidden true match

Models score all 8 candidates. We then measure whether the correct candidate is ranked first, and how highly it tends to be ranked overall.

V2 uses 1,247 validated prompts and adds stricter ambiguity filtering plus harder evaluation slices. The previous published release is being archived as V1.

This is the main result.

The chart uses inverse-rank score, a higher-is-better transformation of the average rank assigned to the correct answer. It is easier to interpret than raw average rank and avoids the old problem where a random model would sit around the middle of the scale.

The subset contains 623 items where at least two distinct full-coverage model buckets miss the item. A case does not enter this subset just because two nearby variants from the same family fail; it needs broader model disagreement.

Main takeaway:

• the frontier cluster is tight
• weaker non-reasoning systems fall off sharply

This heatmap compares model behavior at the item level on the same hard subset.

Models that tend to score the same prompts similarly cluster together, while more unusual models sit further away from the pack. This is useful because raw leaderboard position is only part of the story: two models can have similar overall scores while making different kinds of mistakes.

Here is a representative V2-style example:

Examples

• a surveyor's leveling rod
• a fishpole microphone boom
• a submarine periscope housing

Anti-examples

• a coiled steel measuring tape
• a folding wooden carpenter's rule
• a retractable cord dog leash

Correct candidate

• a collapsible stainless steel drinking straw

Theme

• physical objects that extend and retract by sliding rigid, nested tubular segments along a single axis

This shows the core idea of the benchmark:

• the model must infer a narrow mechanism, not just a broad category like "things that extend"
• the anti-examples are deliberately close enough to tempt a broader but wrong rule
• the correct answer is only obvious if the model identifies the precise latent theme

Multiple strong LLMs generate candidate themes from random seeds. These themes are meant to be narrow, specific, and checkable rather than broad trivia categories.

For each theme, models generate:

• true examples that fit the exact theme
• anti-examples that fit a broader or neighboring pattern but not the exact one

Generated examples and anti-examples are reviewed by other models. Weak or internally inconsistent items are removed.

The benchmark runs explicit validation prompts that ask models to:

• check whether the true candidate matches the stated theme
• infer alternate themes from the examples
• detect cases where the candidate pack is too ambiguous

V2 adds a stricter ambiguity / exclusivity screen here.

Each benchmark prompt shows:

• 3 examples
• 3 anti-examples
• 8 candidates

Exactly one candidate is the hidden fourth true example. Models score all 8 candidates, and the results are turned into leaderboard metrics.

• Sycophancy Benchmark
• Writing Benchmark
• Translation Benchmark
• PACT: Multi-Round Bargaining Benchmark
• BAZAAR: Auction Market Benchmark
• Extended NYT Connections
• Step Game
• Elimination Game