Design benchmarks and this skill's evals
How design output gets measured — DesignBench's generation / edit / repair tasks with compile, CLIP, and MLLM-judge metrics; Design Arena's anonymous pairwise votes aggregated by Bradley-Terry — and this skill's own eval suite that borrows both shapes.
Raw markdown On GitHubreferences/meta/design-benchmarks.md
Taught in
Taste is trained, but it is also measurable, and two public benchmarks define the shapes worth borrowing. Neither measures craft directly — one measures whether generated UI matches and compiles, the other measures whether people prefer it — so this skill runs both shapes side by side with its own checklist-derived assertions. cross-model-testing covers routing evals; this node covers output quality.
DesignBench (WebPAI, arXiv 2506.06251)
900 webpage samples across 11 topics, four stacks (vanilla HTML/CSS, React, Vue, Angular), three tasks that mirror real work: generation (UI image → code), edit (change an implementation from an instruction, with image, code, or both as context, across 9 edit types: add/change/delete × text, color, position, size, shape, component-level), and repair (fix a broken UI across issue categories such as occlusion, crowding, alignment, color and contrast, overflow). Metrics: compile success, visual similarity via CLIP, code-modification similarity, and MLLM-as-judge scoring; difficulty is graded by element count, interdependencies, and cascading scope. Headline findings: models adopt framework components poorly (about 19% Angular component adoption), detect UI issues badly (average accuracy around 0.27), and edit better from code than from images.
What to borrow: the task triad. A design skill that only helps generation is untested on the two-thirds of real work that is editing and repairing.
Design Arena (by Intelligence)
A crowdsourced arena: a prompt is sent to several models, the first outputs are shown anonymously side by side, a human picks the better one, winners are matched again until a 1–4 ranking exists, and every pairwise choice is a vote. Ratings come from a Bradley-Terry fit (iterated to convergence, normalized, then 400 × log10(strength)), models below a minimum vote count are dropped, and identities stay hidden to prevent brand bias. Separate leaderboards per category — websites, UI components, mobile apps, slides, games, 3D, SVGs, data visualizations, logos, video, text-to-speech — plus an overall frontend board.
What to borrow: anonymous pairwise judgment. "Which of these two is better?" is answerable by a person or a judge model where "is this good?" is not, and it is the only honest way to score an aesthetic change.
This skill's suite
| Layer | File | Shape | Asserts |
|---|---|---|---|
| Routing | evals/loading.jsonl, evals/progressive-reads.jsonl | fixtures | The skill loads when it should; the right node opens |
| Contract | evals/review-format.eval.ts, motion-values.eval.ts, sound-values.eval.ts | eve scored | Review tables, concrete values |
| Craft floor | evals/design-bench.eval.ts + skills/design-engineering/evals/design-bench.jsonl | generation / edit / repair fixtures | Deterministic checks lifted from review-checklist: no transition: all, no scale(0), reduced-motion present, inputs ≥ 16px, labelled icon buttons, no z-index: 9999, no hover-only actions, no sound before a gesture |
| Preference | design-bench.jsonl rows tagged arena | pairwise | Two candidates per prompt; a human or judge model picks; log the win rate per node change |
The craft floor is DesignBench's repair task pointed at our own tells: each fixture is a broken snippet and the row the checklist should catch. The preference layer is Design Arena's protocol at PR scale: when a node changes what the agent produces, generate before-and-after candidates for three fixtures, judge them blind, and record the result in the PR. A node that loses the pairwise more than it wins is reverted, however good it reads.
When to apply
Before changing the SKILL.md description or any node that alters output (motion values, palette rules, sound defaults); when comparing models for a design task; when someone asks "is this skill actually helping?"
Gotcha
Compile rate and CLIP similarity reward matching the reference, including its flaws. A repair that fixes an overflow the reference also had can score lower than a faithful reproduction of the bug. Read the pairwise layer before trusting the similarity layer.
Sources
- Xiao et al., DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation (arXiv 2506.06251), and the WebPAI/DesignBench repository and annotation guideline.
- Design Arena methodology notes (notes.designarena.ai/methodology) — Bradley-Terry aggregation, anonymity, vote thresholds.
- Perplexity Agent Skills team — evals as Step 0.
- Related: cross-model-testing, review-checklist, taste-is-trained.
metaevalsbenchmarkstesting