Cross-model testing

Perplexity's Agents team: "Sonnet and GPT behave quite differently when it comes to Skills." What loads on one model may not load on another. What works on Opus may fail on Haiku. Routing precision is model-dependent.

This is non-obvious. Most skill authors test on one model (the one they happen to use) and assume the routing is universal. It isn't.

What to test cross-model

For each model family you support:

1. Loading precision — Does the agent load the skill when it should? Run 10 positive queries; the skill should load on all 10.
2. Loading recall — Does the agent not load the skill when it shouldn't? Run 10 adjacent-but-distinct queries; the skill should not load on any.
3. Progressive read targeting — When the skill is loaded, does the agent open the right atomic node? Test 5 queries each mapped to a specific node; verify the agent reads that node.
4. End-to-end output quality — For a given input (e.g., "review this CSS"), does the agent produce the right output format? Test with the review-format table requirement.

Models to cover (as of 2026)

You don't need to cover every model. Cover the orchestration models your downstream users will most likely use. For this skill, the priority is Claude Sonnet + Opus (designed for) and GPT-5 (most common alternative).

How to run cross-model evals

The evals/ folder in this repo has three eval files:

• loading.jsonl — query + expected_load (true/false)
• progressive-reads.jsonl — query + expected_node
• end-to-end.md — sample tasks + expected output structure

To run them:

# Pseudo — depends on your harness. Common patterns:
agent-eval skills/design-engineering/evals/loading.jsonl --model claude-sonnet-4
agent-eval skills/design-engineering/evals/loading.jsonl --model claude-opus-4
agent-eval skills/design-engineering/evals/loading.jsonl --model gpt-5

Compare results across models. Any divergence ≥5% on the same eval set is a routing problem that needs description-tuning.

Common cross-model failure modes

1. GPT loads on broader keywords — GPT tends to route on noun matches ("CSS", "animation") more aggressively than Claude. If your description uses generic nouns, GPT may load on tangentially-related queries.
2. Claude Sonnet skips on verbose descriptions — Sonnet sometimes skips skills with descriptions over ~80 words. Tighten the description if Sonnet recall is low.
3. Haiku doesn't load multi-node skills well — Haiku can struggle with the MOC → leaf navigation. Often it stops at the SKILL.md and never opens references. If supporting Haiku, consider inlining critical nodes into SKILL.md.
4. Opus over-reads — Opus may open multiple atomic nodes "just in case." Token budget concern; not a routing bug.

When to run

• Before changing the SKILL.md description (the routing-critical line).
• Before any change to atomic node summaries (which the agent uses to decide which to read).
• Before a new release / version bump.
• Quarterly, as model behavior drifts with model updates.

Gotcha

Don't test on synthetic queries you wrote yourself. Source from real user queries (production logs, user feedback, hero queries). Synthetic queries have a flavor that biases the model toward the answer you expect — they tell you the description matches the queries you wrote, not whether it matches reality.

Sources

• Perplexity Agents team — "Designing, Refining, and Maintaining Agent Skills".
• Anthropic — eval harness patterns.
• Related: gotchas, review-format, [[../../CONTRIBUTING|CONTRIBUTING.md]].