Home / Methodology
Methodology v0.2
How we rank
Current status: every ranking is initial editorial, pending tests. No quality test on a task set has been run, so no Loadout Score is shown. Agent-Ready points (criterion A) are now evidence-backed from a public-docs review on 2026-10-19. Smoke tests and third-party evidence reviews are in the test logs, each labeled for what it is.
Principles
- Rankings are never for sale. Featured, sponsored and affiliate status never touches the score.
- Use-case first. A tool is scored for a job, so the same tool can rank #1 in one category and #5 in another.
- Reproducible. Every tested score links to a test log: prompt set, model/version, date, raw outputs and grader notes.
- Status labels.
Initial editorial, pending tests(today) →Tested v1→ re-tested on a cadence. Editorial seeds carry a provisional badge and never show a numeric score until tested.
Score formula
Loadout Score (0–100) = Σ (criterion score 0–10 × weight%) × 10
The criteria are shared across categories. Weights vary by use case.
| Code | Criterion | What we measure |
|---|---|---|
| Q | Output quality | Task success rate / blind-graded quality on a fixed task set |
| S | Speed & reliability | Time to result, failure/retry rate, uptime incidents |
| C | Control & editability | Ability to steer, iterate, edit, keep consistency |
| V | Value | Output per dollar at the cheapest tier that completes the task set; free-tier usefulness |
| A | Agent-Ready | API / MCP / CLI / SDK / docs |
| E | Ecosystem & integration | Exports, engine/IDE/app integrations, file formats |
| T | Trust | Data/privacy terms, commercial-use rights, licensing clarity |
Weights per use case (%)
| Use case | Q | S | C | V | A | E | T |
|---|---|---|---|---|---|---|---|
| 1. AI Coding Agents & IDEs | 35 | 10 | 15 | 10 | 15 | 10 | 5 |
| 2. Vibe-Coding App Builders | 30 | 10 | 15 | 15 | 5 | 15 | 10 |
| 3. General AI Assistants | 35 | 10 | 10 | 15 | 10 | 10 | 10 |
| 4. Research & Search | 35 accuracy + citation correctness | 10 | 10 | 10 | 10 | 10 | 15 |
| 5. Agent Builders & Automation | 25 | 15 | 15 | 10 | 15 | 15 | 5 |
| 6. Agent Dev Stack | 25 | 15 | 10 | 10 | 25 | 10 | 5 |
| 7. Image Generation | 40 | 10 | 15 | 15 | 5 | 5 | 10 commercial rights |
| 8. Video Generation & Editing | 40 | 10 | 15 | 15 | 5 | 5 | 10 |
| 9. Voice, Audio & Music | 35 | 10 | 15 | 15 | 10 | 5 | 10 |
| 10. Game Dev: 3D & Art Assets | 30 topology, UVs, textures | 10 | 15 style consistency | 10 | 10 | 20 engine/DCC export, rigging | 5 |
| 11. Game Dev: Design, Prototyping & Engine AI | 30 | 10 | 15 | 10 | 15 | 15 | 5 |
| 12. Local & Open Models | 25 | 15 | 15 | 15 | 15 | 10 | 5 |
Agent-Ready score (0–10)
| Signal | Points |
|---|---|
| Public REST/GraphQL API with self-serve keys | 3 |
| Official MCP server (remote or local), maintained | 2 |
| Official CLI or headless mode | 2 |
| Official SDK(s) (≥1 of TS/Python) | 1 |
| Machine-friendly docs (OpenAPI spec, llms.txt, examples) | 1 |
| Agent-friendly auth & limits (scoped tokens, documented rate limits, usage API) | 1 |
Badges: 9–10 Agent-native, 6–8 Agent-ready, 3–5 Partial, 0–2 Human-only. Every point is evidence-linked to a doc URL, and an agent re-checks it monthly. Current scores: Agent-Ready leaderboard.
Evidence rules (v0.2)
- Each signal is binary (full points or 0). There's no partial credit.
- A signal must cover the tool's core advertised capability. For example, an API for a side feature doesn't earn the API points.
- API: a documented public API whose keys a customer can get themselves. Keys that need a paid self-serve plan still count (noted). Partner-, enterprise- or waitlist-only access doesn't.
- MCP: an official MCP server exposing the tool's own capability. Docs-search-only servers and MCP client support don't count.
- CLI: an official CLI or documented headless mode. General assistants get credit for a same-vendor official CLI.
- Docs: llms.txt, an OpenAPI spec, or markdown/plain-text docs an agent can fetch.
- Auth & limits: scoped/revocable tokens, OR documented rate limits, OR a usage API.
- We use only public pages, with no account and no API calls. Every 0 links to the page we checked. Each URL's HTTP status is recorded on the check date (pages that block automated fetch are noted).
Agent-run test protocol
- Task sets per use case of 10–20 fixed, versioned tasks. Examples: coding = build a CRUD app, fix 5 seeded bugs in a repo, add tests. Game 3D = 5 prompts → export GLB → import into Godot/Unity → check polycount, UVs and textures. Research = 10 questions with known answers, grading citation accuracy.
- Runner agent executes tasks via API/CLI where possible (otherwise browser automation) on the cheapest paid tier that can complete them, recording time, cost and failures.
- Grading: objective checks first (tests pass, file imports, citation resolves). Then blind pairwise grading by two LLM judges plus human spot-check on 20% of items. Inter-judge disagreements over 2 points go to human review.
- Publish: score, sub-scores, raw logs, harness version, date. Re-test monthly (fast-moving categories) or quarterly. Any major release of a ranked tool triggers a re-test.
- Movement: ranks recompute weekly. Ties are broken by Q. A tool must beat the incumbent by ≥2 points to swap rank (hysteresis prevents noisy flip-flopping).
Test labels
- Tested vN: the tool ran our versioned task set and was graded. Only this produces a Loadout Score. None yet.
- Hands-on smoke test: we installed/ran the tool from this box without credentials (install, version, headless flags). It doesn't measure quality.
- Evidence-based evaluation (not tested by Loadout): a reading of public third-party benchmarks or docs. We never present it as our own test. It can confirm or question an editorial order but can't create a score on its own.
- Harness validation: a check of our own measuring tools against known reference inputs.
Ranks change only on tested scores (with the ≥2-point hysteresis) or on a documented factual error. Evidence reviews published on 2026-10-19 changed no rank. See test logs and movers.
Conflicts
Sponsors cannot buy, alter, or delay a ranking, test, or review. Sponsored tools are tested like everyone else. See disclosures.
Changelog
- v0.2 (2026-10-19): Agent-Ready evidence rules added and all 53 tools scored from a public-docs review. Test-label taxonomy added (hands-on smoke test / evidence-based evaluation / harness validation). First logs published for AI Coding Agents and Game Dev 3D. Weights unchanged; no rank changed.
- v0.1 (2026-10-19): initial rubric and weights published. Seed rankings are editorial, with no tests run.