How We Score Software: Inside Our AI Review Methodology

Most software review platforms collect opinions from self-selected users and average them. That method worked when buyer budgets were small and stakes were low. It does not work for AI software in 2026, where a $40,000 annual contract can depend on whether a single integration ships in the first quarter.

Our methodology exists to answer one question with structure: is this the right tool for the kind of company you run? No star ratings shaped by who had time to leave one. No review bundles bought in exchange for logo placement. A transparent process you can read end to end, applied identically to every product in the catalog.

The six-seat panel

Every product is reviewed by six AI personas, each grounded in a distinct professional lens. The panel is fixed, published on our Meet the Panel page, and identical for every product we score. You can read the same product's review through six different lenses and see exactly where they converge and disagree.

The Decision Maker

A senior buyer's perspective. Evaluates the tool against a business case: does it solve a real problem, does the pricing match the value, does procurement survive a security review, and will adoption hold up 90 days after the demo. Scores heavily on strategic fit and vendor viability.

The Domain Strategist

Reads the product as a market positioning exercise. Asks what the tool is really competing with, how its moat holds against better-funded rivals, and whether its roadmap solves tomorrow's problem or last year's. Strong signal on whether a product will still exist at renewal.

The Finance Lead

Reviews pricing with a line-item mind. Probes per-seat math at team scale, calls out usage-based pricing traps, scrutinizes enterprise tiers that hide features behind a sales call, and flags anything that creates a multi-year lock-in without a matching discount.

The Domain Practitioner

The daily user. Not a decision-maker, not a power user, just the person who opens the tool every morning. Scores ease of setup, day-30 workflow fit, documentation quality, and the friction moments that kill adoption after the first enthusiasm wears off.

The Power User

Pushes the product to its edges. Cares about the API surface, the automation hooks, the advanced features that never make the marketing page, and whether the tool rewards investment or caps your ambition. Often the first to spot where a product stops scaling.

The Skeptic

A structural devil's advocate. Stress-tests every claim the product makes, runs the numbers on benchmark assertions, looks for the feature marked "enterprise-only" that should be standard, and calls out polish that hides a fundamental limitation. The Skeptic's score is intentionally the hardest to earn.

Three models, zero echo chamber

Each persona is evaluated across three foundation models: Claude, GPT, and Gemini. Running the same prompt through three independent model families catches biases that any single model would miss. Claude tends to be stricter on architectural claims. GPT is more forgiving of enterprise marketing language. Gemini weighs ecosystem integration more heavily than either. When all three agree, the signal is strong. When they disagree, the disagreement itself is the signal.

We do not average the three model outputs into mush. Each model's score per persona is preserved, visible on the product page, and surfaced in the review body. A product that scores 8.2 from Claude and 6.4 from GPT tells you something useful before you even read a word of the text.

Eight scoring dimensions

Each persona scores a product across eight standardized dimensions:

• Functionality and Features — Does the product do what it claims, at the depth it claims?
• Ease of Use — Onboarding friction, workflow clarity, time-to-first-value.
• Performance and Reliability — Latency under load, uptime track record, degradation behavior.
• Value for Money — Price per outcome at the likely usage profile for this persona's audience.
• Customer Support — Response quality, escalation paths, self-serve documentation depth.
• Integration and Ecosystem — Native integrations, API completeness, data portability.
• Security and Compliance — SOC 2 posture, data handling, audit trail maturity, third-party exposure.
• Innovation and Roadmap — Recent velocity, public roadmap credibility, evidence of substantive iteration.

Scores are 1–10 per dimension per persona per model. That produces 8 × 6 × 3 = 144 individual data points per reviewed product. The AI Panel Score you see on the product page is the weighted aggregate of those 144 points, normalized to a 0–10 scale and rounded to one decimal.

How weighting works

We do not claim every persona matters equally for every product. A developer tool weights the Domain Practitioner and Power User more heavily than the Finance Lead. An enterprise GRC tool inverts that. The weightings are category-specific, published in the panel documentation, and held constant within a category so every product in that category is judged on the same yardstick.

Within each persona, dimension weights also vary by category. Security and Compliance carries more weight for compliance tools than it does for a design platform. The weighting matrix is reviewed quarterly and any changes are logged in the methodology changelog at the bottom of this page.

What does not influence a score

To keep this list short and concrete:

• No paid placement. A vendor cannot pay for a higher score, a faster review, or removal of a published review. We do not accept vendor fees of any kind before a review is published.
• No review trades. We do not exchange reviews for backlinks, podcast appearances, or social amplification.
• No marketing-copy ingestion. Reviews draw on product documentation, independent customer reporting, public benchmark data, and the tool's actual behavior where we can test it. Vendor-supplied talking points are noted as such and weighted accordingly.
• No vendor review cycles. Vendors do not get to preview, comment on, or request changes to a review before publication. A correction process exists for factual errors, applied after publication.

How disagreement is resolved

When the three foundation models diverge sharply on a single dimension (a spread of more than 2.5 points between highest and lowest), the review is flagged for a second pass. The second pass uses a longer context window and is explicitly asked to explain why the models disagree. The disagreement itself is preserved in the published review body under a "Where the models split" section. We do not silently average out disagreement; the user benefits from seeing where three independent models read the product differently.

What a finished review contains

Every published product review includes the AI Panel Score, a per-persona score breakdown with commentary, a strengths and weaknesses summary, pricing analysis, a competitor and alternatives table, and FAQ section derived from the underlying review data. Individual model scores per persona are surfaced inline. A timestamp and methodology version mark every review so you can see exactly when and under which version of this document the product was scored.

A living methodology

This document is versioned. As foundation models improve, as category weightings are refined, and as we add panel members or retire dimensions, we log the change with a date and a short rationale. The current version is 1.3, published April 2026. Previous versions remain accessible for any review published under them, so a score from September 2025 can be read with the rules it was generated under.

Changelog

• 1.3 — April 2026. Added The Domain Practitioner and The Power User as distinct personas, replacing the earlier combined "End User" role. Expanded the Skeptic's mandate to include benchmark stress-testing.
• 1.2 — February 2026. Added Gemini as a third foundation model alongside Claude and GPT. Introduced per-model score visibility on product pages.
• 1.1 — December 2025. Published dimension weighting matrix by category.
• 1.0 — November 2025. Initial methodology publication.

Questions, corrections, and feedback

Factual errors in published reviews can be reported through our contact page; corrections are investigated within five business days and published with a dated correction note attached to the affected review. Feedback on the methodology itself is welcomed and reviewed as part of the quarterly methodology update cycle.