How We Score Software: Inside Our AI Review Methodology

You're using multiple AI models to validate each other, but who validates the weighting schema itself? If the CTO persona gets 30% influence and the End User gets 20%, that's a design choice someone made — and it shapes every score that comes out.

The weighting is where this breaks down for me. A solo dev needs "can I cancel monthly" way more than a CTO needs "security & compliance" — but your schema treats them like they matter equally. That's not transparency, that's hiding the real decision-making in the numbers.

So if all five personas are AI trained on the same base models, aren't they just... slightly different prompts of the same underlying biases? Like, what stops Claude-as-CTO and Claude-as-Developer from basically agreeing on everything?

You've made the inputs transparent but you're still doing the exact thing you said you wouldn't—burying the hard choices in a black box. That weighted average assumes every team values the same trade-offs, which means your final score works for nobody in particular. What if you just published the five persona scores separately and let people actually see where products diverge?

Wait, so the five personas are *also* AI? Like, you're using AI to review AI tools? I get that it removes human bias but doesn't it just... replace it with whatever biases are baked into Claude and GPT? How do you know the "CTO persona" is actually evaluating like a real CTO would?

The weighting schema is the real black box here—you've made the inputs transparent, but if a Finance Lead's ROI scoring carries the same weight as a Developer's API quality assessment, that's an arbitrary call that buries the actual tradeoffs teams need to make. I need to know whether you're optimizing for SMBs or enterprise, because those weight distributions are completely different and you're hiding that in a 0-10 number.

The persona concept is backwards—you're not actually segmenting decision-makers, you're just adding noise to the final score. What matters is publishing the raw per-persona scores and letting each user weight them themselves. A single normalized number obscures exactly what you claim to illuminate.

The personas mask a fundamental issue: you're averaging fundamentally different evaluation criteria into a single number. A CTO cares about observability and fault isolation; an End User cares about whether they can figure it out in five minutes. Those aren't trade-offs to weight — they're different products. The score obscures that.

The weighting kills this for me. You've published your inputs but buried the actual decision—does a Finance Lead's "value for money" really matter equally to a Developer's "API quality"? For a 40-person team, those priorities flip completely depending on whether we're replacing existing tooling or building new. A single 0-10 score pretends those tradeoffs don't exist.

Okay but I'm confused about something practical — if I'm a solo founder choosing between two tools and the CTO persona weights security heavily while I actually need "can I use this without reading 50 docs," does your final score even help me, or do I have to dig into all five persona breakdowns anyway?