How We Know It Works
“It seems to work” is not a standard.
Every system we build is tested against past work where the right answer is already known, before launch and before every change.
What an evaluation is
An evaluation set is a collection of real past cases (submissions, service requests, documents, questions) where you already know the correct answer. The system runs on all of them, and every output is scored against that answer.
In improvement terms, it is both the baseline you measure against and the check that keeps the gains. That turns a subjective demo into numbers, and it makes change safe. When someone wants to swap a model, rewrite a prompt, or add a data source, the evaluation set is run again, and the numbers say whether the change made things better or worse. Without one, every change is a guess.
What a scorecard measures
For an AI system that classifies commercial submissions, a scorecard might track:
| Metric | What it tells you |
|---|---|
| Industry classification accuracy | How often the proposed class matches what an experienced account manager chose |
| Carrier eligibility accuracy | Whether the market list matches appetite and appointments |
| Missing-information detection | Whether gaps are caught before the submission goes to market |
| Unsupported-information rate | How often the output states something that is not in the source documents |
| Human escalation rate | How much work still goes to a person, and whether that is the right work |
A Real Example
We ran this on our own agency’s platform.
The Way Agency’s operating platform ingests carrier downloads (IVANS, in the ACORD AL3 format) and turns them into policy records. On September 1, 2026 we tested that pipeline against the agency management system as the gold standard, policy by policy.
First run: the test was wrong
The first scorecard paired the newest downloaded record with the newest AMS record for each policy.
| Field | Compared | Reported accuracy |
|---|---|---|
| Effective date | 696 | 73% |
| Expiration date | 696 | 73% |
| Premium | 696 | 26% |
Those numbers looked like a broken parser. They were a broken test: “newest” was often a different term of the same policy, so every field disagreed.
Second run: pair the same term
| Field | Compared | Mismatches | Accuracy |
|---|---|---|---|
| Effective date | 585 | 0 | 100% |
| Expiration date | 585 | 0 | 100% |
| Line of business | 585 | 0 | 100% |
| Carrier | 585 | 1 | 99.8% |
| Premium | 585 | 415 | 29% |
Dates and line of business were exactly right. The single carrier mismatch was a carrier code landing where a name belonged. Premium was the one real problem, and a single percentage said nothing about why.
Breaking the premium number apart
Looking at the premium mismatches one by one found three different causes (a policy can have more than one):
- 349 policies were actually right. The test compared only the newest record of a term, and compared six-month term premiums against annualized ones. That was a flaw in the test.
- 201 policies came through with no premium at all. That was the real defect. One carrier feed put the amount in a remarks field the parser was not reading, so every download from that feed arrived blank.
- 36 policies genuinely disagreed, mostly an endorsement premium against the full policy premium. That is not a bug. It needs a business rule about which one the record should show.
The test also turned up something nobody asked about. On 107 policies, the download held a newer term than the AMS did: the “gold standard” was the stale record. Only 4 policies were true cases of a renewal that never downloaded.
What It Teaches
Four lessons we bring to every engagement.
- Test the test. A bad comparison can make a working system look broken, or a broken one look fine.
- Never stop at one number. “29% accurate” hid a comparison flaw, a real defect, and a policy question, and each needs a different fix.
- Your gold standard can be wrong. Check disagreements in both directions before calling either side the error.
- Measure the output, not the plumbing. What matters is whether the record in your system is right.
That pipeline isn’t AI, and it still needed this.
A file parser follows fixed rules and gives the same answer every time. AI systems are probabilistic: the same input can produce different outputs, and a model update can change behavior overnight. If deterministic software needs a scorecard, AI needs one more.