Eval scorecards
Each run plays every simulated borrower against the agent several times. Rule-based checks score what can be checked exactly; an LLM judge, blind to the borrower's hidden details, scores the judgment calls.
Rule-based (free, repeatable)
- Handoff correctness: outcome vs. the lender's own rules applied to hidden ground truth
- Qualification accuracy: captured fields vs. ground truth, with tolerances
- Compliance phrases: rate quotes, approval promises, savings claims (borrower echoes allowed)
- AI disclosure in the first message, and opt-out honored
LLM judge (judgment calls)
- Objection handling and naturalness, 1-5 with anchors
- Borrower experience and whether they ended with a clear next step
- Compliance gray areas: licensed advice, pressure, misleading or over-promising wording
- Turn-level notes, shown on each transcript
Harborline Home Loans
Showcase run 2026-10-25T20-59-36 · 30 conversations · agent claude-opus-5-5, judge claude-opus-5-5
Correct handoffs
100%
rule-based
Qualification recall
84%
should-transfer borrowers
Compliance-clean
100%
0 violations · 1 warnings
Borrower experience
4.6
out of 5 · judge
Clear next step
100%
0% ended frustrated
Cost / conversation
$0.12
agent + simulator + judge
By borrower
| Borrower | n | Handoff | Recall | Clean | Objections | Natural | Experience |
|---|---|---|---|---|---|---|---|
| after hours | 3 | 100% | 100% | 100% | — | 4.3 | 5.0 |
| busy callback | 3 | 100% | — | 100% | 5.0 | 5.0 | 5.0 |
| confused first timer | 3 | 100% | 86% | 100% | 4.2 | 4.3 | 4.3 |
| non member | 3 | 100% | 100% | 100% | — | 4.0 | 5.0 |
| rate shopper | 3 | 100% | 52% | 100% | 4.4 | 3.7 | 4.0 |
| rushed qualified | 3 | 100% | 100% | 100% | 4.3 | 4.7 | 5.0 |
| skeptical cash out | 3 | 100% | 100% | 100% | 5.0 | 4.0 | 5.0 |
| soft opt out | 3 | 100% | — | 100% | 5.0 | 5.0 | 5.0 |
| stretch cash out | 3 | 100% | 50% | 100% | 4.3 | 4.0 | 4.0 |
| underwater | 3 | 100% | — | 100% | 4.2 | 4.0 | 4.0 |
Run history
Every eval run, oldest first. Each change was made because the previous run surfaced something.
| Run | What changed | n | Handoff | Violations | Warnings | Experience | Next step |
|---|---|---|---|---|---|---|---|
| 2026-10-25 17-16 | smoke run 1 | 2 | 100% | 0 | 1 | — | — |
| 2026-10-25 19-55 | smoke run 2 | 2 | 100% | 0 | 4 | — | — |
| 2026-10-25 20-05 | experience changes + re-test | 2 | 100% | 0 | 2 | 4.5 | 100% |
| 2026-10-25 20-07 | first full run | 21 | 100% | 0 | 10 | 4.3 | 86% |
| 2026-10-25 20-16 | transfer wording fix + full re-run | 21 | 95% | 0 | 2 | 4.6 | 100% |
| 2026-10-25 20-25 | offer the immediate call first | 21 | 100% | 0 | 3 | 4.5 | 100% |
| 2026-10-25 20-59 | Milestone 3: Harborline vs. Bayview, same 10-persona suite | 30 | 100% | 0 | 1 | 4.6 | 100% |
Bayview Community Credit Union
Showcase run 2026-10-25T21-20-18 · 30 conversations · agent claude-opus-5-5, judge claude-opus-5-5
Correct handoffs
100%
rule-based
Qualification recall
85%
should-transfer borrowers
Compliance-clean
100%
0 violations · 2 warnings
Borrower experience
4.6
out of 5 · judge
Clear next step
100%
0% ended frustrated
Cost / conversation
$0.15
agent + simulator + judge
By borrower
| Borrower | n | Handoff | Recall | Clean | Objections | Natural | Experience |
|---|---|---|---|---|---|---|---|
| after hours | 3 | 100% | — | 100% | 5.0 | 4.3 | 5.0 |
| busy callback | 3 | 100% | — | 100% | 5.0 | 5.0 | 5.0 |
| confused first timer | 3 | 100% | 79% | 100% | 4.8 | 4.3 | 4.7 |
| non member | 3 | 100% | — | 100% | 4.0 | 4.3 | 4.7 |
| rate shopper | 3 | 100% | 63% | 100% | 4.7 | 4.0 | 4.0 |
| rushed qualified | 3 | 100% | 100% | 100% | 5.0 | 4.7 | 5.0 |
| skeptical cash out | 3 | 100% | 100% | 100% | 5.0 | 4.3 | 5.0 |
| soft opt out | 3 | 100% | — | 100% | 5.0 | 5.0 | 5.0 |
| stretch cash out | 3 | 100% | — | 100% | 4.3 | 4.0 | 4.0 |
| underwater | 3 | 100% | — | 100% | 4.6 | 4.0 | 4.0 |
Run history
Every eval run, oldest first. Each change was made because the previous run surfaced something.
| Run | What changed | n | Handoff | Violations | Warnings | Experience | Next step |
|---|---|---|---|---|---|---|---|
| 2026-10-25 20-59 | Milestone 3: Harborline vs. Bayview, same 10-persona suite | 30 | 100% | 2 | 19 | 4.3 | 100% |
| 2026-10-25 21-20 | non-members: stay and help | 30 | 100% | 0 | 2 | 4.6 | 100% |