Transcripts and replays
Recorded eval conversations: a simulated borrower (with hidden ground truth) texting the agent, then scored by rules and an LLM judge. Open one to read the judge's notes inline, or press Replay to watch it unfold. No API calls.
Harborline Home Loans
Run 2026-10-25T20-59-36 · 30 conversations · agent claude-opus-5-5
| Borrower | Outcome | Expected | Result | Experience | Turns | Cost |
|---|---|---|---|---|---|---|
| Qualified, Saturday night #1 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 7 | $0.149 |
| Qualified, Saturday night #2 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 6 | $0.148 |
| Qualified, Saturday night #3 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 7 | $0.154 |
| Busy, wants a callback #1 | Callback booked | Callback booked | ✓ Pass | 5.0 | 2 | $0.075 |
| Busy, wants a callback #2 | Callback booked | Callback booked | ✓ Pass | 5.0 | 2 | $0.048 |
| Busy, wants a callback #3 | Callback booked | Callback booked | ✓ Pass | 5.0 | 2 | $0.043 |
| Confused first-timer #1 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 8 | $0.169 |
| Confused first-timer #2 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 4.0 | 8 | $0.152 |
| Confused first-timer #3 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 4.0 | 8 | $0.148 |
| Qualified, but not a member #1 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 8 | $0.160 |
| Qualified, but not a member #2 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 7 | $0.124 |
| Qualified, but not a member #3 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 8 | $0.127 |
| Rate shopper #1 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 4.0 | 4 | $0.120 |
| Rate shopper #2 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 4.0 | 4 | $0.091 |
| Rate shopper #3 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 4.0 | 6 | $0.118 |
| Rushed but qualified #1 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 3 | $0.122 |
| Rushed but qualified #2 | Callback booked | Transferred to loan officer | ✓ Pass | 5.0 | 4 | $0.094 |
| Rushed but qualified #3 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 3 | $0.089 |
| Skeptical, wants cash out #1 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 9 | $0.183 |
| Skeptical, wants cash out #2 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 10 | $0.168 |
| Skeptical, wants cash out #3 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 9 | $0.150 |
| Opts out mid-conversation #1 | Opted out | Opted out | ✓ Pass | 5.0 | 3 | $0.079 |
| Opts out mid-conversation #2 | Opted out | Opted out | ✓ Pass | 5.0 | 3 | $0.053 |
| Opts out mid-conversation #3 | Opted out | Opted out | ✓ Pass | 5.0 | 3 | $0.051 |
| Cash out at 92.5% LTV #1 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 4.0 | 4 | $0.127 |
| Cash out at 92.5% LTV #2 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 4.0 | 4 | $0.106 |
| Cash out at 92.5% LTV #3 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 4.0 | 4 | $0.102 |
| Underwater on the loan #1 | Follow-up booked | Follow-up booked | ✓ Pass | 4.0 | 8 | $0.186 |
| Underwater on the loan #2 | Follow-up booked | Follow-up booked | ✓ Pass | 4.0 | 9 | $0.173 |
| Underwater on the loan #3 | Follow-up booked | Follow-up booked | ✓ Pass | 4.0 | 8 | $0.150 |
Bayview Community Credit Union
Run 2026-10-25T21-20-18 · 30 conversations · agent claude-opus-5-5
| Borrower | Outcome | Expected | Result | Experience | Turns | Cost |
|---|---|---|---|---|---|---|
| Qualified, Saturday night #1 | Callback booked | Callback booked | ✓ Pass | 5.0 | 9 | $0.185 |
| Qualified, Saturday night #2 | Callback booked | Callback booked | ✓ Pass | 5.0 | 8 | $0.169 |
| Qualified, Saturday night #3 | Callback booked | Callback booked | ✓ Pass | 5.0 | 8 | $0.168 |
| Busy, wants a callback #1 | Callback booked | Callback booked | ✓ Pass | 5.0 | 3 | $0.103 |
| Busy, wants a callback #2 | Callback booked | Callback booked | ✓ Pass | 5.0 | 4 | $0.076 |
| Busy, wants a callback #3 | Callback booked | Callback booked | ✓ Pass | 5.0 | 4 | $0.079 |
| Confused first-timer #1 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 9 | $0.200 |
| Confused first-timer #2 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 4.0 | 10 | $0.188 |
| Confused first-timer #3 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 8 | $0.149 |
| Qualified, but not a member #1 | Follow-up booked | Follow-up booked | ✓ Pass | 4.0 | 11 | $0.209 |
| Qualified, but not a member #2 | Follow-up booked | Follow-up booked | ✓ Pass | 5.0 | 11 | $0.174 |
| Qualified, but not a member #3 | Follow-up booked | Follow-up booked | ✓ Pass | 5.0 | 11 | $0.177 |
| Rate shopper #1 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 4.0 | 5 | $0.167 |
| Rate shopper #2 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 4.0 | 5 | $0.124 |
| Rate shopper #3 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 4.0 | 5 | $0.131 |
| Rushed but qualified #1 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 4 | $0.137 |
| Rushed but qualified #2 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 3 | $0.101 |
| Rushed but qualified #3 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 4 | $0.105 |
| Skeptical, wants cash out #1 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 11 | $0.216 |
| Skeptical, wants cash out #2 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 10 | $0.208 |
| Skeptical, wants cash out #3 | Transferred to loan officer | Transferred to loan officer | ✓ Pass | 5.0 | 10 | $0.175 |
| Opts out mid-conversation #1 | Opted out | Opted out | ✓ Pass | 5.0 | 3 | $0.087 |
| Opts out mid-conversation #2 | Opted out | Opted out | ✓ Pass | 5.0 | 3 | $0.061 |
| Opts out mid-conversation #3 | Opted out | Opted out | ✓ Pass | 5.0 | 3 | $0.060 |
| Cash out at 92.5% LTV #1 | Follow-up booked | Follow-up booked | ✓ Pass | 4.0 | 10 | $0.215 |
| Cash out at 92.5% LTV #2 | Follow-up booked | Follow-up booked | ✓ Pass | 4.0 | 10 | $0.187 |
| Cash out at 92.5% LTV #3 | Follow-up booked | Follow-up booked | ✓ Pass | 4.0 | 10 | $0.182 |
| Underwater on the loan #1 | Follow-up booked | Follow-up booked | ✓ Pass | 4.0 | 10 | $0.232 |
| Underwater on the loan #2 | Follow-up booked | Follow-up booked | ✓ Pass | 4.0 | 10 | $0.187 |
| Underwater on the loan #3 | Follow-up booked | Follow-up booked | ✓ Pass | 4.0 | 8 | $0.176 |