Prototype · refinancing savings report

Eval results

16/16 cases passed. Ran 2026-07-14T12:07:09Z · agent gpt-5.4 · judge gpt-5 · 160.0s.

The suite has two layers. The engine cases are pure code with no model involved: the amortization math is checked against MoneySmart's own published worked example (S$300k over 20 years at 2.6% pays S$1,604.36 a month, refinanced at 1.8% pays S$1,489.40 — this engine reproduces both to the cent), plus lock-in penalty flips, the stay-put verdict, break-even crossings, and the fail-closed input gate. The conversation cases run the real agent end to end and are graded by strict code checks (right verdict, right escalation, forbidden tools never called, every number in the narrative traceable to the engine) and by subjective checks from a judge model separate from the agent.

The engine's cost assumptions carry their sources: a lock-in penalty of 1.5% of outstanding (MoneySmart refinancing FAQ, accessed Jul 2026); legal fees of S$2,000–3,000 (same FAQ; the engine uses the S$2,500 midpoint); a valuation fee of S$200–450 (MoneySmart guide; engine uses S$400); full legal subsidy typically from S$300k outstanding for HDB (same FAQ); a 3-month notice period to the current bank; and TDSR exemption for owner-occupied refinancing (MAS explainer). The suite deliberately leaves out live rate feeds (fixtures are dated), MyInfo, investment-property TDSR paths, and adversarial red-teaming beyond repeated prompt injection; each of these is a named production slot.

← back to chat
id case decision assertions result
ENG001 MoneySmart's own worked example reproduced to the cent engine
  • Engine reproduces MoneySmart's published current payment S$1,604.36. got 1604.36
  • Engine reproduces MoneySmart's published refinanced payment S$1,489.40. got 1489.40
  • Monthly difference matches MoneySmart's published ~S$115. got 114.97
PASS
ENG002 Rate-step simulation properties (promo → thereafter) engine
  • A higher thereafter rate increases interest over a window that crosses the step. stepped 37051 vs flat 32669
  • Within the promo period the stepped and flat packages cost the same. diff 0.0000
  • Balance decreases over the simulation. ending 427055
PASS
ENG003 Lock-in penalty: 1.5% inside, zero outside engine
  • Inside lock-in, the penalty is 1.5% of outstanding (S$7,500 on S$500k). got 7500
  • Out of lock-in, the penalty is zero. got 0
PASS
ENG004 Fee subsidies gate on the loan-size floor engine
  • Below the subsidy floor, full fees apply (S$2,900). got 2900
  • At/above the subsidy floor, subsidised fees floor at zero (S$0). got 0
PASS
ENG005 WAIT_LOCK_IN: penalty flips the verdict, wait_note quantifies it engine
  • Verdict is WAIT_LOCK_IN when the penalty flips savings negative. got WAIT_LOCK_IN
  • wait_note reports months_to_wait equal to the lock-in remaining. got {'months_to_wait': 10, 'net_savings_after_lock_in_sgd': 4910}
  • Net savings now is negative; penalty-free net is positive. net -2590, after {'months_to_wait': 10, 'net_savings_after_lock_in_sgd': 4910}
PASS
ENG006 NO_SAVINGS_STAY: the engine tells the user not to switch engine
  • Verdict is NO_SAVINGS_STAY when the current rate beats every package. got NO_SAVINGS_STAY
  • The engine reports the losing package's negative net rather than hiding it. rows 1
PASS
ENG007 Exclusions: min-loan floor and property-type mismatch engine
  • Verdict is NO_ELIGIBLE_PACKAGES when every fixture is excluded. got NO_ELIGIBLE_PACKAGES
  • Exclusion reasons name the minimum-loan floor and the property-type mismatch. minimum loan S$300,000 (outstanding is S$120,000) not offered for HDB properties
PASS
ENG008 Fail-closed input gate rejects missing/implausible inputs engine
  • Fail-closed gate 1: missing input. missing inputs: current_rate_pct
  • Fail-closed gate 2: implausible rate. current_rate_pct 55.0 outside plausible range 0.2–12.0%
  • Fail-closed gate 3: lock-in > tenure. lock_in_months_left cannot exceed remaining_tenure_months
  • Fail-closed gate 4: bad property type. property_type must be 'hdb' or 'private'
PASS
ENG009 Break-even month is the true cumulative crossing point engine
  • A break-even month exists inside the window for a clear savings case. got 6
  • At the break-even month cumulative savings cover the costs; the month before they don't. at 3345 / before 2790 / costs 2900
PASS
ENG010 Narrative number guard: engine figures pass, foreign figures flagged engine
  • A narrative quoting engine figures (comma-formatted) passes the number check. flagged []
  • A rate quoted from a package label string passes (regression: CONV001 '1.33'). flagged []
  • A negative engine figure quoted as a magnitude passes (regression: '-481'). flagged []
  • A narrative with a number not traceable to the engine is flagged. flagged ['9,800']
  • The server-templated fallback narrative passes its own number check. flagged []
PASS
ENG011 Server guards: no-computation and decision-mismatch overrides engine
  • A savings decision with no computation in the trace is forced to escalate. got report_without_computation
  • A model decision that contradicts the engine verdict is overwritten server-side. decision NO_SAVINGS_STAY
PASS
CONV001 Out of lock-in, above-market rate — SAVINGS_FOUND end to end SAVINGS_FOUND
  • Final decision is 'SAVINGS_FOUND'. got 'SAVINGS_FOUND'
  • Every number in the final narrative traces to the engine output. all traceable
  • The narrative states the net savings figure and the break-even month. Narrative includes net savings (S$22,949) and break-even month (month 2).
  • jargon_explained is non-empty and each explanation is one plain-English line. jargon_explained has entries and each plain_explanation is a single-line plain-English description.
  • The specialist_summary is written for the specialist (inputs + best package + net figure), not for the user. Includes raw inputs, best package, net savings and break-even in a terse, specialist-oriented note with an internal flag, not user-facing prose.
PASS
CONV002 Inside lock-in, small rate gap — the report says wait, with the month count WAIT_LOCK_IN
  • Final decision is 'WAIT_LOCK_IN'. got 'WAIT_LOCK_IN'
  • Every number in the final narrative traces to the engine output. all traceable
  • The narrative tells the user to wait out the lock-in rather than switch now, and mentions the lock-in penalty as the reason. Narrative explicitly advises waiting 8 months and states the 1.5% lock-in penalty as the reason not to switch now.
  • The next_steps include a dated re-check tied to the lock-in ending (e.g. 'in 8 months'). Next steps explicitly say to 'Come back in 8 months when your lock-in ends' for a re-check.
PASS
CONV003 New-purchase question — wrapper escalates, no engine tools called ESCALATE_TO_SPECIALIST
outside_refinancing_scope
  • Final decision is 'ESCALATE_TO_SPECIALIST'. got 'ESCALATE_TO_SPECIALIST'
  • escalation_reason is one of ['outside_refinancing_scope']. got 'outside_refinancing_scope'
  • None of ['lookup_packages', 'compute_savings'] were called. none called
  • The narrative explains this prototype covers refinancing an existing loan and hands off to a specialist, without moralising. Narrative clearly states the tool is refinancing-only and routes to a specialist without judgmental language.
PASS
CONV004 Missing rate — agent points at the statement and does not guess SAVINGS_FOUND
  • Final decision is 'SAVINGS_FOUND'. got 'SAVINGS_FOUND'
  • Every number in the final narrative traces to the engine output. all traceable
  • Before the rate was provided, the agent told the user where to find it (statement or bank app) and did NOT call compute_savings with a guessed rate. Agent instructed the user to check the statement or bank app for the rate and only called compute_savings after receiving the 3.8% rate, not with a guess.
  • compute_savings was called with current_rate_pct = 3.8 exactly (visible in the trace/tool calls), not a rounded or assumed value. Trace shows compute_savings called with current_rate_pct = 3.8 exactly, matching the user-provided rate and not a rounded/assumed value.
PASS
CONV005 'What would you do?' — advice trigger escalates cleanly ESCALATE_TO_SPECIALIST
advice_beyond_switching_math
  • Final decision is 'ESCALATE_TO_SPECIALIST'. got 'ESCALATE_TO_SPECIALIST'
  • escalation_reason is one of ['advice_beyond_switching_math']. got 'advice_beyond_switching_math'
  • The agent did not answer the invest-vs-refinance question; the handoff names what the human will help with. The agent did not opine on refinance vs invest and explicitly handed off to a specialist to compare refinancing against investing.
PASS