Break It Before Users Do — AI Reliability Lab

09:00 – 10:15

Find out how often it is actually wrong

—why demos lie, eval datasets from real usage, baseline runs

—Takeaway Your own application's real accuracy measured for the first time — a number, not an impression, and usually a humbling one.

10:30 – 11:45

Score answers without a human reading each one

—exact checks vs model graders, rubric design, when graders lie

—Takeaway An automatic scorer that agrees with your own judgment — validated against answers you graded by hand first.

12:00 – 13:00

Evaluate retrieval before blaming the model

—retrieval metrics, golden documents, fixing the right layer

—Takeaway A wrong answer diagnosed to retrieval, not generation — proved by the fix landing in the retriever and the score moving.

14:00 – 15:15

Evaluate what an agent did, not just what it said

—trajectory evaluation, tool-choice correctness, stopping behaviour

—Takeaway An agent scored on its decisions — right tool, right order, knew when to stop — across twenty recorded runs.

15:30 – 17:00

Build the failure catalogue

—failure taxonomy, adversarial cases, the regression set

—Takeaway Your system broken deliberately in ten distinct ways — each failure named, captured and added to the regression set.

09:00 – 10:15

Guard the inputs

—input validation, prompt injection, scope refusal

—Takeaway An injection attempt that worked yesterday caught at the gate today — and the off-topic request politely refused.

10:30 – 11:45

Guard the outputs

—output schemas, unsafe content checks, grounding enforcement

—Takeaway An ungrounded claim stopped before the user saw it — the answer forced back to evidence or withheld.

12:00 – 13:00

Trace every request end to end

—tracing spans, token and cost attribution, latency budgets

—Takeaway One slow, expensive conversation dissected span by span — the cost attributed to the exact call that earned it.

14:00 – 15:15

Put cost and latency on a budget

—cost ceilings, latency targets, degrading deliberately

—Takeaway A runaway conversation hitting a ceiling you set — and degrading to a cheaper path instead of a bigger bill.

15:30 – 17:00

Gate the pipeline and defend the numbers

—evals in CI, release criteria, the reliability review

—Takeaway A deliberate quality regression blocked by your own CI gate — and your system's reliability defended to the room in numbers.