Break It Before Users Do: AI Reliability Lab

09:00 – 10:15

Find out how often it is actually wrong

why demos lie, eval datasets from real usage, baseline runs

Takeaway Your own application's real accuracy measured for the first time: a number, not an impression, and usually a humbling one.

10:30 – 11:45

Score answers without a human reading each one

exact checks vs model graders, rubric design, when graders lie

Takeaway An automatic scorer that agrees with your own judgment, validated against answers you graded by hand first.

12:00 – 13:00

Evaluate retrieval before blaming the model

retrieval metrics, golden documents, fixing the right layer

Takeaway A wrong answer diagnosed to retrieval, not generation, proved by the fix landing in the retriever and the score moving.

14:00 – 15:15

Evaluate what an agent did, not just what it said

trajectory evaluation, tool-choice correctness, stopping behaviour

Takeaway An agent scored on its decisions (right tool, right order, knew when to stop) across twenty recorded runs.

15:30 – 17:00

Build the failure catalogue

failure taxonomy, adversarial cases, the regression set

Takeaway Your system broken deliberately in ten distinct ways, each failure named, captured and added to the regression set.

09:00 – 10:15

Guard the inputs

input validation, prompt injection, scope refusal

Takeaway An injection attempt that worked yesterday caught at the gate today, and the off-topic request politely refused.

10:30 – 11:45

Guard the outputs

output schemas, unsafe content checks, grounding enforcement

Takeaway An ungrounded claim stopped before the user saw it: the answer forced back to evidence or withheld.

12:00 – 13:00

Trace every request end to end

tracing spans, token and cost attribution, latency budgets

Takeaway One slow, expensive conversation dissected span by span, the cost attributed to the exact call that earned it.

14:00 – 15:15

Put cost and latency on a budget

cost ceilings, latency targets, degrading deliberately

Takeaway A runaway conversation hitting a ceiling you set, and degrading to a cheaper path instead of a bigger bill.

15:30 – 17:00

Gate the pipeline and defend the numbers

evals in CI, release criteria, the reliability review

Takeaway A deliberate quality regression blocked by your own CI gate, and your system's reliability defended to the room in numbers.