New models are released every few weeks, each claiming to be better, faster and cheaper. Public benchmarks and leaderboards cannot tell you whether a new model, prompt or retrieval strategy will improve your application or quietly break it. Most teams still answer that question by manually reviewing a handful of outputs.
This hands-on workshop replaces these spot checks with a repeatable evaluation practice. We start with a simple framework that applies to any evaluation: a dataset, an evaluator, the task and how you interpret the results. We then build evaluations for real-world LLM applications step by step, moving from a single prompt to a full RAG pipeline and, finally, a tool-calling agent.
Along the way, we explore how to design reliable LLM-as-a-judge evaluators, account for evaluator noise and integrate evaluation into production through backtesting and online monitoring. We will use LangSmith, but the concepts and practices apply across different tools and frameworks.
What We Will Cover
Evaluation basics: Datasets, evaluators, tasks and metrics, and the role of public benchmarks such as HumanEval and Chatbot Arena.
Building datasets: Creating hand-written ‘golden’ datasets, converting production logs into test cases, versioning and splitting datasets by version or category.
Choosing evaluators: Code checks and unit tests, LLM-as-a-judge evaluations with and without reference answers, custom scoring rubrics and pairwise comparisons for open-ended outputs.
Evaluating RAG: Answer correctness, hallucination detection (checking whether answers are grounded in retrieved documents), retrieval relevance and evaluating intermediate steps directly from traces.
Evaluating agents: Assessing final answers, individual tool choices and the complete sequence of tool calls, including identifying hallucinated tool calls and unnecessary steps.
Comparing versions: A/B testing, regression testing against a baseline, assessing whether a model upgrade is safe in terms of quality, latency and cost, and using repeated evaluations to distinguish genuine improvements from noise.
Evaluations in production: Backtesting new versions against real traffic, using online evaluators to detect hallucinations and implementing input guardrails for PII and toxicity.
You will leave with a practical evaluation playbook that you can use the next time you change a model, prompt or retrieval setup. You will also get working notebooks for RAG and agent evaluation that you can adapt to your own projects.
Who Is It For?
Engineers building LLM-powered products who are comfortable with Python and want an evidence-based approach to iterating and shipping their applications.
Agenda
09:30 AM – 10:00 AM – Attendance
10:00 AM – 04:00 PM – Workshop
01:00 PM- 02:00 PM – Lunch
