Evals for
AI Engineers
Systematically measuring and improving AI applications
How do you know if your AI application is working? This book shows you how to find its failures and build evaluations that help you improve it.
A practical guide to error analysis, building automated evaluators, and monitoring AI in production. Build a continuous improvement loop that makes your AI more reliable and less costly to run.
We may earn a commission through the book links.
Learn with us
AI Evals for Engineers & PMs
Build and evaluate AI systems with us in our live course. We teach current techniques and continually update the material as AI evolves.
- 8+ hours of live office hours for feedback on your projects.
- Lifetime access to recordings and course materials.
- Unlimited retakes of future cohorts, including office hours.
- Ongoing access to our private Discord community.
Further reading
Choose a guide based on where you are with evals.
| Where are you? | This sounds like me |
|---|---|
| I'm new to evals | I've heard the term, but I'm not sure what evals involve or whether I need them. |
| I don't know what to test | I'm building an AI product, but I haven't figured out which failures to measure or what good performance looks like. |
| I don't trust my eval scores | We have evals, but the scores don't match our judgment of the outputs, or tests pass while users still encounter problems. |
| My product feels too hard to evaluate | Our outputs are subjective, long, or involve many steps. Even a knowledgeable person has trouble deciding whether they're right. |
| Evals take too much time or money | We're spending too much effort reviewing outputs, maintaining tests, or running evaluators. |
Prefer to browse? Read the entire FAQ.