Hamel's Blog
Notes on applied AI engineering, machine learning, and data science.
- Posts
- 83 from 16 Jan 2023 to 25 Oct 2026
- Posting
- weekly (29 in the last year)
- Kind
- personal blog
- Feed
- https://hamel.dev/index.xml
2026 28
- Q: What does a good eval dashboard look like for sharing with stakeholders? – Hamel’s Blog
- How To Use Decision Models for Evals
- Hamel’s Blog
- Claude’s new auto eval tool
- Q: How often should I run my evals? – Hamel’s Blog
- Q: How do I evaluate outputs in a language I don’t speak? – Hamel’s Blog
- Q: What should I do when I can’t get my LLM judge to agree with human reviewers? – Hamel’s Blog
- Q: What should an internal eval platform standardize across teams? – Hamel’s Blog
- Q: What model or LLM should I use to build automated evals? – Hamel’s Blog
- Q: How should I evaluate a coding agent? – Hamel’s Blog
- Q: How do I know if I can trust my automated eval? – Hamel’s Blog
- Q: How do I combine my evals into a single metric? – Hamel’s Blog
- Q: Can I use Jev for evals? – Hamel’s Blog
- AI Evals: Everything You Need to Know
- Q: Should I record problems that aren’t the model’s fault? – Hamel’s Blog
- Q: Can I help with evals if I’m not a domain expert? – Hamel’s Blog
- Q: How can I make AI outputs easier for people to evaluate? – Hamel’s Blog
- Q: What should I do when my “gold” eval dataset becomes stale? – Hamel’s Blog
- Q: How much context should I give a LLM judge? – Hamel’s Blog
- Q: How can I do evals when traces contain sensitive data? – Hamel’s Blog
- Q: Do I need a reference answer or rubric before annotating data? – Hamel’s Blog
- Q: How do you review a trace that is really large? – Hamel’s Blog
- Q: How many examples do I need for an eval? – Hamel’s Blog
- “It’s Hard to Eval” Is a Product Smell
- Hamel’s Blog
- The Revenge of the Data Scientist
- Evals Skills for Coding Agents
- Why I Stopped Using nbdev
2025 44
- Selecting The Right AI Evals Tool
- Q: How do we evaluate a model’s ability to express uncertainty or “know what it doesn’t know”? – Hamel’s Blog
- Q: How do I make the case for investing in evaluations to my team? – Hamel’s Blog
- Q: How should I version and manage prompts? – Hamel’s Blog
- Q: How do I surface problematic traces for review beyond user feedback? – Hamel’s Blog
- Q: Are there scenarios where synthetic data may not be reliable? – Hamel’s Blog
- Q: How do I evaluate complex multi-step workflows? – Hamel’s Blog
- Q: How do I evaluate sessions with human handoffs? – Hamel’s Blog
- Q: How often should I re-run error analysis on my production system? – Hamel’s Blog
- Q: Should I practice eval-driven development? – Hamel’s Blog
- Q: Should product managers and engineers collaborate on error analysis? How? – Hamel’s Blog
- Q: What is a trace? – Hamel’s Blog
- Q: What should go in the system prompt vs. the user prompt? – Hamel’s Blog
- Q: Will today’s evaluation methods still be relevant in 5-10 years given how fast AI is changing? – Hamel’s Blog
- Q: Should I outsource annotation & labeling to a third party? – Hamel’s Blog
- Q: What parts of evals can be automated with LLMs? – Hamel’s Blog
- Q: Can my evaluators also be used to automatically fix or correct outputs in production? – Hamel’s Blog
- Q: Should I stop writing prompts manually in favor of automated tools? – Hamel’s Blog
- Q: Should I use “ready-to-use” evaluation metrics? – Hamel’s Blog
- Q: How can I efficiently sample production traces for review? – Hamel’s Blog
- Q: Are similarity metrics (BERTScore, ROUGE, etc.) useful for evaluating LLM outputs? – Hamel’s Blog
- Q: What are AI Evals? – Hamel’s Blog
- Q: What’s your favorite eval vendor? – Hamel’s Blog
- Q: What’s a minimum viable evaluation setup? – Hamel’s Blog
- Q: How do I evaluate agentic workflows? – Hamel’s Blog
- Q: How are evaluations used differently in CI/CD vs. monitoring production? – Hamel’s Blog
- Q: What’s the difference between guardrails & evaluators? – Hamel’s Blog
- Q: Why is “error analysis” so important in AI evals, and how is it performed? – Hamel’s Blog
- Q: How much of my development budget should I allocate to evals? – Hamel’s Blog
- Q: What makes a good custom interface for reviewing LLM outputs? – Hamel’s Blog
- Q: How should I approach evaluating my RAG system? – Hamel’s Blog
- Q: How do I choose the right chunk size for my document processing tasks? – Hamel’s Blog
- Q: How do I approach evaluation when my system handles diverse user queries? – Hamel’s Blog
- Q: What is the best approach for generating synthetic data? – Hamel’s Blog
- Q: What gaps in eval tooling should I be prepared to fill myself? – Hamel’s Blog
- Q: How many people should annotate my LLM outputs? – Hamel’s Blog
- Q: Should I build automated evaluators for every failure mode I find? – Hamel’s Blog
- Q: Should I build a custom annotation tool or use something off-the-shelf? – Hamel’s Blog
- Q: How do I debug multi-turn conversation traces? – Hamel’s Blog
- Q: Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)? – Hamel’s Blog
- Q: How much time should I spend on model selection? – Hamel’s Blog
- Q: Can I use the same model for both the main task and evaluation? – Hamel’s Blog
- Q: Is RAG dead? – Hamel’s Blog
- A Field Guide to Rapidly Improving AI Products
2024 9
- Building an Audience Through Technical Writing: Strategies and Mistakes
- Using LLM-as-a-Judge For Evaluation: A Complete Guide
- An Open Course on LLMs, Led by Practitioners
- Debugging AI With Adversarial Validation
- Your AI Product Needs Evals
- Is Fine-Tuning Still Valuable?
- Fuck You, Show Me The Prompt.
- How To Debug Axolotl
- Dokku: my favorite personal serverless platform