Find the AI evals that actually catch failures.
40 curated evals, benchmarks and rubrics. Each one links to its original source, says who made it and explains what it tests in plain English. Practitioner reviews are coming next.
SWE-bench
Gives a model a real GitHub issue from an open-source Python project and checks whether its code change makes the project's tests pass.
CodingHumanEval
A classic set of hand-written Python function problems, each checked by running hidden tests. Useful as a baseline, though top models have largely mastered it.
CodingLiveCodeBench
Keeps collecting new contest problems over time, so you can test a model only on problems released after its training cutoff.
CodingAider leaderboards (Polyglot)
Measures how well a model edits code in many programming languages inside the Aider coding assistant, including whether it follows the edit format.
CodingBigCodeBench
Programming tasks that require calling many real libraries correctly, closer to everyday developer work than puzzle-style problems.
Agents & tool useGAIA
Questions that are easy for a person with a browser and tools but hard for AI assistants, each with one short, checkable answer.
Customer supportτ-bench (tau-bench)
Simulates a customer talking to an AI agent in domains like retail and airlines. The agent must follow company policy and use tools; the final database state is checked.
Agents & tool useWebArena
Self-hosted copies of realistic websites (shop, forum, code host and more) where an agent must complete tasks by clicking and typing.
Agents & tool useOSWorld
Puts an agent in a real computer environment and asks it to do tasks across desktop apps, checked by scripts.
Agents & tool useTerminal-Bench
Tasks an agent must finish inside a command-line terminal, such as setting up, debugging or building software.
Agents & tool useBrowseComp
1,266 very hard-to-find facts with short answers that are easy to verify, built to test persistent web-browsing agents.
Writing & instruction followingEQ-Bench Creative Writing
Scores creative writing from models with an LLM judge using a detailed rubric, aiming to capture quality rather than just correctness.
Writing & instruction followingAlpacaEval
Compares a model's answers against a reference model's answers using an automatic LLM judge, with a correction for judges that prefer longer answers.
Writing & instruction followingMT-Bench (LLM-as-a-judge paper)
The paper that popularised using a strong model as a judge, with multi-turn questions and an analysis of where judges agree with humans and where they are biased.
Writing & instruction followingIFEval
Prompts with instructions a program can check automatically (word counts, formats, keywords), so no judge model is needed.
Safety & factualityHarmBench
A standard framework for red-teaming: it tests how easily attacks can make a model produce harmful content and how well defences hold.
Safety & factualityTruthfulQA
Questions designed so that common misconceptions are tempting; checks whether the model repeats popular falsehoods.
Safety & factualityStrongREJECT
A better way to score jailbreaks: it checks whether the harmful answer is actually useful, not just whether the model refused.
Safety & factualitySimpleQA
4,326 short fact questions with one indisputable answer each; rewards models that answer correctly or admit they don't know instead of hallucinating.
Safety & factualityXSTest
Safe prompts that merely sound dangerous, to catch models that refuse too much.
HealthcareHealthBench
5,000 realistic, multi-turn health conversations, each graded against rubrics written by 262 physicians from 60 countries. Includes harder and consensus-validated subsets.
HealthcareMedQA
Multiple-choice questions from medical licensing exams. Easy to run, but exam-style and widely used, so it reveals little about real clinical conversations.
HealthcareMedHELM
A holistic medical leaderboard covering many real clinical task types, not only exam questions.
HealthcareKARMA-OpenMedEvalKit
An open toolkit for evaluating medical AI with an emphasis on Indian healthcare, covering text, image and audio models and adding new India-focused datasets.
HealthcareSAKHI
A maternal-health benchmark built around a WhatsApp chatbot for Hindi- and Marathi-speaking mothers, with rubric verdicts from Indian doctors to calibrate the LLM judge.
Multilingual & Indian languagesIndicGenBench
Generation tasks such as summarisation, translation and question answering across many Indian languages, including low-resource ones.
Multilingual & Indian languagesMILU
A knowledge benchmark in Indian languages that includes India-specific subjects, not just translated Western exams.
Multilingual & Indian languagesGlobal-MMLU
MMLU improved and translated into many languages, with labels marking which questions are culturally specific.
Multilingual & Indian languagesSamiksha
A community-built pipeline for evaluating health chatbots in Hindi, Kannada and Malayalam, combining human raters with LLM judges.
RAG & retrievalRagas
An open-source library of metrics for RAG apps and agents, such as whether an answer is faithful to the retrieved context.
RAG & retrievalBEIR
A collection of retrieval datasets for testing how well search and embedding models work on domains they were not trained on.
RAG & retrievalMTEB Leaderboard
The standard leaderboard for comparing embedding models on retrieval, classification, clustering and more.
RAG & retrievalCRAG
A question-answering benchmark for RAG with mock web and knowledge-graph search, including questions whose answers change over time.
RAG & retrievalFRAMES
Questions that need facts from several documents combined, testing retrieval, reasoning and factuality together.
Reasoning & knowledgeGPQA
Graduate-level science questions written by experts so that web search doesn't easily help.
Reasoning & knowledgeHumanity's Last Exam
Very hard questions contributed by experts across many fields, made to stay difficult after older benchmarks saturated.
Rubric libraries & frameworksInspect Evals
A large, actively maintained collection of ready-to-run evals implemented in the Inspect framework.
Rubric libraries & frameworksOpenAI Evals
A framework and open registry of evals, including templates for model-graded evaluation.
Rubric libraries & frameworksLLMJury rubric library
Paste-ready judge rubrics for questions like 'did the support reply resolve the request?', each with score meanings, examples and known failure modes. Made by a vendor.
Rubric libraries & frameworksPrompt Assay judge templates
Copy-paste YAML judge templates covering basics like relevance and tone plus RAG, code, summarisation and agent traces. Made by a vendor.