Humanity’s Last Exam (HLE) is a demanding AI benchmark created by the Center for AI Safety (CAIS) and Scale AI to measure how well advanced AI models handle difficult, expert-level academic questions. Rather than testing basic knowledge, HLE focuses on problems that require specialized knowledge, reasoning, and careful analysis across mathematics, science, medicine, computer science, humanities, and other fields. The current benchmark contains 2,500 questions spanning more than 100 subjects.
The benchmark was created because older evaluations were becoming too easy for frontier models. For example, the HLE team noted that leading language models were reaching very high scores on established tests such as MMLU, making those benchmarks less useful for distinguishing the strongest systems. HLE was designed to push evaluation closer to the frontier of human academic expertise.
Why Was Humanity’s Last Exam Created?
AI benchmarks are essentially standardized tests for artificial intelligence. They give researchers a common way to compare models and measure progress over time. The problem is that a benchmark can lose value once models become very good at it.
This is known as benchmark saturation. If several advanced models score close to 100%, the test tells researchers little about which model has stronger reasoning or specialized knowledge. HLE was created as a response to that problem, using substantially harder questions intended to challenge state-of-the-art systems.
The name is deliberately provocative. It does not mean that AI has literally reached the end of human knowledge or that HLE will permanently be the final AI evaluation. The researchers themselves emphasize that even if models eventually perform extremely well on HLE, the benchmark would still measure only a particular class of capabilities.
How Difficult Is Humanity’s Last Exam?
The difficulty is one of HLE’s defining characteristics. Questions were contributed and reviewed by subject-matter experts, with the goal of creating problems that go beyond routine textbook exercises.
The benchmark covers more than 100 academic subjects, including areas such as mathematics, physics, chemistry, biology, medicine, computer science, social sciences, and the humanities. It combines multiple-choice and short-answer questions, while some questions require the model to interpret images, diagrams, or other visual information.
Importantly, HLE is not simply a giant collection of obscure trivia. Its purpose is to test whether an AI system can produce a correct, verifiable answer when solving difficult academic problems. That makes reasoning quality particularly important.
How Does Humanity’s Last Exam Work?
HLE is a closed-ended benchmark. Each question has a known answer that can be evaluated, rather than requiring a subjective judgment about whether an essay or creative response is good.
A typical evaluation works roughly like this:
- The AI model receives an HLE question.
- The model analyzes the question and produces an answer.
- The answer is compared with the benchmark’s reference answer.
- The evaluation calculates whether the response is correct.
- Researchers can then compare accuracy and other evaluation metrics across models.
The dataset also includes multimodal questions, meaning that text alone is not always sufficient. A model may need to interpret a figure, diagram, chart, or other visual information as part of solving the problem.
This is an important distinction from a simple web-search test. HLE was specifically designed so that questions are not supposed to be easily solved by retrieving an answer from the internet. Scale AI has also described a process for identifying and removing questions that could be too easily searchable.
What Skills Does HLE Actually Measure?
Humanity’s Last Exam measures several capabilities at the same time, although it should not be treated as a complete test of intelligence.
Specialized Knowledge
Some questions require knowledge that is far beyond everyday facts. A model may need to understand a specialized scientific concept, a mathematical method, or a narrow area of academic research before it can even begin solving the problem.
Multi-Step Reasoning
Knowing a relevant fact is often not enough. The model may need to combine several pieces of information, perform calculations, identify relationships, and reach a specific conclusion.
Mathematical Problem Solving
Mathematics provides a useful environment for evaluating reasoning because many problems have precise answers. An incorrect result cannot easily be hidden behind fluent language.
Multimodal Understanding
Some HLE questions involve visual information. This makes the benchmark relevant to multimodal AI systems that can process both text and images rather than only traditional language models.
Answer Calibration
HLE also highlights an important weakness in AI systems: confidence does not always equal correctness. Early HLE evaluations found that advanced models could be highly confident while still producing incorrect answers.
That matters in real-world applications. An AI system that says “I don’t know” can be safer than one that gives a confidently incorrect answer.
How Did Early AI Models Perform?
The initial results demonstrated why the benchmark was created in the first place.
When HLE was introduced in January 2025, frontier models generally achieved single-digit accuracy, and the organizers reported that current models were answering fewer than 10% of the expert questions correctly.
The results were striking because many of the same models could perform extremely well on older academic benchmarks. HLE therefore provided a more difficult testing environment for frontier systems.
However, benchmark scores should always be interpreted in context. A score on HLE is not a direct measurement of general intelligence, human intelligence, or an AI system’s usefulness in every real-world task.
How Have HLE Scores Changed?
One of the most interesting aspects of Humanity’s Last Exam is how quickly benchmark performance can change.
The original benchmark exposed a large gap between frontier AI systems and expert-level performance. But newer models have improved considerably. Scale AI’s current leaderboard, for example, reports substantially higher scores for newer frontier systems than those seen at HLE’s initial release.
This illustrates a broader problem with static benchmarks: once researchers publish a difficult evaluation, AI developers can optimize models, reasoning methods, inference strategies, and tool-use systems that eventually improve performance.
It also explains why the HLE project has evolved. Scale AI and CAIS have introduced HLE-Diamond, a refined 1,000-question subset created through additional cleaning and review. Its purpose is to provide a more carefully refined evaluation of frontier capabilities.
Why Tool Use Changes the Picture
Modern AI evaluation is no longer limited to asking a model a question and waiting for one response.
Advanced systems may use tools such as calculators, code execution, search, retrieval systems, or other software during inference. This creates an important distinction between what the model knows internally and what the complete AI system can accomplish with external tools.
HLE can therefore be evaluated under different conditions. A model working without external tools is being tested differently from an agent that can repeatedly reason, execute code, inspect information, and revise its approach.
Recent research on AI evaluation has also shown that inference-time compute can significantly affect performance on difficult benchmarks, including HLE. In other words, giving a reasoning model more opportunities to work through a problem can change its measured capability.
For this reason, an HLE score is meaningful only when the evaluation protocol is also understood.
What Humanity’s Last Exam Does Not Prove
A high HLE score would be impressive, but it should not be interpreted as proof that an AI system can do everything an expert human can do.
The HLE researchers explicitly distinguish academic benchmark performance from broader capabilities such as autonomous scientific research. HLE primarily evaluates structured, closed-ended questions with verifiable answers. Real research is often open-ended and involves deciding what questions to ask, designing experiments, handling incomplete information, interpreting unexpected results, and determining whether an apparent discovery is actually correct.
There are other limitations as well.
Benchmark Scores Are Protocol-Dependent
Different evaluation setups can produce different results. Model version, reasoning effort, token budget, tool access, prompting, and grading methodology can all influence performance.
Closed-Ended Questions Are Not Real Life
Many professional tasks do not have a single predetermined answer. Doctors, engineers, researchers, lawyers, and business leaders frequently have to make decisions under uncertainty.
Accuracy Is Not the Same as Reliability
A model can achieve a strong average score while still failing badly on particular questions. For high-stakes applications, users need to understand the system’s failure modes rather than relying only on an overall percentage.
Benchmarks Can Become Training Targets
Once a benchmark becomes influential, developers may use it to guide model development. Research has raised concerns that benchmark-driven optimization can make evaluation less effective at measuring genuinely novel generalization.
What Is HLE-Diamond?
HLE-Diamond is a refined subset of Humanity’s Last Exam containing 1,000 questions. Scale AI describes it as the result of an extended cleaning and refinement process involving research communities.
The goal is not simply to make another leaderboard. The refinement addresses the practical difficulty of maintaining a high-quality benchmark as researchers identify problematic, ambiguous, or otherwise unsuitable questions.
HLE-Diamond also demonstrates how AI evaluation is becoming more sophisticated. Instead of assuming that the original dataset is permanently perfect, benchmark developers can review questions, improve the dataset, and create cleaner evaluation sets.
Can You Try Humanity’s Last Exam Yourself?
Yes. The HLE project has made the dataset publicly available, and the official project provides resources for researchers who want to evaluate models. The GitHub repository includes the dataset and evaluation instructions, including an example workflow for running predictions and grading results.
For someone who simply wants to understand AI capability, trying a handful of questions can be more informative than looking at a leaderboard alone.
A useful experiment is to take one difficult question and compare:
- Your own reasoning.
- A standard language model’s answer.
- A reasoning-focused model’s answer.
- The final verified answer.
The goal is not to see which system writes the most convincing explanation. It is to determine whether the reasoning actually leads to a correct result.
Why Humanity’s Last Exam Matters for the Future of AI
HLE matters because AI development needs increasingly difficult ways to measure progress.
If models can eventually solve most HLE questions, researchers will need new evaluations that test capabilities not captured by academic questions. These could involve long-horizon planning, autonomous research, software engineering, scientific experimentation, real-world decision-making, or interaction with complex environments.
That does not make HLE obsolete. Instead, it places it in a larger evaluation ecosystem.
The benchmark is particularly useful because it reminds us that fluent language is not the same thing as deep understanding. An AI can write an excellent explanation and still get the underlying problem wrong. Difficult, verifiable evaluations make that distinction easier to see.
Humanity’s Last Exam vs. Traditional AI Benchmarks
HLE differs from easier academic benchmarks mainly in its intended difficulty and breadth.
| Aspect | Humanity’s Last Exam | Traditional academic benchmarks |
|---|---|---|
| Main goal | Test frontier AI capabilities | Measure general academic knowledge |
| Difficulty | Expert and highly specialized | Often easier for current frontier models |
| Subject coverage | 100+ subjects | Usually narrower or more standardized |
| Question types | Short-answer, multiple-choice, multimodal | Often primarily text-based |
| Evaluation | Verifiable answers | Depends on benchmark |
| Main use | Frontier model evaluation | General model comparison |
The important point is not that HLE replaces every other benchmark. Different evaluations answer different questions. A good AI evaluation strategy uses multiple tests because no single benchmark can capture every useful capability.
Common Misconceptions About Humanity’s Last Exam
Is HLE literally the last exam AI will ever take?
No. The name refers to the ambition to create a final-style closed-ended academic benchmark, not a permanent end to AI evaluation. The researchers explicitly note that HLE is not the last benchmark needed for AI.
Does a high HLE score mean an AI has achieved AGI?
No. HLE measures performance on difficult, structured academic questions. The benchmark’s creators specifically caution that strong HLE performance alone would not establish autonomous research ability or artificial general intelligence.
Is HLE just a test of memorization?
No. The benchmark is designed to require specialized knowledge and reasoning, and many questions are intended to resist simple information retrieval.
Can AI models simply Google the answers?
The benchmark was designed to reduce this problem. Questions that were found to be easily searchable could be removed during the evaluation-development process.
FAQs About Humanity’s Last Exam
What is Humanity’s Last Exam?
Humanity’s Last Exam is an AI benchmark containing 2,500 difficult questions across more than 100 academic subjects, designed to evaluate frontier AI knowledge and reasoning.
Who created Humanity’s Last Exam?
It was developed through a collaboration between the Center for AI Safety and Scale AI, with contributions and reviews from subject-matter experts.
How many questions are in HLE?
The current HLE dataset contains 2,500 questions. A separate refined evaluation called HLE-Diamond contains 1,000 questions.
Is Humanity’s Last Exam harder than MMLU?
It was specifically created because established benchmarks such as MMLU were becoming saturated among frontier models. HLE is intended to test substantially harder, more specialized capabilities.
Does HLE test multimodal AI?
Yes. HLE includes questions that require models to work with visual information in addition to text.
Does a high HLE score prove AGI?
No. HLE measures a specific class of academic knowledge and reasoning and does not independently establish artificial general intelligence.
Conclusion
Humanity’s Last Exam is important because it addresses a genuine problem in AI evaluation: once models become very good at a benchmark, the benchmark can stop telling us much about the frontier. HLE responds by using difficult, expert-created questions across a remarkably broad range of subjects.
Its results should still be interpreted carefully. A model’s score depends on the evaluation setup, available inference compute, tool access, and grading process. More importantly, solving closed-ended academic questions is only one part of intelligence. Real-world expertise also requires judgment, experimentation, communication, adaptability, and the ability to work with uncertainty.
For that reason, HLE is best viewed not as a final verdict on AI intelligence, but as a demanding measuring instrument. It helps researchers see where advanced models are improving, where they continue to fail, and why increasingly rigorous evaluations will remain important as AI capabilities advance.
1 comment on “Humanity’s Last Exam: What It Tests and Why It Matters”