What is Humanity’s Last Exam

The pace of AI progress is nothing short of staggering. Benchmarks once considered unassailable—like ARC-AGI and FrontierMath—are now falling. ARC-AGI has been beaten, and FrontierMath has reached a modest 32% accuracy, signaling that AI models are finally starting to tackle research-level math problems. Yet, while these improvements are impressive on paper, they spark a heated debate: Do these metrics truly reflect the advancement of artificial intelligence, or are we witnessing a dangerous oversimplification of what it means to be “intelligent”?

Breaking Down the Benchmarks

For years, benchmarks have served as the yardsticks by which we measure AI progress. ARC-AGI, once a seemingly insurmountable benchmark, has now been overcome, and FrontierMath is showing modest improvement with its current 32% accuracy. These numbers indicate that modern AI models are beginning to handle tasks that were once the sole province of human experts in mathematics and other research disciplines.

But the story does not end there. Last week, a new benchmark called “Humanity’s Last Exam” was introduced. Designed as the final closed-ended academic test of human knowledge, it spans a wide range of subjects—from classics to ecology—and involves questions that even seasoned experts would find challenging. Early indications suggest that this benchmark is saturating at a rapid pace. In simple terms, AI models are starting to answer these tough questions more frequently than expected, a fact that both excites and alarms the community.

The Promise and Pitfalls of Hard Benchmarks

Proponents argue that as AI models achieve higher scores on these rigorous tests, we are witnessing a leap toward expert-level understanding and reasoning. For instance, one enthusiastic contributor noted,