Humans vs LLMs: Humanity's Last Exam

Can you beat LLMs on Humanity's Last Exam? (Probably not.) A 10-question multiple-choice quiz built from real HLE questions, with a simulated comparison against real frontier-model benchmark accuracy and a persistent human leaderboard. Hosted on Cloudflare Workers, leaderboard maintained with durable objects.

Check it out on Entire here.

Entire captured the prompts, decisions, and reasoning behind this Opus 4.8-generated code and attaching it to my commits in git so the "why" behind the code won't get lost. Get on the waitlist now!

Local development

Tests

Deployment

How questions are served

Questions are baked in at build time and served from the Worker, so the deployed app never calls Hugging Face or agi.safe.ai at req time.

1. Curated offline into src/questions.json (140 plain-text multiple-choice questions) by scripts/build-questions.py — see Regenerating the question bank.
2. Bundled into the Worker: src/questions.ts imports questions.json directly, so the bank ships inside the Worker bundle — no database, no external fetch.
3. Served to the browser through a small API (src/index.ts), with the correct answers kept server-side the whole time:
   • GET /api/quiz/new — picks a category-diverse pool (QUIZ_POOL_SIZE, 30), shuffles each question's choices and re-letters them A, B, C…, strips the correct answers, and returns the questions plus an encrypted answer-key token (AES-GCM, QUIZ_TOKEN_KEY). The answers never reach the client.
   • POST /api/quiz/answer — { token, id, answer }; the Worker decrypts the token to grade that one question and returns { correct, correctAnswer } for instant feedback.
   • POST /api/quiz/submit — grades the questions actually answered against the token and returns an HMAC-signed result token (RESULT_TOKEN_SECRET) used for the leaderboard and share image, so scores can't be forged client-side.
   • GET /api/questions — the full 140-question bank (answers stripped) that powers the /questions browse page.

Regenerating the question bank

The question bank in src/questions.json is curated offline from the gated cais/hle Hugging Face dataset: the deployed Worker never calls Hugging Face or agi.safe.ai at request time. To refresh it:

1. Accept the dataset terms at https://huggingface.co/datasets/cais/hle
2. hf auth login (or set up a Hugging Face token locally)
3. pip3 install -U datasets
4. python3 scripts/build-questions.py

Data source & citation

All quiz questions, answer choices, and correct answers are taken verbatim from the Humanity's Last Exam (HLE) benchmark by the Center for AI Safety & Scale AI (MIT license). The build script only filters to text-only multiple-choice questions and splits each row's question field into a stem + parsed choices — the wording, options, and answers are unchanged. Each question retains its HLE id and author_name for attribution (surfaced per-question in the results view).

To cite the benchmark:

To cite an individual question, reference its HLE id (and author_name where present) together with the dataset above — e.g. "HLE question 66fcfe789608881c6aecde88, contributed by Mohammad M, Humanity's Last Exam (cais/hle)."

Refreshing model comparison figures

src/models.ts stores each model's published overall HLE accuracy, sourced from Scale AI's official leaderboard (https://labs.scale.com/leaderboard/humanitys_last_exam). Update the accuracy values and the AS_OF date in that file periodically as new scores are published.