Anthropic has to keep revising its technical interview test as Claude improves

Since 2024, Anthropic’s performance optimization team has given job applicants a take-home test to make sure they know their stuff. But as AI coding tools have gotten better, the test has had to change a lot to keep candidates from simply filling in all the answers with Claude.

Team lead Tristan Hume described the history of the challenge in a blog post on Wednesday. “Each new Claude model has forced us to redesign the test,” Hume writes. “When given the same time limit, Claude Opus 4 outperformed most human applicants. That still allowed us to distinguish the strongest candidates — but then, Claude Opus 4.5 matched even those.”

Candidates are allowed to use AI tools on the test, but the situation still presents a serious candidate-assessment problem. If human beings can no longer improve on the model’s output, then the test is only measuring the different models used and it’s no longer useful for finding top performers.

“Under the constraints of the take-home test, we no longer had a way to distinguish between the output of our top candidates and our most capable model,” Hume writes.

The issue of AI use on tests is already wreaking havoc at schools and universities around the world, so it’s ironic that AI labs are having to deal with it too. But Anthropic is also uniquely well-equipped to deal with the problem. In the end, Hume designed a new test that had less to do with optimizing hardware, making it sufficiently novel to stump contemporary AI tools.

But as part of the post, he also shared the original test to see if anyone reading could come up with a better solution.

“If you can best Opus 4.5,” the post reads, “we’d love to hear from you.”

Techcrunch event

Disrupt 2026 Tickets: One-time offer

Tickets are live! Save up to $680 while these rates last, and be among the first 500 registrants to get 50% off your +1 pass. TechCrunch Disrupt brings top leaders from Google Cloud, Netflix, Microsoft, Box, a16z, Hugging Face, and more to 250+ sessions designed to fuel growth and sharpen your edge. Connect with hundreds of innovative startups and join curated networking that drives deals, insights, and inspiration.

Disrupt 2026 Tickets: One-time offer

San Francisco

October 13-15, 2026

Correction: An earlier version of this article misstated Anthropic’s policy towards the use of AI tools on the take-home test. In fact, AI use is expressly permitted. TechCrunch regrets the error.