Claude Got So Good It Broke Anthropic’s Hiring Test

Anthropic’s own AI model just created an awkward problem. Their technical interview test stopped working because Claude keeps getting too smart.

Since 2024, the company’s performance optimization team used a take-home coding challenge to screen job candidates. But each new Claude release forces them to redesign the whole thing. Now they’re stuck in a weird arms race against their own product.

The Test Couldn’t Keep Up

Team lead Tristan Hume laid out the timeline in a recent blog post. First, Claude Opus 4 beat most human applicants when given the same time limit. That was manageable since top candidates still stood out.

Then Claude Opus 4.5 arrived. It matched even the strongest performers.

So the test stopped measuring candidate skill. Instead, it just measured which AI tool people used. That’s useless for finding top talent.

“We no longer had a way to distinguish between the output of our top candidates and our most capable model,” Hume wrote. When humans can’t improve on the AI’s answers, the evaluation breaks down completely.

AI Use Is Allowed, Which Makes It Worse

Here’s the twist. Anthropic actually permits candidates to use AI tools during the test. They’re not trying to ban Claude or catch cheaters.

But that policy creates a fundamental assessment problem. If the best possible answer comes from Claude, then every candidate using Claude gets the best possible answer. Skill differences disappear.

Schools and universities face this exact nightmare right now. Students submit AI-generated work that teachers can’t reliably detect or evaluate. But it’s particularly ironic when an AI lab struggles with the same issue.

Plus, Anthropic has inside access to Claude’s capabilities. So they should theoretically see this coming better than anyone.

The Hardware Pivot

Hume eventually solved it by designing a completely different test. The new version focuses less on hardware optimization, making it novel enough to stump current AI models.

But he also published the original challenge as an open invitation. If anyone reading can beat Claude Opus 4.5 on the test, Anthropic wants to hear from them.

That’s a clever recruiting move. It turns the problem into a filter for exactly the kind of creative thinkers they want to hire. People who can outsmart state-of-the-art AI probably have valuable skills.

What This Says About AI Progress

The bigger story here is the pace of capability improvement. Anthropic had to redesign this test multiple times in roughly two years. Each major Claude release broke the previous version.

That’s a compression of timelines nobody predicted. Most technical assessments stay relevant for years or decades. But AI coding tools improved fast enough to obsolete a professional hiring test in months.

Other companies probably face similar challenges but haven’t talked about it publicly. If Claude can ace Anthropic’s own technical test, it can probably handle most coding interviews at most companies.

So what happens when AI assistants can generate top-tier interview answers for any candidate? Traditional technical screening breaks down. Companies need new ways to evaluate human judgment, creativity, and problem-solving that AI can’t yet replicate.

The Irony Runs Deep

Anthropic built a product so effective that it undermined their own hiring process. That’s simultaneously impressive and problematic.

It also highlights a broader tension. AI tools should make knowledge work easier and more productive. But when they collapse skill differences entirely, how do you identify exceptional talent?

The answer probably isn’t banning AI from the process. Anthropic made the right call by allowing tool use. Real software engineers use AI coding assistants daily now. Testing them without those tools would be artificial and misleading.

Instead, evaluations need to shift toward skills AI can’t easily replicate. Architecture decisions. Trade-off analysis. Communication. Leadership. The parts of engineering that still require human judgment.

But that’s harder to test in a take-home challenge. It requires longer, more expensive evaluation processes. Which creates its own scaling problems for companies trying to hire quickly.

Anthropic found one clever solution for their specific case. But the underlying problem isn’t going away. As AI capabilities improve, every company will face similar assessment challenges across all roles.

The test that worked last year won’t work this year. And next year’s models will break whatever we design next.