Deepmind's research AI occasionally solves what humans can't and mostly gets everything else wrong

2026-02-12 · Source: The Decoder · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Research Methodology & Innovation · Depth: Advanced, medium

Summary

Google Deepmind's AI agent Aletheia, built on Gemini Deep Think, has demonstrated significant capabilities in mathematical research, including independently writing a math paper, disproving a decade-old conjecture, and identifying a cryptography error. However, a systematic evaluation across 700 open math problems revealed that only 6.5 percent of its answers were genuinely useful, with many being fundamentally wrong or trivializing the question. The system employs a three-agent architecture for proposing, checking, and revising solutions, and uses web browsing to verify sources, though it still misrepresents cited content. Deepmind also provides guidelines for scientists to effectively collaborate with AI, treating it as a capable but error-prone junior researcher, and proposes a rating system to classify AI involvement and scientific significance.

Key takeaway

For AI Researchers evaluating the utility of advanced AI agents in scientific discovery, you should adopt a collaborative framework that treats the AI as a junior researcher. Focus on breaking down complex problems into manageable, verifiable sub-tasks and employ balanced prompting to mitigate the AI's tendency to "specification game." Your ability to verify AI outputs remains critical, as the model's confidence does not guarantee correctness, and a peer review crisis could emerge from accelerated paper production.

Key insights

AI can achieve breakthroughs in specific research tasks but requires careful human oversight and structured collaboration.

Principles

• AI excels at connecting distant fields.
• AI often trivializes complex problems.
• AI confidence does not equal accuracy.

Method

Aletheia uses a three-agent system (proposer, checker, reviser) in a loop, augmented with web browsing for source verification, and can admit when it cannot solve a problem.

In practice

• Break large problems into small, verifiable sub-problems.
• Use "balanced prompting" (e.g., "prove or disprove").
• Feed relevant papers directly as context.

Topics

• DeepMind Aletheia
• AI Mathematical Research
• Human-AI Collaboration
• Multi-Agent Systems
• AI Limitations

Code references

• google-deepmind/superhuman
• teorth/erdosproblems

Best for: AI Researcher, AI Scientist, Research Scientist

Related on AIssential

• TAI #192: AI Enters the Scientific Discovery Loop — AI models are transitioning from tools to active participants in scientific discovery, particularly when coupled with robust verification.
• Solutions, challenges and rising tensions in AI and mathematics — AI is rapidly transforming mathematical discovery and problem-solving, prompting critical ethical discussions on human roles.
• Google DeepMind’s powerful AI co-mathematician — Agentic AI systems, like DeepMind's co-mathematician, significantly advance complex problem-solving by orchestrating specialized AI agents.
• Mar 23, 2026ScienceVibe physics: The AI grad student — AI can significantly accelerate expert-guided theoretical physics research, but requires rigorous human oversight.
• Welcoming AI as a New Colleague: How Should We Evaluate AI for Science? — AI's rapid integration into science necessitates robust evaluation frameworks, emphasizing human-AI collaboration to leverage complementary strengths.
• OpenAI just SOLVED MATH.... — AI's breakthrough stemmed from its ability to connect disparate mathematical fields, revealing novel solutions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Decoder.