Promises are cheap

2025-06-07 · Source: Marcus on AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Novice, quick

Summary

The article critiques the trend of tech CEOs, including Microsoft's AI CEO Mustafa Suleyman and OpenAI's Sam Altman, making ambitious, often unfulfilled predictions about AI capabilities, drawing parallels to Elon Musk's past promises regarding robotaxis. It highlights the persistent issues with Large Language Model (LLM) hallucinations, noting an 8x increase in documented cases involving lawyers in less than a year, and pervasive reasoning troubles identified by Caltech and Stanford researchers. The piece also recalls Geoff Hinton's 2016 prediction that deep learning would surpass radiologists within five years, a claim that has not materialized, as the number of radiologists continues to grow. The author argues that these "cheap promises" are a tactic to generate hype, benefiting companies like Tesla, and criticizes media outlets like the Financial Times for amplifying these predictions without sufficient skepticism or independent verification.

Key takeaway

For Directors of AI/ML evaluating new technologies, critically assess grand claims from tech CEOs by examining historical prediction accuracy and seeking independent verification. Do not base strategic decisions solely on hype cycles, especially concerning critical applications like legal or medical AI where issues like LLM hallucinations remain significant. Prioritize solutions with demonstrated real-world performance over aspirational timelines.

Key insights

Unfulfilled AI predictions from tech CEOs generate hype but often lack real-world validation.

Principles

• Hype often precedes actual AI capability.
• Media skepticism is crucial for AI predictions.

In practice

• Verify AI claims with independent data.
• Track CEO prediction accuracy over time.

Topics

• LLM Hallucinations
• AI Hype Cycle
• Unfulfilled AI Predictions
• AI Reasoning
• Media Scrutiny

Best for: VP of Engineering/Data, Director of AI/ML, Executive, CTO, AI Product Manager, Tech Journalist

Related on AIssential

• The AI Answer You Trusted Most Was Probably the Wrong One — AI models are more confident when incorrect, a byproduct of training that rewards human preference for decisive answers.
• In a sea of hype, here are the AI ‘nothingburgers’ you don’t hear about — The AI industry frequently overhypes capabilities, leading to "nothingburgers" and "slop" that fail to deliver real-world value.
• Am I completely insane for thinking AI is mid — Current LLMs excel at text prediction but fundamentally lack robust reasoning and intuition, leading to limitations and "hallucinations."
• Why ChatGPT can’t be trusted with breaking news — Pure LLMs struggle with breaking news due to outdated training data and inherent reasoning limitations.
• AI Gets Rewarded For Lying To You — AI systems based on next token prediction inherently "hallucinate" all outputs due to a lack of meta-cognition.
• LLM Evals: Basics — LLM evaluation is crucial because AI systems are probabilistic, unlike deterministic traditional software.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Marcus on AI.