Clicks Are a Vanity Metric: Ranking Ads for What Actually Converts

2026-07-22 · Source: HackerNoon · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Advanced, medium

Summary

An ads ranking model rebuild, led by an analyst over most of 2025, shifted optimization from click-through rate (CTR) to actual conversions, recognizing CTR as a "vanity metric" that often fails to deliver advertiser value. The project, drawing on experience from DoorDash, YouTube Ads, and Google Assistant NLP, implemented a multi-task, multi-label model. This architecture uses a single shared representation for user and context, with distinct prediction heads for clicks and conversions, allowing ranking based on expected value (P(click) × P(purchase | click) × purchase value). This approach, a variant of Google's 2018 multi-gate mixture-of-experts, resulted in a 7% increase in CTR and a 2.1% increase in conversions. Key challenges included handling sparse and delayed conversion labels, effective task weighting to prevent the abundant click signal from dominating, and ensuring compatibility with existing auction and budgeting systems. The author also highlights retrieval as a subsequent bottleneck, advocating for semantic retrieval using high-dimensional embeddings to match user intent.

Key takeaway

For AI Engineers building ad ranking systems, prioritize optimizing for actual conversions over click-through rate. Your models should predict expected value, not just attention, to ensure advertiser ROI. Implement multi-task learning to jointly model clicks and conversions, accounting for sparse and delayed conversion data. Also, integrate retrieval and ranking as a unified system, as a strong ranker cannot compensate for a poor candidate set.

Key insights

Optimizing ad ranking for clicks alone is a "vanity metric" that can lose advertiser money; focus on conversions.

Principles

• Expected value is P(click) × P(purchase | click) × purchase value.
• Multi-task models with shared layers improve sparse conversion prediction.
• Retrieval and ranking form a single system.

Method

Rebuild ad ranking using a multi-task, multi-label model with shared user/context representation and separate prediction heads for clicks and conversions, then rank on their combination.

In practice

• Implement multi-task learning for ad ranking to predict both clicks and conversions.
• Address delayed feedback for conversion labels during model training.
• Utilize semantic retrieval with embeddings to improve ad candidate sets.

Topics

• Ad Ranking
• Multi-task Learning
• Conversion Optimization
• Click-Through Rate
• Delayed Feedback
• Semantic Retrieval

Best for: Machine Learning Engineer, AI Engineer, MLOps Engineer

Related on AIssential

• Stop tracking hypothetical prompts: how to build a prompt set from real buyer queries — Accurate AI visibility tracking requires prompt sets built from real buyer queries, not speculation.
• The Click That Funded Everything — Google's accidental click-based payment system for content is being dismantled by its own AI Overviews.
• From Clicks to Conversions: Architecting Shopping Conversion Candidate Generation at Pinterest — Optimizing for sparse offsite conversions requires multi-task learning, robust data design, and parallel architectural components.
• Enhancing Ad Relevance: Integrating Real-Time Context into Sequential Recommender Models — Integrating real-time context into sequential recommenders significantly boosts ad relevance and business performance.
• 12 rules that turn AI-generated thumbnails from $18 Fiverr gigs into $500 deliverables. — Applied click psychology, not AI tools, drives high-value, high-performing YouTube thumbnails.
• I flipped one toggle and 77% of the AI’s product recommendations changed — Enabling web search causes a substantial shift in AI product recommendations, largely overriding the model's internal memory.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by HackerNoon.