Whose hotel does the AI recommend? An algorithm audit of reputation signals in LLM-assisted hotel selection

2026-06-15 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

An algorithm audit investigated how large language model (LLM) assistants recommend hotels, analyzing twelve open-weight and proprietary models. Using a randomized choice-based conjoint, researchers assessed the impact of guest rating, review volume/recency, management response, chain affiliation, price, eco-certification, and list position. Findings reveal guest rating and price dominate LLM selections, with a top rating increasing selection by 31.6 percentage points and a high price decreasing it by 30.0. LLMs over-weight eco-certification and ignore management response. Crucially, list position, a content-free artifact, causally shifts recommendations, valued at approximately \$12 per night. Stated reasons for recommendations often imperfectly track these revealed weights.

Key takeaway

For hotel marketers optimizing online visibility, prioritize maximizing guest ratings and competitive pricing, as these signals heavily influence LLM recommendations. Be aware that list position also significantly impacts selection, worth about \$12 per night, suggesting a need to understand platform-specific ranking algorithms. Additionally, scrutinize LLM-generated recommendation rationales, as they may not accurately reflect the underlying decision weights, impacting your strategy for generative engine optimization.

Key insights

LLMs prioritize guest ratings and price in hotel recommendations, but also exhibit biases like over-weighting list position.

Principles

• LLM recommendations reproduce human valence-and-price primacy.
• Content-free artifacts can causally influence LLM choices.
• Stated LLM reasoning may not reflect actual decision weights.

Method

A pre-specified algorithm audit used a randomized choice-based conjoint across personas, prompt templates, and twelve LLMs to estimate marginal component effects of hotel signals.

In practice

• Optimize hotel listings for guest rating and price.
• Monitor LLM-generated reasons for recommendation discrepancies.
• Understand list position's causal impact on visibility.

Topics

• Large Language Models
• Algorithm Audit
• Hotel Recommendation
• Reputation Signals
• Conjoint Analysis
• Generative Engine Optimization
• AI Accountability

Best for: Research Scientist, AI Scientist, AI Ethicist, AI Product Manager

Related on AIssential

• A closer look at how large language models trust humans: patterns and biases — LLMs develop human-like trust patterns, influenced by trustworthiness and demographic biases, especially in financial contexts.
• PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing — LLMs diverge significantly from human peer review behavior, necessitating specialized benchmarks for reliable integration.
• Language Models Exhibit Inconsistent Biases Towards Algorithmic Agents and Human Experts — LLMs exhibit inconsistent biases, preferring human experts in stated trust but algorithms in revealed preference tasks.
• Brands without review profiles appear in only 1% of AI answers — Visible customer trust signals, particularly reviews, are crucial for brand visibility in AI-generated search results.
• A Foundation Model for Generative Recommendation, A Unified Definition of Hallucination, and More! — Recent research advances generative recommendation, LLM integration, and efficiency in information retrieval systems.
• Not All RecSys Problems Are Created Equal — RecSys complexity is dictated by catalog stability, signal strength, and preference subjectivity, not universal architectural mandates.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.