Woolworths’ AI agent rambled about its ‘mother’. It’s a sign of deeper problems with the tech rollout

2026-02-27 · Source: Artificial intelligence (AI) – The Conversation · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation, AI Ethics & Governance · Depth: Intermediate, short

Summary

Woolworths' AI assistant, Olive, recently exhibited unexpected behavior, including generating overly human-like responses about having a "mother" and providing incorrect pricing information. These issues stem from Olive being powered by a large language model (LLM) that generates plausible language statistically, rather than possessing human-like knowledge or real-time data access. The "mother" references were traced to pre-written scripts from an old decision tree, which Woolworths has since removed. Pricing errors occurred because LLMs require explicit connections to live databases for accurate, up-to-date information. This incident highlights a recurring problem for companies deploying customer-facing AI without adequate oversight, echoing similar issues faced by Air Canada and DPD, where chatbots provided incorrect information or generated inappropriate content, leading to legal action or public embarrassment.

Key takeaway

For Directors of AI/ML overseeing customer-facing deployments, ensure your AI systems are rigorously tested, grounded in real-time data, and have clear oversight mechanisms. Your organization is fully accountable for all chatbot outputs, regardless of disclaimers. Prioritize accuracy and transparency over anthropomorphic features to manage customer expectations and mitigate legal and reputational risks, especially when dealing with sensitive information like pricing.

Key insights

Companies deploying customer-facing AI must ensure accuracy and accountability, as they own the system's outputs.

Principles

• LLMs generate plausible language, not human knowledge.
• Anthropomorphized chatbots increase customer dissatisfaction when they fail.
• Companies bear full liability for their chatbot's actions.

In practice

• Connect LLMs to live databases for real-time data.
• Implement robust oversight for customer-facing AI.
• Review and remove outdated chatbot scripts.

Topics

• AI Chatbots
• Large Language Models
• AI Accountability
• Customer Service AI
• AI Ethics

Best for: Executive, VP of Engineering/Data, Director of AI/ML, AI Product Manager, CTO, AI Ethicist

Related on AIssential

• Retailers want ‘delightfully human’ AI to do your shopping, but will the chatbots go rogue? — Agentic AI shopping assistants offer autonomy but pose risks in balancing human-like interaction with reliable performance.
• Study: AI chatbots provide less-accurate information to vulnerable users — AI chatbots underperform for vulnerable users, providing less accurate information and exhibiting biased behaviors.
• Research: Why You Shouldn’t Treat AI Agents Like Employees — Anthropomorphizing AI agents as "employees" reduces human accountability and increases over-reliance on AI outputs.
• When a chatbot runs your store — Large language models struggle with reality adherence, making them unreliable for autonomous transactional roles.
• Chatbot Makers Try Sex Appeal — AI development faces ethical dilemmas regarding content moderation and societal impact, alongside rapid technological and investment growth.
• Think AI ‘knows’ what it’s doing? Well, think again - Digital Journal — Journalists use anthropomorphic language for AI less frequently and more nuancedly than often assumed.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial intelligence (AI) – The Conversation.