Social workers’ AI tool makes ‘gibberish’ transcripts of accounts from children

2026-02-11 · Source: AI (artificial intelligence) | The Guardian · Field: Government & Public Sector — Digital Government & E-Government, Public Policy & Governance, Social Services & Welfare · Depth: Intermediate, short

Summary

A study by the Ada Lovelace Institute, based on research across 17 English and Scottish councils, reveals that AI transcription tools used by social workers are generating potentially harmful errors in official care records. These inaccuracies range from incorrectly suggesting suicidal ideation in clients to producing "gibberish" in transcripts of children's accounts. Despite these issues, dozens of councils, including Croydon and Redcar and Cleveland, have adopted these tools, such as Magic Notes and Microsoft Copilot, to save time amid staff shortages. While some social workers report time savings and improved relational aspects of care, others express concern that AI-produced inaccuracies could lead to incorrect decisions about child welfare and professional consequences, especially given inadequate training and inconsistent checking of AI outputs.

Key takeaway

For Directors of AI/ML in public sector organizations deploying AI tools in sensitive domains like social work, you must prioritize robust validation and oversight. The reported "hallucinations" and misrepresentations, such as false suicidal ideation, underscore the critical need for human-in-the-loop verification and comprehensive training. Your teams should establish clear guidelines for AI use, mandate thorough checking of all AI-generated content, and ensure that time-saving benefits do not compromise the accuracy and safety of client records, potentially leading to severe professional and ethical repercussions.

Key insights

AI transcription tools in social work introduce harmful inaccuracies and "hallucinations" into official care records.

Principles

• AI outputs require rigorous human verification.
• Specialized AI tools may outperform generic ones.
• Time savings do not negate accuracy risks.

In practice

• Implement mandatory AI output verification protocols.
• Provide comprehensive AI tool training for staff.
• Evaluate AI tools for domain-specific accuracy.

Topics

• Social Work AI
• AI Hallucinations
• Transcription Technology
• AI Safety
• Record Inaccuracies

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Ethicist, Policy Maker, Domain Expert

Related on AIssential

• Sullivan & Cromwell apologises to US court for filing errors caused by AI hallucinations - Bar and Bench — AI hallucinations pose significant risks in legal contexts, requiring strict adherence to verification protocols.
• AI Report: Cop Turns Into Frog — AI advancements are driving high talent costs, shifting infrastructure paradigms, and introducing both innovative educational tools and concerning hallucinations.
• Man used AI to make false statements to shut down London nightclub, police say — AI-generated false complaints are an emerging threat to regulatory processes and public administration.
• Australian federal court warns lawyers over ‘unacceptable’ use of AI — Australian federal court mandates disclosure and verification for AI use in legal documents, warning of penalties for errors.
• KPMG pulls report on AI usage due to apparent hallucinations — AI-generated content requires rigorous human oversight to prevent factual inaccuracies and reputational damage.
• After trial judge cites AI-generated judgment, Supreme Court asks BCI to set up expert panel to examine issue - Bar and Bench — AI-generated legal "hallucinations" necessitate judicial accountability and sovereign AI development in India.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI (artificial intelligence) | The Guardian.