Asking the Right Questions: Can expert-prompted LLMs reformulate legal queries from non-experts?

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study published in the 2026 CustomNLP4U proceedings investigates the ability of Large Language Models (LLMs) to reformulate legal queries from non-experts into professional legal language. This research, titled "Asking the Right Questions: Can expert-prompted LLMs reformulate legal queries from non-experts?", utilized a novel evaluation framework informed by legal expertise on "in-the-wild" data from non-experts seeking legal advice. Findings indicate that LLMs possess significant potential in legal reasoning for this task, which could streamline discussions between lawyers and clients. However, the study also identified unexpected safety concerns. Additionally, the research found that incorporating linguistically-aligned in-domain text samples can enhance the performance of smaller models, even if these samples are not factually aligned with the specific question. The paper spans pages 167–181.

Key takeaway

For legal professionals developing AI tools or NLP engineers specializing in legal tech, LLMs offer a promising avenue to bridge the communication gap between non-experts and lawyers by reformulating initial queries. You should prioritize robust safety evaluations for these applications. Consider integrating linguistically-aligned legal text samples to enhance smaller model performance, which can streamline initial client interactions and improve access to professional help.

Key insights

LLMs can reformulate non-expert legal queries, improving lawyer-client communication, but require careful safety consideration.

Principles

• LLMs hold significant legal reasoning potential.
• In-domain text samples boost smaller model performance.
• Safety concerns can emerge with LLM legal applications.

Method

The study evaluated LLM-generated legal question reformulations using a novel, legal expertise-informed framework on "in-the-wild" non-expert data.

In practice

• Use LLMs to pre-process client queries.
• Incorporate domain-specific linguistic samples.
• Develop safety protocols for legal AI tools.

Topics

• Large Language Models
• Legal AI
• Query Reformulation
• Non-expert Communication
• Domain Adaptation
• AI Safety

Best for: Research Scientist, AI Scientist, NLP Engineer, Legal Professional

Related on AIssential

• When Rules Learn: A Self-Evolving Agent for Legal Case Retrieval — An LLM-based agent can self-evolve query rewriting rules to enhance BM25 for legal case retrieval, leveraging an automatic evaluation environment without parameter training.
• Claude Legal Prompt Shock, LegalOn GPT 5.4 Review, Legal Innovators + — LLMs are steadily improving in legal task accuracy, reducing the need for extensive human oversight.
• Adaptive Interviewing for Persona Simulation in LLMs: Evidence-Grounded Reasoning Improves Decision Alignment — Adaptive interviewing improves LLM decision simulation by selectively grounding predictions in user-specific, follow-up derived evidence.
• LLM-based Models for Detecting Emerging Topics in Service Feedback — Integrating fine-tuned, quantized LLMs with human expertise reliably detects emerging service quality topics and potential inequities in feedback.
• Litera Webinar: Education + Training in the Age of Legal AI — Legal education and training must evolve to integrate AI, preparing lawyers to practice with, not just alongside, AI.
• Legal AI? Apparently, it’s over. Hello, 'Agentic Law' — The legal tech landscape is shifting from "Legal AI" to "Agentic Law," driven by advanced LLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.