Dual Hierarchical Dialogue Policy Learning for Legal Inquisitive Conversational Agents

2026-05-15 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

This research introduces Inquisitive Conversational Agents (ICAs), a new class of dialogue systems designed to proactively extract information to achieve specific objectives, contrasting with traditional user-driven systems. The authors developed an ICA tailored for U.S. Supreme Court oral arguments, employing a Dual Hierarchical Reinforcement Learning framework. This framework features two cooperating RL agents: an Appraisal Agent that evaluates attorney responses in real time, and a Hierarchical Dialogue-Policy Agent that coordinates strategic dialogue management and fine-grained utterance generation across a three-level action taxonomy. The system learns to ask probing questions, emulate judicial questioning, and systematically uncover crucial information. Evaluations on a U.S. Supreme Court dataset demonstrate that this method outperforms various baselines across metrics like Conformity, Progression, Outcome Relevance, and Probing Effectiveness Scores, as well as multi-turn Coverage and Marginal Relevance Scores, marking a significant step towards high-stakes, domain-specific conversational AI.

Key takeaway

For research scientists developing advanced conversational AI, this work demonstrates a robust framework for building proactive, goal-driven agents in complex, non-collaborative domains. You should consider adopting a dual-agent hierarchical reinforcement learning architecture, particularly for applications requiring strategic information extraction and nuanced response evaluation, such as legal tech or investigative tools. This approach allows for more effective dialogue steering and superior performance compared to traditional or fine-tuned LLM baselines in high-stakes environments.

Key insights

Inquisitive Conversational Agents proactively extract information using a dual-hierarchical reinforcement learning framework for high-stakes dialogues.

Principles

• Dialogue systems can be categorized into collaborative, negotiation, and inquisitive types.
• Proactive agents require long-term dialogue and questioning strategies in non-cooperative contexts.
• Hierarchical action spaces simplify complex domain-specific utterance generation.

Method

A Dual Hierarchical Reinforcement Learning framework uses an Appraisal Agent to evaluate responses and a Hierarchical Dialogue Agent for multi-level action selection, optimizing for goal-relevance, novelty, and succinctness rewards.

In practice

• Implement a dual-agent RL framework for proactive information gathering.
• Discretize appraisal types and dialogue acts for domain-specific contexts.
• Use Poincaré embeddings to represent hierarchical dialogue acts.

Topics

• Inquisitive Conversational Agents
• Dual Hierarchical Reinforcement Learning
• U.S. Supreme Court Oral Arguments
• Dialogue Policy Learning
• Reinforcement Learning Rewards

Code references

• infosenselab/Dual-Hierarchical-Dialogue-Policy-Learning-for-Legal-Inquisitive-Conversational-Agents

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Scaling up multi-agent systems: an interview with Minghong Geng — Hierarchical and hybrid designs are key to scaling multi-agent systems for complex, long-horizon problems.
• ATPO: Adaptive Tree Policy Optimization for Multi-Turn Medical Dialogue — ATPO uses uncertainty-aware adaptive tree search and efficient execution to optimize multi-turn medical dialogue LLMs.
• Joint Learning of Experiential Rules and Policies for Large Language Model Agents — JERP jointly learns experiential rules and policies from shared trajectories, aligning them for improved LLM agent performance.
• Hey Chat, Can You Teach Me? Structuring Socratic Dialogue for Human Learning in the Wild — Effective LLM-based Socratic tutoring requires explicit curriculum structuring and knowledge inference, not merely larger models.
• Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents — STEP-HRL uses hierarchical and local progress modules to enable efficient step-level learning for LLM agents.
• One Model, Multiple Goals: Adaptive Multi-Objective Learning for E-commerce Dialogue Systems — MORE jointly optimizes reasoning accuracy and linguistic naturalness in e-commerce dialogue systems via constrained reinforcement learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.