Large-Language-Models-as-a-Judge in Theory-Agnostic Adaptive Metric-Alignment for Prototypical Networks in Personality Recognition

2026-07-09 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Social Sciences & Behavioral Studies · Depth: Expert, quick

Summary

The JAM (Judge for Adaptive Metric-Alignment) framework introduces a theory-agnostic approach to personality recognition, moving beyond traditional theory-dependent models that fit predefined psychological taxonomies. This framework aims to discover unified latent pseudo-facets and generalizable psychological representations directly from textual samples, without requiring theory-specific labels. JAM utilizes an Attention-Pooled Graph Prototypical Network for structured representation learning via embedding space clustering. It also incorporates a Cross-Theory Harmonization (CTH) approach, integrating Human-Guided Linkage and Machine-Induced Consensus to unify diverse datasets. To enhance robustness and data quality, an LLM-as-a-Judge mechanism operates in "LLM-before-the-loop" and "LLM-in-the-loop" configurations, identifying ambiguous samples to guide adaptive metric learning. Experiments demonstrate that JAM improves cross-framework generalization and performance, advancing theory-agnostic personality inference and supporting low-resource personality theories. Related code and artifacts are available at https://research.jingjietan.com/JAM.

Key takeaway

For NLP Engineers developing personality recognition systems, you should consider adopting theory-agnostic frameworks like JAM to overcome limitations of predefined psychological taxonomies. This approach allows you to infer latent psychological profiles directly from textual data, enhancing generalization across diverse contexts. Integrate LLM-as-a-Judge mechanisms, operating in "before-the-loop" or "in-the-loop" configurations, to identify ambiguous samples and guide adaptive metric learning, thereby improving data quality and robustness, especially for low-resource personality theories.

Key insights

JAM enables theory-agnostic personality recognition by learning unified latent psychological structures directly from text, guided by LLMs.

Principles

• Personality recognition benefits from theory-agnostic models.
• Latent psychological profiles can be inferred directly from text.
• LLMs can identify ambiguous samples for metric learning.

Method

JAM employs an Attention-Pooled Graph Prototypical Network with Cross-Theory Harmonization (Human-Guided Linkage, Machine-Induced Consensus) and an LLM-as-a-Judge mechanism (before-the-loop, in-the-loop) for adaptive metric learning.

In practice

• Infer personality profiles from text without specific labels.
• Unify heterogeneous personality datasets.
• Improve robustness in low-resource personality theories.

Topics

• Personality Recognition
• Large Language Models
• Prototypical Networks
• Metric Learning
• Cross-Theory Harmonization
• Latent Traits Inference

Best for: AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Large-Language-Models-as-a-Judge in Theory-Agnostic Adaptive Metric-Alignment for Prototypical Networks in Personality Recognition — JAM enables theory-agnostic personality recognition by learning unified latent "pseudo-facets" across diverse psychological frameworks.
• Large-Language-Models-as-a-Judge in Theory-Agnostic Adaptive Metric-Alignment for Prototypical Networks in Personality Recognition — A theory-agnostic framework uses LLMs and graph networks to discover unified latent personality traits from text.
• Hierarchical Multi-Stage Modeling of Adaptive and Maladaptive Self-States in Social Media Timelines — Integrating multi-task learning and structured generation enhances interpretable mental health analysis from social media data.
• Fine-Tuned Multi-Agent Framework for Detecting OCEAN in Life Narratives — A multi-agent LLM framework uses psychometric supervision and perspective conditioning to accurately infer OCEAN personality traits from text.
• Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry — Small LMs encode strong evaluative signals in internal representations, enabling efficient "Representation-as-a-Judge" evaluation.
• LEX-EC: A Lexical Evidence-Channel Audit Framework for Zero-Shot LLM Personality Classification in Black-Box Settings — LEX-EC audits black-box LLM personality classification by combining prevalence, agreement, and lexical ablation to identify trait-associated signals.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.