SocialPersona: Benchmarking Personalized Profiling and Response with Multimodal Social-Media Context

2026-06-25 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

SocialPersona is a new benchmark designed to evaluate multimodal large language models (MLLMs) on their ability to infer user preferences from longitudinal social-media timelines and apply these in dialogue. Unlike evaluations focused on explicit memory, SocialPersona addresses comprehensive personalization by using "revealed preferences" from natural user traces. It comprises data from 171 everyday social-media users, including text, images, and timestamps, alongside 2,597 human-verified preference tags across seven interest domains. The benchmark supports tasks like structured user profile construction and personalized response generation. Initial experiments show MLLMs identify broad interests but struggle with fine-grained or recent preferences, especially when personalizing dialogue. Text and images offer complementary signals, highlighting ongoing challenges in robust cross-modal, long-horizon user modeling.

Key takeaway

For AI scientists and ML engineers developing personalized assistants, SocialPersona highlights critical gaps in current MLLM capabilities. You should prioritize robust cross-modal, long-horizon user modeling to infer fine-grained and recent interests from multimodal social data. Focus on improving how inferred profiles translate into genuinely personalized dialogue, as this remains a significant performance bottleneck.

Key insights

SocialPersona benchmarks MLLMs' ability to infer and act on user preferences from multimodal social media data for personalized dialogue.

Principles

• Comprehensive personalization requires inferred preferences.
• Multimodal data offers complementary preference signals.
• Long-horizon user modeling remains a key challenge.

Method

SocialPersona constructs user timelines from 171 social-media users, annotates 2,597 preferences across seven domains, and supports tasks for profile construction and personalized response generation.

In practice

• Evaluate MLLMs on fine-grained interest inference.
• Test cross-modal preference signal integration.
• Benchmark dialogue personalization with inferred profiles.

Topics

• Multimodal LLMs
• User Profiling
• Personalized Response Generation
• Social Media Analysis
• Preference Inference
• Benchmark Datasets

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Assessing Capabilities of Large Language Models in Social Media Analytics: A Multi-task Quest — Modern LLMs were systematically evaluated across key social media analytics tasks using a Twitter (X) dataset.
• Learning Transferable Latent User Preferences for Human-Aligned Decision Making — CLIPR learns transferable natural language rules from minimal user conversation to align LLM decisions with latent preferences.
• Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench — LLMs struggle to reconstruct specific, socially grounded consumer reactions in public discourse, despite strong technical performance.
• Assessing Capabilities of Large Language Models in Social Media Analytics: A Multi-task Quest — Modern LLMs demonstrate varied capabilities across social media analytics tasks, with GPT-4 excelling in authorship verification and Gemini in attribute inference.
• HippoCamp: Benchmarking Contextual Agents on Personal Computers — Current AI agents struggle with multimodal perception and evidence grounding in realistic personal file management.
• Personalized Turn-Level User Conversation Satisfaction Benchmark — Personalized user satisfaction in AI conversations demands evaluation methods that integrate user memory and turn-level context, moving beyond generic response quality.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.