A Unified Moral-Value Dataset for Instruction Tuning

2026-07-25 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

A new unified moral-value dataset for instruction tuning, developed by Florian Mai and colleagues, addresses the challenge of aligning large language models (LLMs) with human values. This dataset, available on Hugging Face, integrates 421,662 samples from ETHICS, UNIMORAL, and SOCIAL-CHEM-101, converting them into an instruction-response format. It employs ModernBERT models to impute missing normative ethics and moral foundation labels. Experiments with Qwen3-1.7B-Base show that training on a mixed dataset, combining general tasks with this moral-value data, preserves general-task performance (average OLMES scores 70.8-71.2) and improves value-oriented task performance, with a 10% value data mix yielding an F1 score of 0.8521.

Key takeaway

For Machine Learning Engineers developing value-aligned LLMs, consider integrating this new moral-value instruction tuning dataset into your supervised fine-tuning pipeline. A small proportion, specifically a 10% mix with general task data, can significantly improve value alignment (F1 0.8521) for models like Qwen3-1.7B-Base without degrading general task performance. This approach offers a practical path to more ethically robust LLM deployments.

Key insights

A unified moral-value dataset and instruction tuning pipeline improves LLM value alignment while preserving general task performance.

Principles

• Instruction tuning enhances LLM generalization and instruction-following.
• Mixing value-specific data with general data preserves overall performance.
• Explicitly modeling ordinal structure improves moral judgment prediction.

Method

A three-stage pipeline: standardize moral-value datasets, impute missing values with ModernBERT, generate instruction-response templates via self-prompting (GPT-5.2), and fine-tune LLMs with mixed data.

In practice

• Use a 10% moral-value data mix for optimal value alignment.
• Employ vLLM for structured output and batch inference in evaluations.
• Leverage ModernBERT for imputing missing moral framework labels.

Topics

• Instruction Tuning
• Value Alignment
• Moral-Value Datasets
• Large Language Models
• Supervised Fine-Tuning
• ModernBERT

Code references

• allenai/open-instruct
• allenai/olmes
• huashen218/value_action_gap
• vllm-project/vllm

Best for: Research Scientist, AI Scientist, AI Ethicist, Machine Learning Engineer

Related on AIssential

• VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models — VALUEFLOW unifies LLM value extraction, evaluation, and steering with calibrated intensity control, addressing pluralistic alignment challenges.
• System Report for CCL25-Eval Task 5: New Dataset and LoRA-Fine-Tuned Qwen2.5 — Domain-specific datasets and LoRA fine-tuning significantly enhance LLM performance in specialized tasks like classical poetry analysis.
• Document-tuning for robust alignment to animals — Document-tuning with synthetic data effectively instills robust, generalizable compassionate values in LLMs, outperforming instruction-tuning.
• StoicLLM: Preference Optimization for Philosophical Alignment in Small Language Models — Small LLMs can align with complex philosophical virtues using minimal data, but struggle with outward-facing concepts.
• D2VBench: Benchmarking Large Language Models with Value Dilemmas in Daily Scenarios — D2VBench offers a robust benchmark for LLM value alignment in complex, daily ethical dilemmas.
• Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation — Reward-free alignment significantly improves code LLM non-functional properties, with pretrained models showing higher relative gains and finetuned models offering better absolute performance.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.