Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

A systematic investigation explored reward function design for improving Large Language Model (LLM)-generated Business Process Model and Notation (BPMN) process models using reinforcement learning (RL). Researchers trained two LLM families, Llama 3.1 8B and Qwen 2.5 14B, across 48 configurations with Group Sequence Policy Optimization (GSPO). Rewards were derived from 38 metrics spanning syntactic, pragmatic, and semantic quality. Key findings indicate RL significantly enhances pragmatic and syntactic quality while preserving semantic fidelity, reducing output variability by over sixfold. Counter-intuitively, equal reward weighting consistently outperformed targeted weighting, which often led to policy collapse. Furthermore, design choices like invalidity penalties and Supervised Fine-Tuning (SFT) initialization interacted non-trivially with model architecture, proving essential for one model but irrelevant or counterproductive for another. These results highlight reward composition as a primary determinant of optimization outcomes, with effects as significant as applying RL itself.

Key takeaway

For AI Engineers developing LLMs for structured generation tasks like BPMN modeling, you should prioritize equal reward weighting in your reinforcement learning setups. Avoid aggressive targeted weighting, as it can destabilize optimization and lead to policy collapse. Critically, evaluate your base model's inherent validity to determine if an invalidity penalty is needed, as it can implicitly maintain output diversity. Your decision to apply Supervised Fine-Tuning (SFT) before RL should be informed by your model's architecture and its existing task-relevant capabilities.

Key insights

Reward function design is critical for multi-dimensional structured generation, impacting RL optimization as much as RL application itself.

Principles

• Equal reward weighting is the most robust default.
• Targeted weighting can destabilize optimization.
• SFT initialization effects are architecture-dependent.

Method

The study used Group Sequence Policy Optimization (GSPO) with multi-dimensional rewards from 38 metrics (syntactic, pragmatic, semantic) to fine-tune LLMs for BPMN generation.

In practice

• Prioritize equal reward weighting for multi-objective tasks.
• Assess base model validity behavior for penalty design.
• Evaluate SFT necessity based on model architecture.

Topics

• Large Language Models
• Reinforcement Learning
• BPMN Generation
• Reward Function Design
• Supervised Fine-Tuning
• Group Sequence Policy Optimization
• Model Architecture

Code references

• chlauer99/RL_for_process_modeling
• hiyouga/LlamaFactory

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning — CoUR uses LLMs, uncertainty quantification, and decoupled Bayesian optimization to create more efficient and robust RL reward functions.
• RL Scaling Laws for LLMs — Scaling laws, while precise for LLM pretraining, are more complex and context-dependent in reinforcement learning.
• Reward hacking in Reinforcement learning — Reward hacking is an inevitable outcome of imperfect objective functions in RL, especially with powerful LLM optimizers.
• OPD+: Rethinking the Advantage Design for On-Policy Distillation — Stop-gradient in on-policy distillation biases reward estimates; OPD+ offers a corrected framework for improved teacher-student model transfer.
• Reinforcement fine-tuning with LLM-as-a-judge — LLM-as-a-judge RFT offers flexible, context-aware alignment for LLMs, outperforming traditional methods in complex, nuanced domains.
• Reinforcement Learning without Ground-Truth Solutions can Improve LLMs — RiVER trains LLMs on score-based tasks without ground-truth, using calibrated reward shaping for general coding ability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.