SU NLP 29 at SemEval-2026 Task 5: DynaOrd - Hybrid Dynamic Ordinal Regression with LoRA-Fine-Tuned DeBERTa-v3

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Advanced, quick

Summary

SU NLP 29's DynaOrd system, submitted to SemEval-2026 Task 5, addresses the challenge of rating word sense plausibility in ambiguous narrative contexts. This system predicts human-perceived plausibility scores on a 1-5 scale, tackling issues like limited training data and the ordinal nature of target labels. DynaOrd integrates a DeBERTa-v3-large encoder with Low-Rank Adaptation (LoRA) for efficient fine-tuning. Its core innovation is a dynamically weighted hybrid CORAL-MSE loss function designed for ordinal regression. This loss formulation intelligently adjusts the balance between ranking and regression objectives during training, initially emphasizing ordinal consistency and later shifting focus to regression refinement. The research specifically analyzes how this dynamic loss weighting contributes to the system's overall performance in the task.

Key takeaway

For NLP Engineers developing ordinal regression models, especially with limited data, consider adopting a dynamically weighted hybrid loss approach. Your models can achieve better performance by prioritizing ordinal consistency early in training and then refining regression objectives later. This strategy, exemplified by DynaOrd's use of LoRA with DeBERTa-v3, offers a robust method for tasks like plausibility rating on a 1-5 scale, improving both ranking and score prediction accuracy.

Key insights

DynaOrd combines LoRA-fine-tuned DeBERTa-v3 with a dynamic hybrid CORAL-MSE loss for improved ordinal regression in word sense plausibility.

Principles

• Ordinal consistency benefits early training.
• Regression refinement improves later stages.
• Dynamic loss weighting optimizes objective balance.

Method

DynaOrd uses a DeBERTa-v3-large encoder, fine-tuned with LoRA, combined with a dynamically weighted hybrid CORAL-MSE loss. This loss adapts, prioritizing ordinal consistency then regression refinement during training.

In practice

• Apply LoRA for efficient model adaptation.
• Use dynamic loss weighting for ordinal tasks.
• Combine ranking and regression objectives.

Topics

• Ordinal Regression
• Word Sense Disambiguation
• DeBERTa-v3
• LoRA Fine-tuning
• SemEval-2026 Task 5
• Dynamic Loss Weighting

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• PreLort: Prefix-Nested LoRA for Federated Fine-Tuning under Rank Heterogeneity — PreLort enables robust federated LoRA fine-tuning for LLMs by hierarchically structuring adapter ranks to manage client heterogeneity.
• A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget — Compute-constrained LLM training trajectories reveal non-monotonic degradation and instability, challenging endpoint-only evaluation.
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation — SDS-LoRA mitigates anisotropic gradient scaling in LoRA by decoupling singular values, improving fine-tuning performance.
• Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation — LoRA-Pre reduces optimizer memory overhead by applying low-rank approximation to momentum states in large model training.
• IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resources — Vector-based semantic deduplication and domain-balanced pretraining significantly enhance Persian language model performance across diverse NLU tasks.
• Data Synthesis and Parameter-Efficient Fine-Tuning for Low-Resource NMT: A Case Study on Q'eqchi' Mayan — Synthetic data effectively bootstraps NMT structural learning for low-resource languages, but requires authentic data for semantic depth.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.