Lexilogic@IWSLT 2026: Pairwise Ranking Fine-tuning of CometKiwi for Speech Translation Quality Estimation

2026-06-29 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, short

Summary

Pranav Gupta's Lexilogic submission to the IWSLT 2026 Speech Translation Metrics Shared Task presents a novel method for evaluating ASR text to translated text within a specific evaluation scenario. The core of the approach involves fine-tuning CometKiwi-22, a substantial 580M-parameter quality estimation model, by applying a pairwise ranking objective. This fine-tuning process meticulously constructs within-document translation pairs, which are then used for training with an adaptive margin ranking loss. This loss function is further combined with mean squared error (MSE) calibration to refine the model's predictions. The system achieved a notable performance of 35.2% per-source Kendall's τ on the development set. This underscores its potential for robust speech translation quality assessment.

Key takeaway

For NLP Engineers evaluating speech translation quality, consider fine-tuning large quality estimation models such as CometKiwi-22. You should apply a pairwise ranking objective, constructing within-document translation pairs. Integrate an adaptive margin ranking loss, combined with MSE calibration, to improve evaluation accuracy. This method can significantly enhance your system's ability to assess ASR text to translated text quality.

Key insights

Fine-tuning CometKiwi-22 with pairwise ranking significantly improves speech translation quality estimation.

Principles

• Pairwise ranking enhances quality estimation models.
• Adaptive margin loss improves ranking accuracy.
• MSE calibration refines model predictions.

Method

Fine-tune CometKiwi-22 using a pairwise ranking objective. Construct within-document translation pairs. Train with adaptive margin ranking loss combined with MSE calibration.

In practice

• Apply pairwise ranking to QE models.
• Use adaptive margin loss for ranking tasks.
• Integrate MSE calibration for prediction refinement.

Topics

• Speech Translation
• Quality Estimation
• CometKiwi-22
• Pairwise Ranking
• Ranking Loss
• ASR Evaluation

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Closing the Quality Gap in Low-Resource Text-to-Speech: LoRA Fine-Tuning of VoxCPM2 for Khmer and Korean — LoRA fine-tuning significantly improves low-resource TTS quality, but effectiveness depends on base model weakness.
• Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets — Automated framework improves multilingual LLM benchmark translation quality using USI and T-RANK.
• PrefSQA: Pairwise Preference Prediction for Speech Quality Assessment and the Critical Role of High Quality Datasets — Pairwise preference prediction offers a more reliable approach to speech quality assessment than traditional MOS due to cleaner labels.
• Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER) — Benchmark contamination significantly inflates reported ASR performance, especially for dialectal variations.
• Improving Code Translation with Syntax-Guided and Semantic-aware Preference Optimization — CTO unifies syntax-guided and semantic-aware preference optimization for robust, accurate code translation.
• SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval — SHIFT is a training-free indexing method that corrects language bias in multilingual retrieval by adjusting document embeddings.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.