Tie-Calibrated COMETKiwi for Speech Translation Quality Estimation: IWSLT2026 Metrics Track

2026-06-29 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Advanced, quick

Summary

Tie-Calibrated COMETKiwi is a system submitted to the IWSLT 2026 Speech Translation Metrics shared task, designed for reference-free quality estimation in English-to-German and English-to-Chinese speech translation. This system integrates COMETKiwi-22, applied to ASR transcripts, with a novel post-processing technique called tie calibration. Tie calibration is a learned score-bucketing method that consolidates near-identical scores into exact ties, effectively mitigating noisy within-document pairwise ranking errors. On the official development set, the method achieved an average segment-level Kendall tau-b of 39.4%, outperforming plain COMETKiwi (34.6%), SpeechQE (29.2%), and BLASER 2.0 QE (24.4%). Its system-level Soft Pairwise Accuracy was 88.0%, comparable to COMETKiwi's 89.4% and superior to SpeechQE's 86.0%. The approach requires no audio input, no model retraining, and only one hyperparameter per target language, tuned solely on the training split.

Key takeaway

For NLP Engineers developing speech translation quality estimation systems, consider integrating lightweight post-processing steps like tie calibration. This method significantly improves segment-level Kendall tau-b scores, achieving 39.4% compared to plain COMETKiwi's 34.6%, without requiring audio or retraining. You can enhance your existing reference-free QE pipelines by tuning a single hyperparameter per target language on your training data, leading to more robust and accurate quality predictions.

Key insights

Tie calibration enhances COMETKiwi-22 for reference-free speech translation quality estimation by reducing noisy pairwise ranking errors.

Principles

• Score bucketing reduces ranking noise.
• Post-processing enhances QE metrics.
• Reference-free QE needs no audio.

Method

Apply COMETKiwi-22 to ASR transcripts, then use tie calibration, a learned score-bucketing, to collapse near-identical scores into exact ties, reducing within-document pairwise ranking errors.

In practice

• Evaluate English-German ST quality.
• Evaluate English-Chinese ST quality.
• Enhance existing QE systems.

Topics

• Speech Translation
• Quality Estimation
• COMETKiwi
• Tie Calibration
• ASR Transcripts
• IWSLT

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics — Cross-lingual scoring bias in translation metrics can be identified and mitigated using controlled, parallel-quality datasets.
• BaltiVoice: A Speech Corpus and Fine-tuned Whisper ASR System for the Balti Language — Fine-tuning pre-trained ASR models with modest, newly created corpora can establish functional speech recognition for low-resource languages.
• A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation — A new evaluation method for long-form SimulS2ST uses ASR, forced alignment, and sentence embeddings to measure latency and quality.
• CN-NewsTTS Bench: a target-level automatic benchmark for raw-input Chinese news TTS pronunciation — CN-NewsTTS Bench evaluates raw-input Chinese news TTS systems on complex written forms without external aids.
• PASQA: Pitch-Accent-Focused Speech Quality Assessment Model Trained on Synthetic Speech with Accent Errors — PASQA accurately assesses pitch-accent correctness in synthetic speech by leveraging a specialized dataset and architectural enhancements.
• KIT's Submission to Cross-Lingual Voice Cloning in IWSLT 2026 — Explicit language tags and lexical matching significantly improve cross-lingual voice cloning performance and reduce accent leakage.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.