InstructAV2AV: Instruction-Guided Audio-Video Joint Editing

2026-05-18 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

InstructAV2AV is a novel, end-to-end framework designed for instruction-guided audio-video joint editing, addressing the common issue of audio-video desynchronization in diffusion-based video manipulation methods. The framework introduces InsAVE-80K, the first large-scale dataset for audio-video editing, created via a scalable data synthesis pipeline. InstructAV2AV adapts an audio-video generation backbone, concatenating audio-video input with noisy latent codes to maintain source context. It employs source-instruction gated attention for enhanced instruction following and content preservation, alongside a two-stage training strategy to transfer pre-trained priors effectively. Experiments show InstructAV2AV surpasses current methods across 11 metrics in three aspects on two evaluation sets, demonstrating its capability for controllable content creation.

Key takeaway

For research scientists developing multimedia content manipulation tools, InstructAV2AV demonstrates a robust approach to integrating audio and video editing. You should consider its data synthesis pipeline and two-stage training strategy to overcome desynchronization issues in your own diffusion-based models, potentially leading to more coherent and controllable creative outputs.

Key insights

InstructAV2AV enables instruction-guided audio-video joint editing by leveraging a new dataset and a specialized diffusion framework.

Principles

• Joint audio-video editing requires synchronized manipulation.
• Large-scale, high-quality datasets are crucial for training.
• Pre-trained priors enhance model transferability.

Method

InstructAV2AV uses a scalable data synthesis pipeline for InsAVE-80K, adapts an audio-video generation backbone, concatenates input with noisy latent codes, and applies source-instruction gated attention with a two-stage training strategy.

In practice

• Create high-quality audio-video editing datasets.
• Adapt pre-trained generation models for new tasks.
• Use gated attention for instruction following.

Topics

• InstructAV2AV
• Audio-Video Editing
• Diffusion Models
• Instruction-Guided Editing
• InsAVE-80K Dataset

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• DirectAudioEdit: Inversion-Free Text-Guided Audio Editing via Diffusion Prediction Contrast — DirectAudioEdit enables inversion-free, training-free audio editing in diffusion models by directly constructing target states via diffusion prediction contrast.
• Controllable Video Object Insertion via Multiview Priors — Multi-view priors and dual-path conditioning enable consistent, realistic video object insertion.
• Building Generative Image & Video models at Scale - Sander Dieleman (Veo and Nano Banana) — Diffusion models generate high-quality audiovisual data through iterative denoising of learned compressed latent representations.
• Native Audio-Visual Alignment for Generation — NAVA improves joint audio-video generation by aligning modalities natively before context-conditioned denoising.
• Semantic Editing with Coupled Stochastic Differential Equations — Coupling SDEs with shared Brownian motion enables precise, training-free semantic image editing with high fidelity.
• FreeSonic: Training-Free Temporal-Aware Decoupled Attention for Precise Audio Editing — FreeSonic offers a training-free approach to precise audio editing by decoupling attention and injecting task-oriented noise.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.