Seeing sounds

2026-03-26 · Source: MIT News - Artificial intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Gaming & Interactive Media · Depth: Intermediate, medium

Summary

Mariano Salcedo ’25, a master’s student in MIT’s new Music Technology and Computation Graduate Program (MTCGP), is developing an AI system to visualize music and other sounds. His research focuses on Neural Cellular Automata (NCA), which combines classical cellular automata with machine learning to generate self-regenerating images. When stimulated by music, these NCA-driven visuals can "show" sounds in real time, allowing users to adjust the relationship between music energy and the NCA system via a web interface to create unique visual performances. Salcedo, an Alex Rigopulos (1992) Fellow, previously earned a BS in artificial intelligence and decision making from MIT and is part of the MTCGP's inaugural cohort, a program directed by Eran Egozy and a collaboration between MIT Music and Theater Arts and the School of Engineering.

Key takeaway

For AI scientists exploring creative applications, consider Neural Cellular Automata (NCA) as a powerful tool for real-time music visualization. This approach offers a unique blend of machine learning and self-organizing systems, enabling novel expressive outputs. You should investigate NCA's potential beyond music, such as modeling biological or social systems, to understand its broader implications for AI development and ethical considerations.

Key insights

Neural Cellular Automata can visualize music in real-time, offering expressive and unpredictable visual dynamics.

Principles

• Self-organized systems exhibit complex emergent behaviors.
• AI can enhance human creativity in multidisciplinary fields.

Method

Utilize Neural Cellular Automata (NCA) paired with audio input to generate dynamic, self-organizing visual patterns. A web interface allows users to control the interaction between music energy and NCA parameters.

In practice

• Develop music-driven visualizers using NCA.
• Explore NCA for modeling complex self-organized systems.

Topics

• Neural Cellular Automata
• Music Visualization
• Artificial Intelligence
• Music Technology and Computation Graduate Program
• Self-Organized Systems

Best for: AI Scientist, AI Student, Research Scientist, Creative Technologist

Related on AIssential

• Pulse-Driven Neural Architecture: Learnable Oscillatory Dynamics for Robust Continuous-Time Sequence Processing — Biologically-inspired oscillatory dynamics significantly improve neural network robustness to temporal input gaps.
• Human–AI jam session shapes live music with swarm intelligence — Human-Swarm Interactive Music Systems (HS-IMSs) enable live musical co-creation through simple, data-light swarm intelligence.
• Fed on Reams of Cell Data, AI Maps New Neighborhoods in the Brain — CellTransformer AI maps novel brain subdivisions, enhancing neuroanatomical understanding beyond existing frameworks.
• NeuralMUSIC: A Hybrid Neural-Subspace Framework for Robot Sound Source Localization — A hybrid neural-subspace framework enhances robot sound source localization by combining deep learning with classical signal processing.
• Modeling of ASD/TD Children's Behaviors in Interaction with a Virtual Social Robot During a Music Education Program Using Deep Neural Networks — Deep neural networks can effectively model and distinguish ASD and neurotypical child behaviors during robot interaction.
• Generative Modeling of Bach-Style Symbolic Music: A Comparative Study of Autoregressive, Latent-Variable, and Adversarial Approaches — Autoregressive LSTMs excel at Bach-style music coherence, while VQ-VAEs improve structure over VAEs, and GANs struggle with generalization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by MIT News - Artificial intelligence.