As AI Moves from Training to Inference, Optics Moves Closer to the Chip

2026-07-09 · Source: Big Data & AI News - EE Times · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Emerging Technologies & Innovation · Depth: Advanced, medium

Summary

Imec researchers, including Peter Ossieur and Imene Jadli, argue that as AI workloads shift from training to inference, connectivity becomes a critical bottleneck, particularly for scale-up networking connecting accelerators within a rack. While co-packaged optics (CPO) is a logical next step, it will be insufficient for future AI systems due to high power consumption; for example, 250 Tb/s bandwidth could consume 1.25 kW with CPO. Imec proposes moving towards 2.5D optical I/O, integrating optics at the interposer or substrate level, utilizing a "wide and slow" approach with many moderate-speed lanes to reduce optical power to below 200 W. The long-term vision is 3D optical I/O, where optics are native to the 3D compute stack, though this presents significant challenges in materials, manufacturing, and thermal management.

Key takeaway

For AI Architects designing next-generation inference systems, current co-packaged optics solutions will prove insufficient due to prohibitive power consumption, potentially reaching 1.25 kW for 250 Tb/s bandwidth. You must plan for 2.5D and eventually 3D optical I/O integration, prioritizing "wide and slow" approaches to reduce optical power below 200 W. Focus on materials science and thermal management to enable these advanced, power-efficient interconnects.

Key insights

AI inference demands advanced optical I/O beyond co-packaged optics, moving towards 2.5D and 3D integration for power efficiency.

Principles

• Inference requires reactive, low-latency scale-up networking.
• Optical I/O must minimize electrical path length.
• Co-design software, system, and physical layers.

Method

Imec proposes a "wide and slow" approach for 2.5D optical I/O, using many moderate-speed lanes to achieve high aggregate bandwidth with lower energy per bit, reducing optical power from 1.25 kW to below 200 W.

In practice

• Explore 2.5D optical I/O for future AI systems.
• Investigate III-V compounds for CMOS-compatible integration.
• Prioritize thermal management in 3D optical I/O designs.

Topics

• AI Inference
• Optical Interconnects
• Co-packaged Optics
• 2.5D Integration
• 3D Optical I/O
• Scale-up Networking

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Architect, AI Engineer, AI Hardware Engineer

Related on AIssential

• This Will Power Everything — Optical interconnects are critical for scaling AI data centers beyond copper's physical and thermal limitations.
• This Breakthrough Could Kill the GPU Race — Optical computing with active metasurfaces offers a path to dramatically reduce AI's energy footprint by performing computation directly with light.
• Scaling Power-Efficient AI Factories with NVIDIA Spectrum-X Ethernet Photonics — Co-packaged optics in Ethernet switches significantly enhance AI factory scalability, power efficiency, and network resilience.
• This NPU Is 5,000% Faster Than A GPU | Photonic Chips Explained — Photonic MPUs offer extreme energy efficiency for AI inference by using light waves, overcoming silicon's resistive heat limitations.
• Lightmatter CEO: AI’s Next Scaling Challenge Is Interconnect, Not Compute — AI's scaling challenge has shifted from compute to interconnect, making silicon photonics essential for future data center infrastructure.
• All-optical signal processor developed to break AI data center transmission bottleneck — An all-optical signal processor addresses AI data center transmission bottlenecks by eliminating O/E/O conversions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Big Data & AI News - EE Times.