When it comes to generative AI, disruption is not just inevitable, it is the engine of progress. Amid this dramatic shift, memory solutions are racing to catch up, spurred by the demand for enhanced bandwidth, capacity, area efficiency, power management, heat dissipation, and cost-effectiveness. But there is no one-size-fits-all here, and the answer is custom memory, a space set to expand significantly by 2026 when HBM4 is poised to debut. Currently, seven to eight major IT players such as NVIDIA, Amazon, Microsoft, Broadcom, and Marvell are driving custom HBMs.

How Critical Would Custom Memory Be?

The Architectural Crossroad. Custom HBM is at the heart of this evolution, providing a tailored approach to integrating memory into AI systems. Two compelling options for customizing HBM packages are making waves – 1) Mounting HBM directly onto SoC; 2) Adding logic functions to the Base Die while preserving the existing structure. While the first option is more innovative, the second option offers a more practical performance-enhancing alternative.

Competitive Landscape with IPs and Designs. It is not just architecture, the innovations within Custom HBM center around improving data efficiency – 1) To enable custom HBM, IP that facilitates GPU communication is essential, including controllers, PHY (Physical Layer), and interfaces; 2) Photonics and other innovative technologies would improve the efficiency of data transmission; 3) Compatibility with major SoC chips as well as design flexibility should remain critical.

Can Custom Memory Structurally Change the Volatile Dynamics of Memory? Marvell’s projections suggest custom compute opportunities in data centers could grow to 25% by 2028 from 16% in 2023. Some memory manufacturers expect custom HBM to account for 30%-40% of the total HBM market by 2030. However, customers prefer standardized solutions to reduce costs. Ultimately, custom memory’s fate lies in striking a balance – unlocking performance gains while navigating the practicalities of adoption.

Access full report here.

If you are not a client, reach out to us at info@counterpointresearch.com