HBM Dominance Continues, but the Race for Next-Generation Technology Has Already Begun

Intel and OpenAI engineers have said that "CXL cannot replace HBM." Speaking at a panel discussion at the AI Infrastructure Summit in Santa Clara, California, on Tuesday (local time), they pointed out the limitations of Compute Express Link (CXL), an interface technology that connects chips and memory more broadly and quickly, predicting that high bandwidth memory (HBM), a memory chip that transfers data at ultra-high speeds, will continue to dominate the AI semiconductor market.

CXL is a next-generation standard for efficiently connecting AI accelerators and memory. Some in the industry had floated CXL as a next-generation alternative, hoping it could partially substitute for the costly HBM and lower the cost of building AI servers.

Vidya Thiagarajan, Intel's head of system-on-chip (SoC) architecture, pointed out that "data traveling through CXL to and from the graphics processing unit (GPU) is not as fast as HBM," noting that CXL serves to complement auxiliary storage rather than replace HBM. Daniel Morris, a researcher at OpenAI who works on AI accelerator design, said, "From the perspective of actually running AI models, I can't find a use for CXL." He explained that while CXL could be used to separately store infrequently accessed inactive data in large models, he had not seen use cases specific to running models.

With the CXL alternative losing steam, Samsung Electronics and SK hynix are expected to maintain their dominance in the global HBM market. According to Counterpoint Research, SK hynix held 50% of global HBM revenue in the second quarter of this year, with Samsung Electronics at 33%. Samsung's share rose 12 percentage points from the previous quarter (21%).

At the event, Samsung Electronics introduced 'zHBM,' which stacks HBM directly on top of the AI accelerator. Compared with the conventional approach of placing HBM beside the accelerator, the structure shortens data travel distances, boosting processing speed and power efficiency. Kim In-dong, vice president of memory product planning at Samsung Electronics' Device Solutions America (DSA), likened the vertical connection between the accelerator and memory to "having a private elevator in a hotel room that goes straight down to the lobby." According to the company, zHBM can deliver up to 8 times the performance and more than 3 times the power efficiency of HBM5. Kim said that "conversational AI systems currently respond at around 100 tokens per second per user," and pledged to raise that figure to 1,000 tokens per second to support agentic AI.

SK hynix presented 'processing-in-memory (PIM),' which performs computations directly inside the memory. Instead of moving data to the accelerator for calculation, simple operations are handled within the memory itself. Lim Chi-chul, vice president in charge of Solution AT at SK hynix, explained that "PIM offers roughly 300 times the capacity of SRAM in the same footprint while maintaining high internal bandwidth." He said hybrid bonding, which vertically connects memory dies and compute dies, could resolve capacity and cost issues.