AI inference is running into a memory bottleneck. Huawei wants to tackle it with OceanStor M900

As organizations move from straightforward AI interactions towards longer conversations, complex reasoning and autonomous agents, the demands on their infrastructure are changing. Processing increasingly large amounts of context places significant pressure on memory capacity, while keeping accelerators supplied with data becomes just as important as raw computing power. One particular challenge is the key-value (KV) cache, which stores information generated during inference so models can reuse previous computations rather than repeatedly processing the same context. As context windows expand and multi-turn interactions become more demanding, these caches grow considerably. GPU memory and system memory provide fast access but are expensive and capacity constrained. Traditional storage offers greater capacity, yet its longer, CPU-mediated data paths can introduce latency that leaves valuable accelerator resources waiting for data. Huawei believes addressing this gap requires a different approach to storage. …