How Does Direct-to-Chip Liquid Cooling Support the Future of AI Inferencing?

Is AI inferencing the greatest challenge the data center world has ever faced?

Perhaps. To understand why, let’s explore what inferencing entails and how to tackle its demands.

Inferencing is the practice of applying trained machine learning models to new, unseen data to generate predictions, decisions, and responses. In essence, it’s the application phase of machine learning, where the model uses the knowledge gained from the training data to generate outputs for new inputs.

We all use inferencing whenever we access Gemini, Meta, ChatGPT, or the other large language models that emerged over the past few years. Every time we type in a prompt, inferencing generates the response. For example, Meta leverages inferencing to make real-time decisions about content displayed to users on platforms like Facebook and Instagram, utilizing trained AI models to analyze user data and predict which posts, ads, or recommendations will be most relevant and engaging for each user based on their previous interactions and behaviors.

Of course, the value of inferencing isn’t limited to individuals. Nearly every business is evaluating or deploying new, generative AI applications on inferencing hardware. According to Salesforce, 86% of IT leaders expect generative AI to quickly play a prominent role in their organizations.

And that means that we’ll see a shift in organizational IT infrastructure priorities.

So far, training infrastructure, with its hot and dense cluster designs that are akin to HPC architectures, has soaked up most of the investment and attention because there’s been a race to build the best models.

However, as companies move from training models and testing generative AI applications to deploying them at full scale, demand for robust inferencing infrastructure will skyrocket, far surpassing the need for training. Current estimates suggest that the market will soon be split into 15% for training, 45% for data center inferencing, and 40% for edge inferencing. This shift highlights a fundamental challenge: today’s data center infrastructure is simply not designed to support the massive and evolving demands of inferencing.

1. Specialized, High-Density Hardware AI hardware for inferencing isn’t standard; it relies on GPUs and TPUs with high-speed interconnects similar to HPC systems. The power demand for Nvidia’s next-gen GPUs is expected to reach 1,000W per card in 2024, with single 8U AI servers consuming over 10kW—far above typical rack limits. Uptime Institute’s Hardware for AI: What Makes It Different report estimates that by early 2025, IT infrastructure supporting Nvidia AI workloads will reach 3,000 MW globally—a surge driven by unprecedented demand for specialized inferencing setups, according to JetCool.
2. Exponential Demand and Limited Data Center Capacity Inferencing demand is growing rapidly, and with it, the need for vast amounts of power and cooling. At the same time, we have a shortage of data center capacity that’s equipped for inferencing hardware.
3. Extreme Power and Cooling Needs GPUs in data center AI applications are increasingly power-intensive, with Nvidia chips using six times the memory they did in 2017. This surge in power and memory capacity strains existing electrical and cooling systems. In response, large-scale inferencing clusters will soon mirror supercomputer layouts, with racks requiring thousands of copper and optical connections to maintain high-speed data flow across GPUs.
4. Latency Sensitivity Many applications, such as real-time video analysis or autonomous driving, require low-latency inferencing, which can be challenging to achieve in large-scale data centers without specialized deployments, often at the edge. Low-latency multi-modal models (that allow input and output of any media, not just text), will require scalable support across various computing environments—including centralized cloud, edge computing, and IoT devices. We’ll see accelerated obsolescence of so much of our existing data center and edge capacity. New designs will be needed.
5. Environmental Impact and Sustainability Goals The computational power required for inferencing can lead to substantial energy consumption, making it a significant environmental concern. That’s happening in the face of power generation and distribution limits that can’t be met with today’s electrical infrastructure. Managing inferencing at scale must align with sustainability goals to reduce water and energy use, minimizing strain on existing infrastructure.

Meeting these challenges sustainably is a monumental task, especially as organizations look to deploy latency-sensitive, multi-modal AI applications globally. Inferencing clusters require dense configurations, full performance under strict power limits, and minimal environmental impact—a tough balance to achieve with conventional infrastructure.

These are hard problems. Let’s take a step back for a moment and consider them in context.

Let’s say an organization wants to deploy a latency-sensitive, multi-modal generative AI application at scale. To do that, they need to deploy fifty clusters of inferencing hardware around the world.

What challenges will they face?

1. Finding spaces to support service demands won’t be easy. They may have to put some of the hardware in data centers and the rest of it at the edge. Variations in environmental conditions are likely from one location to another.
2. Access to physical space is restricted. They have to pack their computationally intensive hardware as densely as possible to fit limited space, even though it’s hot enough to melt itself if it isn’t cooled properly.
3. They have to deliver the highest level of performance within a given power envelope. They must drive maximum efficiency, and can’t waste power anywhere, but they have to do that without throttling or restricting hardware performance.
4. They must be able to minimize the environmental impact of their deployment. They have to deploy and run the application, at full performance, in the most sustainable way.
5. And of course, they have to support these needs for today’s hardware while supporting subsequent generations of inferencing hardware, which is likely to be hotter and denser than before.

How can organizations work through these challenges in a proven, mature, and cost-effective way?

They can use liquid cooling.