A cross-stack solution for energy-efficient data centers

According to projections, data centers around the world could double their electricity consumption by 2030. Driven by AI, data centers could claim 945 terawatt hours of energy in the next five years, a figure that’s greater than the current electricity consumption of Japan. 

GPUs, or graphics processing units, are the building blocks for many AI tools. These hardware units are good at carrying out simple but massively parallel tasks, a workflow that comes in handy for the mathematical matrices that power large language (LLM) and optimization models. 

“GPUs provide immense compute and memory bandwidth capabilities,” said Dimitrios Skarlatos, an assistant professor in the computer science department at Carnegie Mellon University, but a single unit’s power consumption can reach a thousand watts, which is similar to the amount of electricity needed to power a microwave or a dishwasher. 

With support from The Scott Institute for Energy Innovation’s seed grant program, Skarlatos will develop a proof-of-concept for a virtualization layer capable of performing fine-grained, efficient power management of GPUs. As GPUs increase dramatically in terms of resources, using them efficiently is critical. 

The proposed virtualization layer is intended to break a single GPU into a hundred pieces, enabling multiple models to share the same hardware. Furthermore, it enables techniques such as “right-sizing” that allow a computing task to use the right amount of resources for each workload. The computing tasks that GPUs are responsible for rely on mathematical algorithms that are used for training and inference of deep learning models. Skarlatos aims to improve energy efficiency in GPUs by a factor of twenty. 

“Beyond hardware resources alone, each AI model’s workload responds differently to frequency scaling. By adjusting GPU frequency, we can deliver the same performance with a lower power budget,” said Skarlatos. “What we need to identify dynamically is the optimal frequency for your performance target.” 

Skarlatos is also thinking about this project in terms of facilitating easy adoption in industry. 

“We’re proposing a fully transparent framework that developers can use without making changes to their AI models,” said Skarlatos. This not only allows for faster uptake, but makes sure that the hardware solution doesn’t become outdated as AI tools continue to evolve at a rapid pace.