Scaling AI without breaking the grid: The path to sustainable innovation

No one knows exactly how big an impact the GenAI revolution will have on society and the global economy. If all the hyperbolic predictions are accurate, however, it will spark a massive increase in productivity and innovation. But, that can only come true if everything goes right and one of the biggest factors that could go wrong is power. Deploying AI at scale requires a massive buildout of data centres to accommodate the computing capacity to run it.

As the demand for AI grows, so does the load on the power grid. Around the world, researchers and governments are predicting that within two to three years, power demand could outstrip supply, leading to constrained operations and higher electrical rates for everyone.

The typical computing rack — the standard framework for computing equipment installed in a data centre — consumes about 20 kilowatt hours per day. That’s a little lower than the daily power consumption of the average household in the US, about 30 kilowatt hours a day.

The typical AI deployment, however, has caused the power needs for data centres to skyrocket. A single rack based on Nvidia's GPUs will consume as much as 120 kilowatt hours per day or the equivalent of 4.5 US homes. It doesn’t seem like much until you start accounting for the scale required to install hundreds of thousands of these racks across a few thousand data centres and then the power demand quickly adds up.

An estimate by Goldman Sachs from May 2024 pegged the increase in power demand just from AI alone at about 200 terawatt-hours per year by 2030. To put that into perspective, New York City consumes about 55 terawatt-hours per year, an average of about 10,000 gigawatt-hours per day. If Goldman Sachs is right, AI will add the equivalent of three New York City’s to the US power grid.

Obviously, there’s a need for more power. Some of the biggest corporations — Microsoft, Amazon and Google — are buying or building their own nuclear power plants to secure their future supply. This is a great option if you have the resources, but most organizations don’t. For the rest of us, corporate stakeholders and government policymakers must come together to find the best way forward to add more capacity and improve the grid.

All these AI deployments will be a big boon to productivity, driven by agentic AI, where autonomous software agents complete tasks and interact with each other with little or no human interaction. Agentic AI will be a catalyst for the shift in the percentage of AI workloads running. In 2023, training accounted for the majority of AI workloads by a ratio of about 2 to 1. There's an emerging consensus that in 2025, inference workloads will roughly equal training workloads as agents take hold and, next year, inference will become the dominant workload by an ever-widening margin.

The computing capacity and power required to enable this shift will be bigger than anything we’ve seen. In many places around the world, it will require more power than the current grids are prepared to generate or deliver. Recent estimates in the US, Europe and APAC, say power demand in areas where there’s a high concentration of data centres — Northern Virginia is the most cited example in the US — will outpace supply within two or three years without significant grid and generation investments. The result will include data centres that aren’t able to run at full capacity, while construction of others may be delayed. It may also mean higher electric bills for everyone.

Or will it? As the CEO of SambaNova, I talk about this subject every day with customers, government leaders and partners. We build our chip into a hardware platform that offers 10 times the performance of GPU-based solutions, while consuming about one-tenth the power. Our approach to more efficient silicon and hardware offers an entry point into a wider discussion we must have about making AI more efficient as it evolves.

If AI is going to succeed, the companies who build, deploy and deliver it are going to have to get better about using the power that’s available now in smarter and more efficient ways. That requires some new approaches across the AI ecosystem.

In the last year, open-source models, such as Meta’s Llama, have opened up the potential for companies to create their own custom-built models. These pre-trained models can be downloaded for free, fine-tuned on existing troves of data and run in combination with smaller specialized models. They’re getting better all the time and will soon achieve parity with proprietary foundation models. Open-source models help organizations bypass the expensive and power-intensive model-training process. They go straight to fine-tuning and deploying a model they own and can improve on forever.

It’s also important that we get the most utilization out of the hardware that gets deployed. There’s a robust ecosystem of companies — some of them startups — who are leading the way in using innovative software-based optimization to ensure that AI workloads are run in the most efficient way possible. This can reduce the total cost of operation and help reduce the power load on utility grids.

Finally, we can combine all of these approaches, open-source models and more efficient hardware running in a fully optimized environment. The result is more AI productivity while making the most of available power.

Building super-efficient AI is exactly the kind of challenge that the tech industry has met again and again over the last several decades, building on multiple successive layers of research and innovation. Achieving it requires a multi-faceted approach and we can accomplish it together.