Why it is time to prioritize  the sustainable development of AI

I believe that artificial intelligence (AI) will be the most consequential technology of this century and perhaps beyond. It will change every facet of our lives and help us address some of the world’s most pressing problems, everything from climate change to road safety to cancer. In the last couple of years, fueled by the generative AI boom, we’ve started to see some of that potential come to fruition.

But, I’m troubled by the direction the industry is headed and its impact on future generations and the planet. The capabilities of modern AI are remarkable, but the predominant method for developing this technology is through brute force. Better performance requires bigger models that use more data, more compute and consume vastly more energy. Already we're seeing complications and inequities resulting from this trajectory.

If we stay on this path, the unsustainable cycle will continue until we hit the limits of our resources. We must change course and turn our attention towards sustainable AI to unlock AI's full potential, while fostering an equitable future for this transformative technology.

Over the past couple of years, AI models have exploded in size. While each new generation has advanced in some way, they’ve also become much more capital intensive to develop. In 2024, OpenAI will reportedly spend as much as $3 billion on training costs for ChatGPT and its newer models, alongside $4 billion on servers to run inference workloads. Capital expenditure at Microsoft, Meta, Amazon and Google has increased dramatically — set to top $200 billion in 2024 as they look to stay in the AI race.

While computational costs have continued to go up, so has energy consumption. A single training run for a large language model (LLM) is estimated to consume as much electricity as 130 US homes would consume in a year. That number will only increase as models become even larger. According to the International Energy Agency (IEA), data centre electricity usage will double by 2026 with demand rising somewhere between 650TWh and 1,050TWh. At the high end, the IEA states that this would be like adding the power consumption of an entire country like Germany.

This unsustainable trajectory also has broader societal implications. The brute-force approach to AI creates barriers to entry, limiting access to AI innovation for those without vast resources. We’re starting to see compute emerge as a new form of geopolitical capital with wealthier countries vying for control of advanced chip manufacturing. This trend risks creating a world where only a select few control this technology and benefit from its applications. Such a scenario could exacerbate existing inequalities and stifle innovation.

There is a better way.

Given today's neural network architectures have demonstrated that they are successful at scaling, the industry’s attitude has been: why shouldn't we stay the course? But AI scaling laws, which predict continuous performance improvements with increased model size, are beginning to break down as models approach saturation and further gains come at a much larger cost.

We must act now — waiting to hit a barrier is not an option. The brightest minds in industry, academia, venture capital and the government must redirect their attention and resources towards sustainable AI. This means not only creating new AI models with advanced reasoning capabilities that can generalize from a much smaller number of examples, but also completely reimagining the learning paradigm and the role of data during training.

Today's dominant AI models are inefficient to train because they rely on pattern recognition, instead of genuine understanding. Their capabilities resemble 'system 1' thinking in humans, a concept popularized by Daniel Kaheman, which involves rapid, intuitive and reactive cognitive processing that occurs unconsciously and without deep reasoning. Humans rely on system 1 thinking for quick decision making, like slamming on the brakes when a pedestrian unexpectedly darts in front of us as we are driving. Most AI models use this type of thinking for all decisions.

For solving more complex problems, humans utilize 'system 2' thinking, which involves more conscious and deliberate decision-making. In the previous example, if given more time, we reason about other actions we can take, their consequences and then choose the safest action. Rather than immediately reacting by slamming on the brakes, we might choose to swerve. This maneuver could help avoid a collision with the pedestrian, as well as prevent a pileup behind us. I have spent the last two decades building several generations of AI models that can 'think' in much the same way. Like humans, these models can generalize from fewer examples and adapt more effectively to new situations. This approach leads to safer, more performant models that are also more sustainable to develop.

Traditional approaches to training AI rely on massive, static datasets and brute force learning, where the AI acquires knowledge by looping over the examples in the dataset. This is a naive and inefficient way to learn.

A paradigm shift is necessary, one that mirrors the nuanced and dynamic nature of human learning. Instead of volume, the emphasis should be on high-quality, highly informative data that evolves over time as the AI matures in its skills and learning abilities. As an example, in first grade we might learn to perform simple addition, but by the time we’re in sixth grade our teachers will have advanced the curriculum to include algebra and geometry. In practice, this means we need to focus on providing the AI with data where the amount of learning information being provided per example unit is maximized during the learning process.

Additionally, just as a teacher shares complex feedback throughout the learning process, we must provide the AI with data rich in supervision – moving beyond basic feedback to offer detailed insights into its errors, what can be better in its reasoning process and how it can improve its output.

Lastly, in this new learning paradigm, we must enable the AI to interact with the task it’s trying to solve in an active way — just as humans do. We don't learn to drive by watching videos of people driving so that we can imitate their actions on the road. We learn by sitting behind the wheel and driving. This 'closed loop' learning process allows the AI to receive feedback on the execution of its outputs and gain insights into its own internal reasoning — fostering a deeper understanding and more efficient learning. Reinforcement learning algorithms have tried to achieve this but have so far been too data hungry to offer a sustainable solution.

By focusing our attention on these two areas of innovation — developing AI models capable of genuine understanding and reimagining the learning paradigm — we can unlock a new era of AI.

We are at a pivotal moment. How we navigate the path ahead matters immensely. The decisions we make today about where to invest our resources and what to build will shape our future. To ensure genuine innovation and prevent unintended consequences, we must prioritize responsible development.

In the interests of our world and future generations, it’s time to prioritize efficiency and ingenuity over the brute force scaling of today’s models. This is the only way to unlock the true potential of AI while ensuring this technology benefits all of humanity.