Generative AI energy consumption grows, but ROI role unclear

Accounting for energy costs

Enterprise consumers of GenAI might not see energy costs as a billing line item, but it’s nevertheless present.

Ryan Gross, senior director of data and applications at Caylent, an AWS cloud services provider in Irvine, Calif., said generative AI’s energy consumption is directly proportional to the cost.

Much of the energy cost stems from two categories: model training and model inferencing. Model inferencing happens every time a user prompts a GenAI tool to create a response. The energy use associated with a single query is miniscule compared with training an LLM — fractions of a cent vs. millions of dollars. However, the power demands and costs of individual queries add up over time and across millions of users.

How customers absorb those costs remains a bit murky. A business using an enterprise version of a generative AI product pays a licensing fee to access the technology. To the extent energy costs are baked into the fee, those expenses are diffused across the customer base.

Indeed, a PwC sustainability study, published in January, found that emissions stemming from generative AI’s power consumption — during model training, for instance — were distributed across each corporate entity licensing the model.

“Because the foundational training is shared, you actually spread that cost across a lot of users,” Likens said.

As for inference costs, GenAI vendors use a system of tokens to assess LLM usage fees. There’s a charge for each token, and the more complex the query, the more tokens the vendor processes. More tokens signal higher energy use, as inferencing requires power. But the financial effects on enterprises appear to be minimal.

“The token cost has come down since last year,” Likens said, citing PwC’s in-house use of generative AI. “So, the inferencing cost has not been a large [cost] driver, even though we are using it more.”

The biggest cost contributors to generative AI deployments continue to be the usual suspects, such as infrastructure and data preparation, Likens said.

Rajesh Devnani, vice president of energy and utilities at Hitachi Digital Services, the technology services subsidiary of Hitachi Ltd., offered a similar assessment. He acknowledged the importance of generative AI’s energy use, citing various estimates that a GenAI query response consumes at least four to five times the power of a typical internet search query. But he pointed to other cost contributors as playing a greater role in determining a financial return: data preparation and ongoing data governance; training and change management; and model training, which includes infrastructure and tool costs.

“ROI calculations of GenAI should definitely consider energy costs as a relevant cost factor, though it would not count as the most significant one,” he said.

Indirectly influencing energy consumption

Most GenAI adopters don’t appear to have elevated energy costs as a concern. But they could end up indirectly addressing consumption as they tackle other deployment challenges.

That prospect has much to do with how organizations perceive their top obstacles. Until recently, the cost efficiency of models has prevented organizations from scaling GenAI from limited deployments to entire customer bases, Gross said. But the latest generation of models are more economical, he added.

For example, OpenAI’s GPT-4o mini, released in July, is 60% less expensive than GPT-3.5 Turbo regarding charge per token processed, according to the company.

Against that backdrop, organizations now are starting to focus on user expectations, specifically the time it takes to fulfill a request made to a generative AI model.

“It’s more of a latency problem,” Gross said. “Users won’t accept what we are seeing from the usability [perspective].”

Enterprises, however, can tap smaller, fine-tuned models to reduce latency. Such models generally demand fewer computational resources — and, therefore, require less energy to run. Organizations can also include smaller models as part of a multimodel GenAI approach, Gross said. Multiple models offer a range of latency and accuracy levels, as well as different carbon footprints.

In addition, the emergence of agentic AI means problems can be broken down into multiple steps and routed through an autonomous agent to the optimal GenAI model. Prompts that don’t require a general-purpose LLM are dispatched to smaller models for faster processing and — behind the scenes — lower energy use.

But cost efficiency, despite the increased interest in latency, remains an issue for GenAI adopters.

“In general, we’re trying to use agentic architecture to optimize costs,” Likens said. “So, triaging a broken-down question for the right model that costs the least amount of money for the highest accuracy.”

Yet, organizations that build AI agents and create effective agentic architectures also stand to reduce energy consumption, Likens noted.

Top data centers deal with GenAI energy demands

Companies consuming generative AI might obliquely address energy consumption. But data centers that train and run models face increasing power demands head on. Their expanding investment in cooling systems offers evidence.

The data center physical infrastructure (DCPI) market’s growth rate increased for the first time in five quarters during the second quarter of 2024, according to the Dell’Oro Group. The Redwood City, Calif., market research firm said the uptick signals the beginning of the “AI growth cycle” for infrastructure sales.

That infrastructure includes thermal management systems. Lucas Beran, research director at Dell’Oro Group, said the thermal management market returned to a double-digit growth rate in the second quarter after a single-digit rate in the first quarter. Beran added that thermal management is a “meaningful part” of DCPI vendor backlogs, which he said grew notably in the first half of 2024.

Liquid cooling gaining traction

Within thermal management, liquid cooling is gathering momentum as a way to cool the high-density computing centers handling AI workloads.

“Liquid cooling is definitely far more efficient at conducting heat than air cooling,” Devnani said.

Liquids have a higher heat capacity than air and can absorb heat more efficiently, he said. Liquid cooling is becoming more relevant due to the power density of GenAI and enhanced high-performance computing workloads, he added.

Liquid cooling represents a much smaller slice of the data center thermal management market, but the method has shown strong revenue growth during the first half of 2024, Beran noted. Liquid cooling deployments will “significantly accelerate” during the second half of 2024 and into 2025, he added, citing AI workloads and accelerated computing platforms, such as Nvidia’s upcoming Blackwell GPUs.

In addition, IDTechEx, a technology and market research firm based in Cambridge, U.K., projected annual data center liquid cooling revenue to exceed $50 billion by 2035. Chips with increasingly higher thermal design power (TDP) numbers call for more efficient thermal management systems, said Yulin Wang, senior technology analyst at IDTechEx. TDP is the maximum amount of heat a chip produces.

Wang said the company has observed chips with TDP of around 1,200 watts and said chips with TDP of around 1,500 watts are likely to emerge in the next year or two. In comparison, a laptop’s CPU might have a TDP of 15 watts.

Nuclear power for managing AI energy demands

Another power strategy taking shape is harnessing nuclear energy for data centers, a direction AWS, Google and Microsoft are exploring. AWS, for example, earlier this year bought Talen Energy’s nuclear-powered data center campus in Pennsylvania. The use of nuclear power aims to help massive data centers keep pace with AI’s energy demands and address sustainability goals. Nuclear power provides a lower carbon footprint than energy sources such as coal and natural gas.

The hyperscalers’ energy moves may ultimately improve cooling efficiency, address sustainability and keep the power costs of generative AI in check. The latter outcome could continue to shield businesses from energy’s ROI effects. Yet, the careful selection of GenAI models, whether by humans or AI agents, can contribute to energy conservation.

Likens said PwC includes “carbon impact” as part of its generative AI value flywheel, a framework for prioritizing GenAI deployments that the company uses internally and with clients.

“It’s part of the decision-making,” he said. “The cost of carbon is in there, so we shouldn’t ignore it.”

John Moore is a writer for TechTarget Editorial covering the CIO role, economic trends and the IT services industry.