Is Inference the Future of Enterprise AI?

For much of the generative AI boom, the conversation revolved around training ever-larger models. Every new release from OpenAI, Anthropic, Google, Meta and xAI pushed the industry toward more powerful GPUs, larger clusters and record-breaking capital expenditure from hyperscalers.

That phase is far from over. But another shift is beginning to emerge.

As enterprises move beyond pilots and start deploying AI agents, copilots and customer-facing applications at scale, the infrastructure challenge is no longer just about building smarter models. It is increasingly about running them reliably, cheaply and with low latency.

Goldman Sachs Asset Management says this transition was one of the clearest takeaways from its recent meetings with technology companies in Silicon Valley. The firm's investment team believes enterprise AI demand is steadily shifting from model training to inference, the process of serving AI responses in production.

"The companies doing this at scale are using a tremendous amount more compute than what the typical company is using," said Brook Dane, Technology Equity Co-head at Goldman Sachs Asset Management, in a company blog post. According to one company Goldman met during the trip, the top 5% of enterprise AI customers already consume three times as many AI tokens as the median customer, and that gap continues to widen.

That trend matters because inference, unlike training, never really stops.

The Economics of Enterprise AI Are Changing

Training a foundation model is an intensive but relatively infrequent event. Once the model is deployed, every prompt submitted by an employee, every AI-generated customer response and every software agent executing a workflow become inference requests.

Unlike model training, inference creates a recurring operational workload.

An employee asking Microsoft 365 Copilot to summarize meetings, a customer interacting with an airline chatbot, or an AI coding assistant generating software all consume compute in real time. As those interactions multiply across thousands of employees or millions of customers, infrastructure requirements begin to look very different.

"The computing infrastructure that supports inference can be very different from the one that underlies training," noted Sung Cho, Technology Equity Portfolio Manager at Goldman Sachs Asset Management.

Instead of focusing solely on GPUs, enterprises increasingly need high-speed networking, low-latency memory, storage, cooling systems and interconnect technologies capable of serving AI workloads continuously.

That distinction is becoming increasingly visible across enterprise software.

Microsoft is embedding AI across Microsoft 365 and Azure. Salesforce is pushing Agentforce deeper into customer workflows. SAP continues to expand Joule across its enterprise applications, while ServiceNow, Oracle and Workday are all positioning AI agents as the next evolution of enterprise software.

Each successful deployment increases inference demand.

Infrastructure Vendors Are Already Seeing the Shift

The changing nature of AI workloads is beginning to show up in corporate earnings.

Arista Networks, whose switches connect thousands of servers inside AI data centres, reported first-quarter revenue of $2.71 billion, up 35.1% year over year. The company also raised its 2026 AI fabric revenue target to $3.5 billion, reflecting stronger demand from AI infrastructure deployments.

Its latest optical networking platform promises to reduce networking racks by up to 75% while shrinking data-centre floor space by 44%, illustrating how networking efficiency is becoming increasingly valuable as AI clusters grow.

Cho believes networking could become one of AI's biggest bottlenecks.

"As processing speeds increase, the bottleneck isn't necessarily semiconductors but how fast processors can speak to other processors," he said, adding that faster transmission speeds will increasingly require fibre optics rather than copper interconnects over the coming years.

Memory suppliers are seeing a similar trend.

High-bandwidth memory (HBM), which feeds data to AI accelerators significantly faster than conventional DRAM, has become one of the industry's most constrained components.

Micron has already said its HBM production is fully booked through 2026. Samsung and SK Hynix continue to allocate more manufacturing capacity to HBM as hyperscalers and AI chipmakers compete for supply.

SK Hynix recently said demand from agentic AI and inference workloads is expanding the market for premium DRAM and NAND products. The company also expects tight supply and sustained pricing strength to continue as enterprise AI deployments scale.

The impact extends beyond semiconductors.

Vertiv Holdings, which supplies power and liquid-cooling systems for AI data centres, ended 2025 with a $15 billion order backlog, up 109% year over year. In the first quarter of 2026, the company reported revenue growth of 30% and raised its full-year sales guidance to $13.5 billion to $14 billion.

For companies like Vertiv, the rise of production AI means servers operating around the clock, increasing demand for cooling, power management and energy-efficient infrastructure.

Compute Is No Longer the Only Constraint

The first phase of the AI race largely centred on access to GPUs. Today, infrastructure constraints extend much further.

Networking equipment, fibre optics, advanced packaging, memory bandwidth, power delivery and cooling are increasingly determining how quickly enterprises can expand AI deployments.

Dane described the current environment as "compute-constrained," but noted that the bottlenecks now extend across ASICs, memory and the suppliers supporting them.

Goldman argues that these shortages could make the infrastructure investment cycle more durable rather than shorter.

"Our ability to meet this inflection in demand might be less than what the market wants," Cho said. "If you can't meet demand over one year, then you have to meet it over three to five years. The spending becomes more durable and sustainable."

That optimism stands in contrast to broader debates about AI valuations and return on investment.

While infrastructure suppliers continue to report strong growth, many analysts remain cautious about whether enterprises are generating sufficient business value from AI to justify the pace of investment. Goldman Sachs Research itself has previously questioned how quickly AI spending outside the semiconductor ecosystem will translate into measurable productivity gains.

That tension reflects where the enterprise AI market stands today.

Companies are increasingly convinced AI belongs in production. What remains less certain is how quickly those deployments will deliver sustained economic returns.

The Next Phase of AI May Be Less About Models

The first chapter of generative AI rewarded companies that built the biggest models and accumulated the largest GPU clusters.

The next chapter could reward a different set of companies altogether.

If enterprise AI adoption continues to accelerate, the beneficiaries may increasingly include networking vendors, memory manufacturers, cooling specialists, and infrastructure providers that keep AI systems running every second of the day.

Goldman Sachs believes that the transition is already underway. Recent earnings from Arista Networks, Vertiv, Micron and SK Hynix suggest the industry's supply chain is beginning to reflect the same shift.

Whether enterprise inference grows quickly enough to sustain today's infrastructure spending remains an open question. But one thing is becoming increasingly clear: the future of enterprise AI will depend not only on who builds the smartest models, but also on who can run them at scale.

>

>