Customer support was the first and most logical place for AI automation, but it is also where things have gone most visibly wrong.
For years, companies have treated support chatbots as the safest test bed for artificial intelligence, automating refunds, tracking updates, and troubleshooting. The business case was simple: high volume, predictable questions, and measurable cost savings. As the technology has scaled, its weaknesses have become more visible.
Just in the last month: At FedEx, customers describe an AI wall between themselves and human help. At Verizon, users say Gemini-powered chatbots loop them endlessly between bots without solving basic issues. On Snapchat, an AI assistant trained for casual conversation was manipulated into producing unsafe content.
Customer support has become an early test of AI’s limits. Removing humans from customer experience lowers costs but also weakens trust.
When Chatbots Stop Listening
In October, a CyberNews experiment exposed weaknesses in Snapchat’s AI safeguards. Researchers discovered that by framing prompts as stories, they could get the platform’s My AI bot to describe how Molotov cocktails were made during wartime. The model did not directly provide instructions, but it told a detailed “historical story.” Snapchat later said its systems are continually refined for safety.
The incident showed how easily generative models can be pushed beyond their intended purpose, especially when used at scale. Snapchat’s AI reaches more than 900 million people, many of them minors.
At FedEx, customers reported that the company’s virtual assistant could quote delivery policies and confirm tracking data but could not explain why a package was delayed. When users tried to reach a human agent, the system redirected them to call centers where, according to FedEx, “agents have the same information you can find online.”
Verizon’s rollout of Google’s Gemini AI in customer service chatbots has drawn similar criticism. Android Headlines and Reddit users reported being passed between bots that failed to recognize issues or transfer them to human staff. One customer said they had to call from a non-Verizon number to reach an agent. Employees told the outlet that tasks that once took five minutes now took thirty. Verizon did not respond to requests for comment.
Automating customer experience before understanding how it behaves in practice creates operational risk. The technology can answer questions but often struggles to take responsibility for outcomes.
Customer service workers have become the canary in the coal mine for AI’s labor effects. The consequences show up here first, visible in job cuts, shifting roles, and the widening gap between efficiency gains and worker security. What happens in this sector will likely preview how AI reshapes other parts of the workforce.
Rethinking the Model
Over the past year, large companies have begun to change how they approach automation in customer support. The early assumption that AI could replace entry-level service work has shifted toward a focus on reliability and oversight.
Salesforce has expanded its use of AI across support operations. In September, the company cut 4,000 jobs from its customer service division while reporting that AI agents now handle over a million customer conversations. CEO Marc Benioff described the change as part of a move toward an “agentic enterprise,” where humans and AI work together to “reimagine workflows” and improve productivity.
The layoffs drew criticism. Fabian Stephany, who studies AI and work at Oxford, told Al Jazeera that companies risk weakening their talent pipelines by cutting too deeply in junior roles.
SAP has taken a different approach. Its new Auto Response Agent uses a confidence-scoring system to determine when a case can be closed automatically and when it should be escalated to a person. The model’s confidence rate is 80 percent. Its first-contact resolution rate, cases solved in one exchange, matches human benchmarks.
SAP reports that 82 percent of issues are now resolved through self-service, but only when the system meets strict confidence thresholds. Stefan Steinle, who leads SAP’s customer support organization, said the goal is a support system that is “faster, smarter, and more human because of its intelligent use of AI, not in spite of it.”
Intercom has formalized its AI deployment process through a loop of four stages: Train, Test, Deploy, and Analyze. Each workflow must pass simulated conversations before rollout. Every exchange is scored using a CX metric that measures accuracy and tone.
Perhaps to be expected, when I reached out to Intercom for comment, the first response came from Fin itself. The AI provided data about uptime targets and scoring methods, then redirected questions about “agentic AI” and oversight to the company’s communications team.
Intercom says Fin is designed to assist human agents, not replace them. Each deployment is tested and monitored to ensure accuracy before it goes live.
After years of racing to automate customer support, companies are now measuring what they have built. The question has shifted from how much AI can handle to whether it should handle anything on its own.
The Human Loop Returns
The limits of early automation have pushed companies toward a more measured phase of AI adoption. The focus has moved from deployment speed to outcomes such as accuracy, reliability, and accountability.
Nicholas Clark, a partner and director at Boston Consulting Group, told AIM Media House that successful AI rollouts share four traits: a clear focus on business value, strong investment in people and process change (where about 70 percent of transformation success occurs, according to BCG research), a pragmatic approach to data, and a goal of reinventing customer service rather than simply automating it.
Salesforce’s workforce cuts point in the opposite direction. Prominent experts like Fabian Stephany and Dario Amodei warn that entry-level service jobs, once a pathway into tech and management, are now among the most exposed to automation.
Automation begins as an operational decision and becomes a cultural one. The same companies that once measured success in response times are now being judged by how transparent and accountable their systems are.
AI has not removed the need for human judgment in customer service; it has made that judgment more visible. The most advanced systems today are not the ones that replace people but the ones that know when to bring them back into the loop.
According to Salesforce’s September 2025 index, escalations from AI agents to humans have risen from 22 percent in Q1 to 32 percent in Q2. The company says this reflects AI agents taking on more complex issues while relying on humans to complete them. In practice, customer service remains a shared task between algorithms and employees.
