Turns Out, the First Use Case for AI is Also the Worst

Customer support was the first and most logical place for AI automation, and also where its failures have been most visible. For years, companies treated chatbots as a safe proving ground for artificial intelligence, handling refunds, order tracking, and troubleshooting. The logic was clear: high volume, repetitive questions, and measurable cost savings. As the systems scaled, their weaknesses became harder to ignore. Just in the last month: At FedEx , customers describe an AI wall between themselves and human help. At Verizon , users say Gemini-powered chatbots loop them endlessly between bots without solving basic issues. On Snapchat , an AI assistant trained for casual conversation was manipulated into producing unsafe content. Customer support has become an early test of AI’s limits. Removing humans from customer experience lowers costs but also weakens trust. When Chatbots Stop Listening In October, a CyberNews experiment exposed weaknesses in Snapchat’s AI safeguards. Researchers discovered that by framing prompts as stories, they could get the platform’s My AI bot to describe how Molotov cocktails were made during wartime. The model did not directly provide instructions, but it told a detailed "historical story." Snapchat later said its systems are continually refined for safety. The incident showed how easily generative models can be pushed beyond their intended purpose, especially when used at scale. Snapchat’s AI reaches more than 900 million people, many of them minors. At FedEx , customers reported that the company’s virtual assistant could quote delivery policies and confirm tracking data but could not explain why a package was delayed. When users tried to reach a human agent, the system redirected them to call centers where, according to FedEx, “agents have the same information you can find online.” Verizon’s rollout of Google’s Gemini AI in customer service chatbots has drawn similar criticism. Android Headlines and Reddit users reported being passed between bots that failed to recognize issues or transfer them to human staff. One customer said they had to call from a non-Verizon number to reach an agent. Employees told the outlet that tasks that once took five minutes now took thirty. Verizon did not respond to requests for comment. Automating customer experience before understanding how it behaves in practice creates operational risk. The technology can answer questions but often struggles to take responsibility for outcomes. Customer service workers have become the canary in the coal mine for AI’s labor effects. The consequences show up here first, visible in job cuts, shifting roles, and the widening gap between efficiency gains and worker security. What happens in this sector will likely preview how AI reshapes other parts of the workforce. Rethinking the Mode l Over the past year, large companies have begun to change how they approach automation in customer support. The early assumption that AI could replace entry-level service work has shifted toward a focus on reliability and oversight. Still, companies like Salesforce have expanded its use of AI across support operations. In September, the company cut 4,000 jobs from its customer service division while reporting that AI agents now handle over a million customer conversations. CEO Marc Benioff described the change as part of a move toward an “agentic enterprise,” where humans and AI work together to “reimagine workflows” and improve productivity. SAP has taken a different approach. Its new Auto Response Agent uses a confidence-scoring system to determine when a case can be closed automatically and when it should be escalated to a person. The model’s confidence rate is 80 percent. Its first-contact resolution rate, cases solved in one exchange, matches human benchmarks. SAP reports that 82 percent of issues are now resolved through self-service, but only when the system meets strict confidence thresholds. Stefan Steinle, who leads SAP’s customer support organization, said the goal is a support system that is “faster, smarter, and more human because of its intelligent use of AI, not in spite of it.” Intercom has formalized its AI deployment process through a loop of four stages: Train, Test, Deploy, and Analyze. Each workflow must pass simulated conversations before rollout. Every exchange is scored using a CX metric that measures accuracy and tone. Perhaps to be expected, when I reached out to Intercom for comment, the first response came from Fin itself. The AI provided data about uptime targets and scoring methods, then redirected questions about “agentic AI” and oversight to the company’s communications team. Intercom says Fin is designed to assist human agents, not replace them. Each deployment is tested and monitored to ensure accuracy before it goes live. After years of racing to automate customer support, companies are now measuring what they have built. The question ha

Turns Out, the First Use Case for AI is Also the Worst

When Chatbots Stop Listening

Keep reading. It's free.

Related stories