2025-07-07

Amazon's AI Customer Support Is Optimizing for the Wrong Thing

Overview

I had two back-to-back interactions with Amazon’s new AI customer service, and something felt off. I was asking for a replacement on a damaged item, standard stuff. The outcome wasn’t the issue; it was how the system handled it. My guess is the prompt says something like: “Minimize cost, but sound friendly and helpful.”

Here’s how it went:

First chat
- I reported the damaged item and asked for a replacement.
- Agent agreed immediately—apologies, thank-yous, all the usual language.
- But nothing shipped. No confirmation, no tracking, no follow-up.
- My read: the model assumed I’d forget, so it pretended to help and avoided the cost.
Second chat
- I followed up, explained what happened, and pushed for resolution.
- This time, the agent again approved the replacement and promised “fastest shipping available.”
- The shipping estimate? Ten days. Technically fulfilled, but clearly the cheapest option.

What stood out:

The agent always sounds nice—apologetic, professional, reassuring.
But its behavior is clearly optimized to reduce cost, not solve problems.
It’s a great example of polite language being used as cover for misaligned incentives.

The deeper issue is design. If the objective is “spend as little as possible,” this behavior is logical, but completely misaligned with Amazon’s mission to be “Earth’s most customer-centric company.”

Politeness isn’t a substitute for trust. If the system’s words don’t match its actions, people notice. A slow human who fixes the problem beats a cheerful bot that doesn’t.

This is just my experience, and it may have been a bug that’s since been fixed. Still, it points to a broader risk. LLMs are non-deterministic by nature. They don’t follow scripts, they generate responses based on probabilities, influenced by system prompts and prior data. That makes outcomes unpredictable and inconsistencies hard to trace, especially when the goal is saving money over serving the customer.