Frustrated Users Reveal The AI Failures That Accuracy And Compliance Tests Miss

Teams run accuracy benchmarks, bias audits, red-team exercises, and compliance reviews before an enterprise AI system ships. Only systems that clear all of those gates are declared ready. Then actual users arrive and hit failures that no pre-launch suite was built to catch. They swear at the system, demand a person, or press zero until the call transfers or they rage-quit and write a negative review. Most Americans say they would still rather deal with a person than an AI agent to begin with. Every one of those reactions is production telemetry about a system that is failing, and most organizations are collecting none of it, or not using it as a tool to continually improve user experiences.

Aaron Weller leads Privacy Innovation & Assurance at HP and has spent more than two decades in privacy and information security, including consulting leadership positions. Weller joined HP to build its privacy engineering function, a role that has since expanded to cover assurance and AI governance. He works across privacy review, product design, and AI risk, which puts him in the room when teams decide what a system is allowed to do and how anyone would find out if it stopped behaving.

"Every time a chatbot talks to a person, it should be configured to detect if the person is having a bad experience, detect frustration or use of profanity and then be ready to escalate that to a human. That's functionality that should be table stakes across the board," Weller said. The escalation he described depends on noticing that the person on the other end has stopped cooperating with the system, which is a condition software can be configured to detect in the moment and act on without a human reading every transcript.

• The failures tests miss: Weller described a chatbot built on top of HP's careers site to help candidates narrow down thousands of open roles. "If you put in that I'm a male and I'm looking for a job, and I'm a female and I'm looking for a job, you would expect it to give you the same search results," said Weller. “That’s table stakes, but that shouldn’t be the whole story. There’s an opportunity there to reinforce with the user that we don’t feed certain parameters through to the back-end engine to make decisions. It’s not just that the system works, it’s that it can explain to the user how it works in real time. Weller divides the thinking in in two: guardrails for the failures a team can anticipate, and monitoring for the ones it cannot, in addition to the use of transparency as another feedback loop to the user.

• The zero button as a metric: The same evidence exists for systems already in production. "How do you almost measure the feedback loop of, are people actually happy using this thing? How often, when you go through the telephone tree, how often do people just spam zero until they get to talk to a person? You should be able to detect that," Weller said. The pattern is familiar to anyone who has called a support line. A caller listens through seven or eight options, forgets which one applied, starts the menu over, and eventually presses zero to get past it. Others never reach a person at all. A team that counted those calls would know which parts of its system are failing and how often.

Collecting this kind of evidence means giving up the idea that testing ends at launch. Continuous monitoring and feedback belongs alongside pre-deployment testing rather than after it, and the reason has more to do with how these systems behave than with how carefully anyone tests them. Teams accustomed to shipping deterministic code are still working out what the equivalent test suite looks like for non-deterministic systems, and whether they have one at all.

• Rules run out: "You can't write 5,000 rules to try and cover every possible situation," Weller said. "You're never going to work all the bugs out, because every time you ask it the same question, it's going to give you a slightly different answer." Weller applies the same reasoning to launch decisions. "If you have a good process for gathering what the system is actually doing, you can be a little bit more relaxed on trying to cover every single edge case in a way that could block or delay launch," he noted.

• A moving definition of good: "If good is 'bad things don't happen', that's still not necessarily a great customer experience," Weller said. Most control frameworks measure the absence of harm, which establishes that nothing went badly wrong and little else. Weller holds his team to a different standard. "It's really the customer experience. Whoever the customer is who's using the tool, what should they reasonably expect, and how do we make sure they're going to be getting that experience?" he explained. "It's a little bit different when you're going from a traditional controls architecture, where I need to check the boxes on these 10 things, to more of a, 'but will the customer be happy'?" The second question changes what a team measures, and what it accepts as proof that a system works.

• Supervision, not deployment: Weller returned to an analogy he uses with teams weighing how much autonomy to grant a system. "If you treat the chatbot like an intern, are you going to let them go out and talk to your top 10 customers? No, you never would do that. You might put them under supervision and have somebody that knows what they're doing watching," he said. Profanity detection and escalation belong in every deployment, in Weller's view, with use-case-specific checks added on top. "The further down the stack you build the control, the broader reach it has."

Deciding which controls belong in every deployment requires the leads of the various governance functions to agree. Weller was candid about his experience on how agreement gets reached at most organizations: over the phone, between people who already know and trust each other. The arrangement works while those people are reachable and have time to talk, but it leaves nothing behind for whoever inherits the question later. Weller has his own team documenting its repeatable processes partly for that reason, to provide resilience and reduce key-person risk.

• Ad hoc doesn't scale: Two teams reached out to Weller in the same week to ask how he was approaching AI failure detection, and both aligned with his answer, an outcome that depended on him being reachable. "A lot of it, I find in practice, is very much ad hoc, person to person, across the leads of the various governance pillars. Which doesn't really scale," he said. He has ruled out the most common alternative. "We can't have a committee that meets every quarter, or even every month that discusses these things. The delay in taking action would really impact being able to achieve our goals" Guidance given in one of those conversations often goes unrecorded, and the team that needs it next has no way to find it. "We need to get better at not writing the 20-page memo, but writing the half-page executive summary," said Weller, describing a step toward the centralized coordination larger firms are building.

Weller sees a related gap at the point of sign-off, where the person carrying the accountability for failure is rarely the person who built the system. The technical team supplies the raw data to form the basis of an executive risk decision, but the half-page summary is what gets read and usually where the decision happens.

In Weller's argument, that summary has to cover more than the pre-launch test results. It has to say what the system is doing in practice, and how anyone would know if that changed. "You can't expect that the person who's accountable for signing off to know or care about all the technical details. They need to understand enough about the risk to know whether to proceed with launch or not," Weller concluded.