An independent audit of a system that replies to customer reviews: we examined the code and rules, measured reply quality on a live sample, and delivered a verdict — improve or rewrite — with a work plan
KEO sells on Wildberries, a major Russian marketplace, and customer reviews remain one of its main channels for talking to buyers. An outside contractor built an autonomous system to answer them: it generates draft replies that operators review and edit, and in some categories it sends replies automatically.
A reply to a review affects the product listing’s reputation, and a mistake like admitting fault without evidence or making an unnecessary promise costs money. So replies have to be precise and say nothing more than they should.
Operators were rewriting drafts more and more often, and the automation rate hadn’t grown in two months. Operator edits weren’t logged anywhere, so no one knew which rules had shaped a reply or what operators disliked about the drafts. The same mistakes kept coming back, even though the system had already been “trained” on those very examples.
KEO needed an independent assessment to answer two questions: should the system be improved or rewritten, and how can quality control be handed over to operators?
We hand-labeled a sample of 50 negative reviews into three categories: unchanged, minor edits, and substantially rewritten. The data was pseudonymized, so the sample contains no personal data.
We worked out how to hand quality control over to operators and dedicated a separate wave of improvements to it.
We examined the code, rules, and knowledge base on a copy of the system, without exporting any production data. For each edit, we identified the cause using the client’s taxonomy and found exactly what was teaching the system to say too much.
The executive got a jargon-free version of the report; the developer got one linked to specific files and lines of code. It includes a system map, prioritized findings, a verdict on time and cost, and an improvement plan.
Here’s the audit step by step: what we check at each stage and what the client gets.
First, we figure out how the system actually works. It turned out to be a pipeline, not an autonomous AI agent: it takes in a review, normalizes it, analyzes it with the model, selects rules in code, writes a draft, checks it, and sends the reply — in that order. The model is called in only two places; everything else is plain code. Replies reach Wildberries through a single path.
Next, we measure quality on live data. In the sample, 24% of replies went out unedited, 52% had minor edits, and 24% were substantially rewritten, so 76% were edited in total. The system’s own database confirms it: of 583 replies sent by operators, 441 had been changed.
Labeling shows why replies get edited. The top reasons are tone (39.5%) and unnecessary escalation, such as sending the customer to the seller chat (21.1%). Less common are the wrong scenario and the wrong next step (13.2% each), misreading the context (7.9%), and made-up details (5.3%). There are no legal or factual errors: the system’s hard guardrails do their job.
Finally, we name the main cause of weak replies and show where it hides: the examples in the rules teach the system to write overloaded replies. At the meeting, we walk through the deck and discuss next steps.
Also used: Operator dashboard — draft review · VPS with system-level protection — starting and restarting services · independent watchdog — health monitoring
The client got an independent verdict: improve the system, don’t rewrite it. The audit showed where reply quality breaks down and in what order to fix the problems.
An audit like this works for any AI system: chatbots, assistants, internal tools, and automations. It’s for companies that want to know how far they can trust their system and what to fix first.
We build solutions like this around your needs: turnkey AI products, AI agents and automation, integrations, and data.
Our servicesTell us about your project — we’ll talk it through and show you how to make this work for your business
We use cookies and Yandex Metrica to improve our website. Privacy policy
