A visitor may dislike an answer because it is wrong, incomplete, hard to read or unable to finish the task. Those problems need different fixes. Commission a feedback workflow that preserves enough context to investigate while avoiding unnecessary collection of private conversation data.
Related service: Custom AI chatbot development- 01Collect useful feedback
- 02Locate the failure stage
- 03Assign and fix
- 04Run regression checks
SECTION 01
Use categories that lead to action
Separate missing or outdated source material from retrieval failure, unsupported wording, inaccessible output and failed tool actions. Include a category for a correct answer that does not meet the user's practical need. Give reviewers an option to mark the report unresolved rather than forcing a confident diagnosis.
Microsoft's RAG evaluation guide separates retrieval and generation decisions. Use that separation to ask whether the necessary passage was available, retrieved and represented accurately. Keep source, prompt and model versions with the report so the reviewer does not unknowingly investigate a different system state.
SECTION 02
Illustrative example: a reported cancellation answer
A visitor marks a booking assistant's answer as unhelpful and writes that the cancellation link did not work. The policy explanation was accurate, but the link led to an expired session. Rewriting the policy prompt would not fix the problem. The issue belongs to the authenticated handoff path.
A second visitor reports that an exception is missing. Review finds that the approved source lacks that exception, so the business owner must decide whether the source should change. The agency can implement the approved update but should not invent a policy from a comment. Both reports become distinct tests with different owners.
SECTION 03
Close the loop without collecting everything
Agree which context is necessary: question category, answer reference, source versions and optional user explanation. Restrict access to any retained transcript and follow the approved retention policy. OWASP's logging guidance is useful for avoiding secrets and excessive personal data in operational records.
At handoff, ask for triage labels, ownership, status history and a way to link a fix to its acceptance result. Review trends by failure type instead of using a raw satisfaction score as proof of correctness. Automated grouping can assist reviewers, but consequential source changes and release decisions still need accountable human judgment.
SECTION 04
Frequently asked questions
Should every negative vote change the prompt?
No. Investigate the source, retrieval, interface and connected workflow first. The prompt may not be responsible for the reported problem.
Can we automatically train on feedback?
Do not assume it is suitable or trustworthy. Review permissions, quality, sensitive data and the intended improvement method before reuse.
How do we know a fix worked?
Reproduce the issue, define the expected behavior and rerun it with relevant regression cases. A closed ticket alone is not technical evidence.
PRIMARY REFERENCES
Sources and further reading
These references cover the standards, platforms, or published prices discussed in the guide. Worked examples and checklists are our editorial guidance.
- Design and evaluate a RAG solutionMicrosoft Learn
- Logging Cheat SheetOWASP
EDITORIAL METHOD
About this guide
We use AI to assist with drafting and editing. Catapult AI Work is responsible for the published content. Examples illustrate possible approaches; they are not client case studies unless identified as such.
Budget examples are not Catapult package prices. Check linked provider pages for current fees and plan limits before making a purchase.
Read the editorial policy