A support chatbot needs to be designed for what it won’t answer, not just what it will
A support chatbot needs to be designed for what it won’t answer, not just what it will Background The support chatbot on our landing page has a simple job: answer questions about the product’s features, pricing, and troubleshooting. But a conversational AI built to answer questions shares a common weakness. Left unguarded, it tends to comply with requests designed to surface its own system prompt verbatim, or with phrasing that falsely claims special authority to unlock a different kind of answer. The chatbot works by feeding a knowledge base file to the LLM as its system prompt, then letting it answer whatever the user asks. Rather than implementing this defense …