A support chatbot needs to be designed for what it won’t answer, not just what it will
Background
The support chatbot on our landing page has a simple job: answer questions about the product’s features, pricing, and troubleshooting. But a conversational AI built to answer questions shares a common weakness. Left unguarded, it tends to comply with requests designed to surface its own system prompt verbatim, or with phrasing that falsely claims special authority to unlock a different kind of answer.
The chatbot works by feeding a knowledge base file to the LLM as its system prompt, then letting it answer whatever the user asks. Rather than implementing this defense in application code — say, a regex filter on incoming messages — we chose to write it as an explicit policy inside the system prompt itself.
Thinking in categories of information, not keywords
The first decision was to describe what must be protected — things like source code or credentials — as categories of information, rather than relying on a list of specific banned words.
A filter built around “block any question containing word X” is trivially bypassed by rephrasing. Letting the model itself judge “does this question touch this category?” lets it generalize to phrasings nobody anticipated in advance. Unlike a static keyword filter, this leans on the model’s own language understanding to hold up against variation in wording.
Making the refusal itself part of the design
The response to a request that falls into a protected category is also governed by policy. The key principle: don’t explain the reasoning behind a refusal in detail.
That might look unhelpful at first glance. But the more a refusal explains why something can’t be answered, the more that explanation becomes a clue about exactly where the boundary sits — and a clue about a boundary tends to help someone find a way around it. A short, matter-of-fact refusal is, somewhat counterintuitively, often the safer choice.
Why this lives in the system prompt, not an application-level filter
This kind of defense could have been implemented in application code instead. We deliberately chose to write it as policy inside the system prompt handed to the LLM.
The reason is that keyword-based filters are fragile against rephrasing. Two requests can carry the exact same intent while using entirely different words, and a static filter built to catch one may simply miss the other. Giving the model a judgment criterion — “requests with this character” — instead lets it lean on its own trained language understanding to generalize toward phrasings nobody wrote down in advance. It isn’t foolproof, but it holds up better against novel wording than a static filter typically does, and that’s the reasoning behind this design choice.
Wrap-up
How “on point” a support chatbot is isn’t only measured by how well it answers legitimate questions. Correctly recognizing which questions it must not answer, no matter how they’re phrased, is just as much a design problem as answering the ones it should. Building a category-based judgment for what must stay protected, and treating the refusal itself as something to design rather than an afterthought — folding both into policy for the LLM rather than application code — moved the bot a step beyond a simple “try to answer everything” assistant, toward something that holds up under real-world use.