Placing corrections exactly where an LLM is tempted to sound plausible instead of right
Background
The support chatbot on our landing page feeds the contents of a knowledge base to an LLM as its system prompt, then lets it answer whatever a user asks. Running it in practice surfaced something a generic instruction like “answer accurately” doesn’t cover: the same plausible-sounding mistake, repeated consistently, for certain specific kinds of questions.
An LLM has absorbed a huge amount of general technical knowledge. That’s exactly why, the moment it’s asked something, it tends to combine that general knowledge into an answer that sounds thoroughly convincing. The tricky part is that the answer is convincing enough that being wrong is hard to notice.
Example 1: guessing a hosting provider’s SSH hostname format
When a user asks “I don’t know my SSH hostname,” an LLM is tempted to lean on general patterns and offer something like “it’s usually formatted like such-and-such.” And to be fair, some hosting providers do have typical hostname conventions.
But that’s a dangerous kind of helpfulness. The actual format varies by hosting provider, plan tier, and even contract vintage — sometimes even within the same provider across different plans. An LLM offering “the common format” has no guarantee whatsoever that it matches this particular user’s actual contract. If it’s wrong, the user ends up with a new kind of confusion on top of the original one: “I looked exactly where the app told me to, and it’s not there.”
The correct answer is “check your hosting provider’s control panel for the SSH connection details, and copy the exact string shown there” — not a guessed format sample. But the judgment that not offering a specific format is the right move isn’t something a generic “answer accurately” instruction reliably produces on its own.
Example 2: a plausible-sounding but wrong “you need to convert this” instruction
Another case involves the format of private keys issued by one hosting provider we support. These keys are issued in PKCS#8 format, and since version 1.6.3 the app reads that format natively, with no extra steps.
But if an LLM is only given the question “my private key won’t load,” it can reach for general SSH knowledge and suggest something plausible-but-wrong, like “you might need to convert it to PuTTY format (.ppk)” — a step that’s both unnecessary and beside the point for the current version. This is the same pattern: a suggestion that sounds helpful and technically coherent, but is simply incorrect for this product. The correct answer is the far simpler “update the app to the latest version, then point it at the downloaded key file as-is” — but the LLM finds the general-purpose conversion procedure more plausible-sounding.
Why a generic instruction alone isn’t enough
Instructions like “answer accurately” or “don’t guess when you don’t know” can be written once, up front, in the system prompt. But they’re too abstract to catch individual traps like these. From the model’s perspective, both guessing a hostname format and describing a key-conversion procedure are “generally correct technical knowledge” — nothing in an abstract instruction signals that this specific product’s circumstances make that generally-correct knowledge wrong here.
The fix: place a targeted correction right next to the trap
The approach we took was to attach a note — invisible to the end user — directly at the exact spot in the knowledge base where a particular question and fact live. Right next to the relevant FAQ entry, a note says, in effect: “when this question comes up, there’s a plausible-sounding mistake the model tends to make here; the actual answer is this instead.”
Rather than stating one abstract “don’t hallucinate” rule in a single place, the idea is closer to administering a targeted inoculation at each specific spot where a mistake actually occurs. Whenever a new trap surfaces, a note gets added right at that spot. In practice, a pile of small, localized corrections does more for real answer quality than a single sweeping rule stated once at the top.
Wrap-up
An LLM’s confidence and its accuracy are often hard to tell apart. Users tend to trust specific, confidently-stated information more, which makes it worse when that confidence turns out to be misplaced. Rather than relying solely on generic instructions to improve accuracy, identifying the exact spots where mistakes actually happen and placing a targeted correction there — one trap at a time — is the unglamorous foundation that makes a genuinely knowledgeable-sounding chatbot actually knowledgeable.