Does Your Enterprise AI Keep Getting It Wrong?

In a standard language model query scenario, a user submits a prompt, it searches through its general knowledge (memory) and presents the most relevant answer it could find. Alternatively, in a retrieval-based setup, that model isn’t just relying on its own general knowledge it also has access to your enterprise’s specialist context, e.g., your policies, product documents, internal knowledge etc.

For the language model to analyse and understand these documents, the data within them needs to be chopped up into small pieces, called chunks. These chunks are then stored and organised in a vector database. So, when a query comes in, the system can retrieve the most relevant chunk/chunks from the vector database and feed them to the model to formulate the output. This is the basis of retrieval augmented generation (RAG).

It may sound simple in theory, but every step of that journey is a place where something can subtly fall apart, causing those confident wrong answers.

The user’s prompt is too vague.

If a user’s question is too general, the system will not be able to work out which chunks are actually relevant. Therefore, it either retrieves the wrong data or falls back on the model’s general knowledge. Think of this as; rubbish in, rubbish out.

The wrong language model

Sometimes the language model does not understand the prompt accurately enough because it is not the right model for the task. Not every model suits every task. It may be too small a language model to reason over a prompt and a pile of chunks, causing it to lose information and overflowing its memory. It may also be poor at following instructions and drift away from the context it was given. Then there’s its understanding of your native language or domain terminology, which may also be incorrect or weak. The bottom line is, the wrong choice of model will undermine even a well-built pipeline.

The model’s own knowledge overriding your organisation’s context.

This one catches people out constantly. Say you ask about annual leave entitlement. The context provided is your own HR policy, but the model’s vast, internet-scraped knowledge can barge in and hand you generic information it has found elsewhere. Meaning, your organisation’s documents lose the argument to the average of the internet.

The wrong chunks getting retrieved from the vector database.

This is the failure that is most common, most problematic, and the least understood. Your chunks are distributed across the vector database by meaning, and if the retrieval step is not confident about which chunks matter, the not so relevant chunks can sneak into the response. The answer may sound plausible, because it’s built from your own documents, but in the end, it’s been formulated by the wrong parts of them.

The source material is flawed.

Finally, this may seem obvious, but it happens! If your organisation’s source material is contradictory, outdated, or just simply incorrect, no amount of clever engineering will save you. The language model will always generate a wrong answer, so make sure that all the data you include is consistent.

Honestly, it is not that difficult, but it does require attention on several fronts at once.

Manage and train your users.

Everyone talks about it, but few actually do it. Users need to understand how the language models they are using reason and formulate its answer. To get the best response from a model, the prompt going in needs to be a good one. Remember, garbage in, garbage out.

Stay flexible with the models available to you.

You want the ability to test different language models and move between them with ease, so you can be sure you’ve got the right model for the task at hand. You do not want to use a weaker language model for a complex task. Conversely, you do not want to use an advanced model for a simple task, this will cost more, run slower and over complicate a task that just needed a crisp, one line answer. The goal isn’t consistently using the most advanced model available; it’s using the one that is best suited for the job. The ideal set-up is to save the heavyweights for the genuinely hard problems and letting faster, leaner models handle the high volume and routine tasks.

Regulate what knowledge the language model can access.

As I am sure you know, every model carries a huge store of general information that gets pulled from the internet, and in a retrieval-based setup, that isn’t always a good thing. This relates to the annual-leave problem I mentioned earlier; how your organisation’s documents can be lost in generic information found on the internet. The ideal solution is to have a flexible scale of what knowledge your models can access, not just an on/off switch. On one end, you can close the model off from its own knowledge entirely, only keeping its ability to understand language using semantic understanding. Short of that, you can regulate how the two knowledge bases interact, letting the model draw on its general knowledge when useful, but making your enterprise’s context the authority when there is a disagreement.

Control how your context is vectorised.

This is the most technical but critical piece, it is the hardest to see, and the easiest to overlook. To have reliable enterprise AI you need to be able to change the size of the chunks that your source documents get broken down into. Some tasks may require the chunking to be by section, while other tasks may require you to chunk by character count. Also, being able to control how many chunks are retrieved by the model allows it to adopt for each task. These settings massively affect the quality of what comes back to the user and most organisations never touch them.

Even a relatively simple retrieval-based use case needs to tick all these crucial factors off for it to work consistently.

Firstly, the user need to know what prompts to use in order for the language model to understand the task. There needs to be real flexibility around both the language models available to you and the knowledge those models can access. You need to ensure consistency in the material each model uses, how it chunks data and distributes that data across the vector database. Finally, you need the ability to configure how those vectorised chunks are returned back to the model, formulating the quality output.

Getting these factors right is the foundation needed for you to build and scale your organisation’s AI usage. By being aware of the areas where language models can go wrong, and how to fix them if they do, dramatically increases your chances at creating something that leads to vast improvements across your organisation’s processes for many years to come.

However, by getting them wrong, and you’ll keep having that moment I mentioned at the beginning. Receiving a confident, plausible but frustratingly wrong answer and wondering why the AI “just doesn’t work!”

The truth is it can work. It just needs to be built accurately.