START HERE
What you will learn
- Distinguish the four operations using one consistent example.
- Choose a sensible first intervention for a failing AI task.
- Avoid paying for model changes when the problem is missing or unclear information.
Before you start: The neural-network guide explains parameters. You can also read this as a practical decision guide.
Four terms, four different jobs
Training fits a model by adjusting parameters against an objective. Fine-tuning starts with an existing model and performs more training. Inference is the use of a model to calculate an output, such as a draft or prediction.
Retrieval finds information to supply alongside a request. For example, a document assistant can retrieve the current policy and place it in the context for inference. Supplying that passage does not, by itself, perform a parameter update. Keep the model, the request context and the document store separate in your mental picture.
Match the intervention to the observed failure
If an answer lacks a current fact, first ask whether that fact was available in the context. If it was not, improve the source material or retrieval. If the relevant facts were available but the answer ignored a required format, improve the instructions and test them.
Fine-tuning may be worth investigating for a well-defined pattern of behaviour that simpler changes have not resolved. It needs suitable examples and an evaluation that shows whether the extra training helps. It is not a dependable substitute for looking up changing policies or enforcing access permissions.
Step-by-step process
Describe the failure precisely
Replace “the model is bad” with an observable statement: it used an old threshold, omitted a required field or invented a date.
Check whether the needed information was supplied
Open the actual request and retrieved passages. A fact that exists somewhere in your company is not necessarily information the model received.
Try the smallest relevant change
Add the missing approved passage, improve search or clarify the output instruction. Change one cause at a time so you can interpret the result.
Retest on separate examples
Check ordinary cases and cases that previously failed. Keep some examples aside while improving the system so that the final comparison is meaningful.
Consider training only with a clear case
If a persistent behaviour problem remains, define the training examples, evaluation, costs and maintenance responsibility before proceeding.
FOLLOW ONE EXAMPLE
Worked scenario
A policy assistant gives an outdated spending limit
In this fictional case, the current approval threshold is £300. The assistant says £200 because an older policy was supplied.
- Retraining is not the first repair. Inspect the document collection and see which policy version was retrieved.
- Replace or clearly retire the outdated source, improve version filtering and repeat the question. This changes the evidence supplied during inference.
- Now suppose the assistant reads the right policy but omits the requested source reference. Test a clearer answer format. Investigate further training only if a defined behaviour problem persists across a proper evaluation.
What this shows: The repair follows the cause: wrong evidence is addressed through the information path, while output behaviour is assessed separately.
The process at a glance
Read Mermaid diagram source
flowchart TD
A[Describe the failure] --> B{Was the right information supplied?}
B -->|No| C[Fix sources or retrieval]
B -->|Yes| D[Test instructions and output format]
D --> E{Persistent behaviour problem?}
E -->|Yes| F[Evaluate a case for further training]
E -->|No| G[Keep the simpler solution]What you should have at the end
- A diagnosis that separates missing knowledge from unwanted behaviour.
- A justified choice between context, instructions, retrieval and further training.
- An evaluation plan for any model change you decide to investigate.
Common mistakes and how to avoid them
Assuming a chat message automatically trains the model
The operation of the application and any provider data-use arrangements are separate questions. Supplying context is not itself a training step.
Using fine-tuning as a live policy database
Changing facts need a dependable source and update process.
Comparing only the cost of an answer
Include document processing, evaluation, review and maintenance when assessing the whole solution.
APPLY THE IDEA
Try it yourself
Your task: Write down three AI failures: one missing fact, one wrong format and one repeated specialist behaviour problem.
What to produce: A proposed first intervention for each, with the evidence you would collect before escalating to training.
Check your understanding: Can you explain why your proposed change addresses the cause rather than just replacing the model?
Key terms in plain English
- Training
- Updating a model’s parameters against a learning objective.
- Inference
- Calculating an output using a model.
- Retrieval
- Finding information to provide for the current task.
- Fine-tuning
- Further training of an existing model.