5.3 Pretrained Models, Fine-Tuning and RAG

Why pretrained models dominate real projects

Learning objective AI-3.1.4 (K2) asks you to summarize the use of pretrained models, fine-tuning, and retrieval-augmented generation (RAG). The practical starting point is economic. Training a new AI model from scratch is costly and time-consuming. Collecting enough labeled data, paying for compute, and waiting out training cycles is rarely the first move once a capable pretrained neural network already exists for a nearby task.

Testers should hear that sentence as a risk statement as well as a savings statement. Reuse is faster. Reuse also imports whatever the previous model already believes, including its blind spots. The rest of this section is about two reuse patterns—fine-tuning (change weights with new training) and RAG (do not change weights; change the prompt with retrieved documents)—and the fact that testing does not become optional because someone else’s training bill was already paid.

Fine-tuning: extra training on a pretrained network

Fine-tuning takes a pretrained neural network and adapts it to perform a new, different task. The key benefit is that it usually needs much less training data and training effort than building from scratch. You are not randomly initializing every weight. You are continuing training from a network that already represents useful features.

Mechanically, the pretrained model is fine-tuned by additional training with data specific to the new task. CT-AI lists three scopes of that extra training:

• The entire neural network can be tuned.
• Only specific layers, typically near the output end, can be tuned, leaving earlier layers closer to their pretrained state.
• Additional layers can be added and trained (a common pattern when the new task needs a new output head).

After this training, the model’s ML functional performance is evaluated. Based on those results, further fine-tuning may continue until acceptance criteria are met. Fine-tuning is still model generation in miniature: train, evaluate, maybe tune again. It is not a ceremony that skips metrics.

From a tester’s view, the scope of tuning changes the defect surface. If only the last layers move, earlier feature extractors still carry pretrained behavior—including pretrained failure modes. If the entire net moves, you have more capacity to fit the new task and more capacity to destroy useful pretrained structure or to overfit a small new dataset. If you add layers, you added new code-like structure that must be included in evaluation.

Task similarity decides whether fine-tuning is a small step

Fine-tuning success depends on the similarity between the original task and the new task. Small differences can produce highly effective fine-tuning. The syllabus contrast is meant to be memorable:

• Adapting a cat-breed image classifier to identify dog breeds is likely to work well. Both tasks are fine-grained visual recognition of animals. Low-level filters for edges, fur texture, and animal-like shapes remain useful. The output head changes from cat identities to dog identities.
• Adapting that same cat-breed classifier to spoken accents is much less effective. The gap is not a cute trivia fact. Images of cats and audio of speech are different modalities and different tasks. Pretrained visual features do not give you a cheap accent recognizer.

A second syllabus example sits closer to this exam’s world. Fine-tuning a large language model (LLM) for ISTQB-defined boundary value analysis (BVA) is a small change for the LLM and is readily achievable when the training data is good. The pretrained model already handles language. You are teaching a relatively narrow additional skill: how BVA is defined and applied in ISTQB terms. If the fine-tuning set is thin, contradictory, or full of non-ISTQB folklore about boundaries, the small-change story collapses. Data quality still governs the small change.

Use these examples as a similarity meter on the exam. Same family of inputs and a nearby output space: fine-tuning is plausible. Different senses, different outputs, distant goals: do not expect miracles from a few extra epochs.

RAG: keep the weights, enrich the prompt

Retrieval-augmented generation (RAG) is an alternative to fine-tuning when you need task-specific knowledge inside an LLM-style system. The idea is to provide data sources specific to the required task. Those sources are transformed into a searchable format so they can be compared to the subject of the prompt. Once relevant documents are identified, they are incorporated into an enhanced prompt, which is passed to the LLM. With more pertinent information in the prompt, the response is likely to be more precise.

The sentence CT-AI wants you to be able to repeat in your own words: with RAG, no change is made to the pretrained model. You are not updating weights. You are retrieving and inserting context. If the document store is wrong, stale, or incomplete, the model can still sound fluent while citing yesterday’s procedure. If retrieval returns the wrong neighbor documents, the enhanced prompt is an enhanced mistake.

Tester-relevant failure modes for RAG are therefore retrieval failures as much as generation failures:

• The searchable form of the sources may drop tables, units, or warnings.
• Similarity search may retrieve a look-alike document from the wrong product line.
• The enhanced prompt may overflow or bury the critical clause.
• Confidential sources may leak into answers if retrieval is too broad.

None of those require a weight update to go wrong. That is the point of the pattern, and the point of testing it.

You can combine RAG and fine-tuning

A pretrained model can use RAG, fine-tuning, or both together. They are not enemies. Fine-tuning can teach a style or a skill (how ISTQB BVA is talked about). RAG can supply the current local facts (this year’s internal examples, this product’s limits). Combining them can improve performance. Combining them also combines failure modes: you can inherit pretrained bias, add fine-tuning artifacts, and then retrieve a bad document on top.

A practical selection sketch for exam scenarios:

Inherited bias and vulnerabilities still require testing

Typically, any biases or vulnerabilities in the pretrained model will carry over to the new model. Fine-tuning does not magically audit the pretraining corpus. Frozen weights in RAG do not magically remove biased associations; they can still appear in generations, and retrieved documents can add their own skew. Testing is necessary to confirm that the system performs reliably and fairly on the new task.

That last sentence is the tester’s punchline for AI-3.1.4. Reuse reduces training cost. It does not retire acceptance criteria, fairness checks, robustness checks, or task-specific oracles. If a pretrained vision model was weak on a lighting condition that still exists in your warehouse, fine-tuning on a small indoor set may not fix it. If a pretrained LLM produces unsafe advice, RAG will not automatically refuse unless your retrieved policy documents are found, inserted, and actually followed.

When you review a design that says we used a famous pretrained model, therefore we can skip independent test, you are looking at a workflow skip, not a savings. Name what was reused, what was changed (weights, prompt, both), how similar the new task is, and what inherited harm you still have to hunt.