3.3 Machine Learning Development Frameworks

What an ML development framework actually does

CT-AI v2.0 learning objective AI-1.1.7 (K2) asks you to summarize the functionality ML development frameworks provide. Think of a framework as a toolkit for building and training models, not as a guarantee of quality. Testers use the same toolkit the developers use. That is convenient and dangerous. Convenient, because evaluation helpers, data loaders, and export tools speed the work. Dangerous, because a green metric produced by the framework is still software, and software can implement a metric incorrectly.

Typical functionality falls into five groups. Learn the groups as a checklist you can apply to any toolkit, rather than as a brand list the exam will grade.

Data handling

Frameworks help load, preprocess, and manage the data used to train and test the model. That can include cleaning, formatting, and transforming examples into the tensors or tables the chosen model expects. From a testing view, every transform is a specification. If training images are resized, normalized, or augmented in the framework pipeline, inference must apply the compatible transform. A surprisingly large class of defects that look like model inaccuracy is actually a production path that skipped the training-time scaler.

Write those transforms down. If a loader drops corrupt rows in training but production accepts them, you have a silent distribution shift. If a tokenizer used in training is not the tokenizer bundled in the mobile build, you are not testing one system.

Model building

Frameworks offer libraries of ML algorithms and tools to design architecture: the type of model (neural network, decision tree, and so on), the number of layers and connections, and the mathematical operations inside the model. Testers should know which knobs were set, because those knobs are part of the configuration under test. Two runs that people describe as the same algorithm with different layer widths are different products. Architecture dumps, configuration files, and frozen graph exports belong in the evidence pack next to the metric.

Training and optimization

Frameworks provide algorithms that iteratively adjust internal parameters from training data toward a desired result, such as classification or ML regression. Some frameworks support distributed training across machines and can enhance or fine-tune pretrained models. Distributed training introduces extra failure modes: stale workers, mismatched seeds, and silent truncation of a shard. Fine-tuning a pretrained model introduces extra test obligations: you must know what the base model was, what data it must not memorize in your domain, and whether your fine-tune actually changed the behaviors you care about.

Optimization is not only faster loss. Early stopping, learning-rate schedules, and mixed precision are behaviors. If they are on in the trainer and off in a reproduction script, you will chase ghosts.

Evaluation

Frameworks offer tools to measure how the trained model performs on unseen data. Classification work commonly reports accuracy, precision, and recall. Regression work commonly reports error rates. These numbers are not self-proving. Testers should treat framework evaluation utilities as components under test. A precision function that averages the wrong way, ignores a class, or quietly drops rows with missing labels can hide defects. Independent checks (a confusion matrix from raw predictions, a second implementation, a hand-counted fixture of labeled cases) are how you keep the dashboard honest.

This is the point that separates a K2 summary from a slogan. Using the framework's evaluation tool is expected. Believing it without challenge is not. If the helper reports 97 percent accuracy while a minority class has zero recall, the defect may sit in the metric implementation rather than in some unexplained model mystery.

Deployment

Some frameworks can convert a trained model into a form that integrates with web applications, mobile devices, edge devices, or embedded systems. Export is another place defects hide. The notebook metric and the mobile runtime are different programs. If you only evaluate in the training framework, you have not evaluated the deployed artifact. Serialization flags, operator support on the target, and quantization at export time are all in scope for a tester who claims the shipped binary was checked.

Abstraction level: low-level versus high-level APIs

The same family of tools can expose different abstraction levels. A lower-level API gives more control over model building and requires more coding expertise. A higher-level API simplifies creation and offers fewer customization options. Neither level is more correct. Low-level control can encode a subtle numeric bug that a high-level default would have avoided. High-level defaults can freeze a behavior you did not intend, such as a shuffle, a padding value, or an implicit preprocessing step.

Testers should ask which API layer produced the artifact under test and which defaults shipped with it. Rapid prototyping through a friendly high-level interface is a selection factor, not a substitute for evaluation on unseen data.

General-purpose versus domain-specific frameworks

Some frameworks are general purpose and support a wide range of application areas. Others specialize in image recognition, speech recognition, language translation, or similar domains. A domain-specific toolkit may get you to a working prototype faster because loaders, augmentations, and pretrained backbones already assume that domain. It may also smuggle in domain assumptions (color space, sample rate, tokenization) that are wrong for your product. General-purpose kits demand more wiring and give you fewer hidden assumptions. Domain kits hide wiring and hide assumptions. Testers should surface those assumptions as explicit checks.

How teams should choose, and how testers should read the choice

Selection can depend on:

• the application area
• the need for a user-friendly interface for rapid prototyping
• configurability for complex models
• the expertise of the users
• deployment considerations, including resource-constrained environments
• level of community support
• ecosystem maturity

Those factors are product-risk factors. A framework with a thin community may ship metric bugs that nobody has filed. A framework that is weak on edge export will tempt the team to test in a desktop language and ship through a converter, which is exactly the gap you should close with artifact-level tests.

Worked example: a team reports 97 percent accuracy from the framework's evaluation helper. A tester rebuilds a confusion matrix from saved predictions and finds that one minority class has zero recall. The helper was averaging in a way that hid the failure, or it had dropped the class's rows. The defect was in the evaluation implementation and in over-trusting the toolkit. Independent OpenExamPrep teaching treats that skepticism as core tester craft: use the framework, then verify the framework.