6.3 Structure of Deep Neural Networks

Artificial neural networks were first sketched as a crude picture of a brain: many simple units, densely interconnected, passing signals. Biological neurons fire; artificial neurons (also called nodes) produce activation values. The analogy is a doorway, not a physiology exam. Learning objective AI-3.4.1 (K2) asks you to explain the structure and working of a deep neural network — layers, the arithmetic inside a neuron, and how training nudges weights — so you can test and talk about these models without treating them as magic.

From a single-layer perceptron to depth

The single-layer perceptron is an early artificial network with just one layer of adaptive weights. It is a supervised learner for binary classifiers on linearly separable problems: the two classes can be divided by a straight line (or a hyperplane in higher dimensions) in the input space. The syllabus example is email: a perceptron can separate spam from not spam when a linear cut in feature space is enough — perhaps a weighted mix of “urgent prize” tokens versus known-contact tokens.

If the truth is a curved or interlocking boundary, a single perceptron cannot draw that curve. That limitation is why most current networks are deep: they comprise several layers. Stacking layers lets later neurons combine the decisions of earlier ones, so the overall mapping can be nonlinear even though each unit is a simple weighted sum plus a nonlinearity.

Fully connected networks, where every neuron in one layer feeds every neuron in the next, can be viewed as multi-layer perceptrons. When testers say “MLP,” they usually mean that fully connected stack, not a single perceptron.

Three kinds of layers

A deep net is typically described with three layer roles:

• The input layer receives raw (or already prepared) features. For a camera model this might be pixel values. Those input units usually do not run the same “weighted sum plus activation” recipe as hidden neurons; they are the place the numbers enter.
• Hidden layers sit between input and output. They are made of artificial neurons. In many common fully connected architectures, each neuron connects to every neuron in the next layer. Successive layers may have different widths: 784 inputs, then 128 hidden units, then 64, then 10 outputs is a perfectly ordinary shape. Width is a design choice, not a law of nature.
• The output layer hands a result to the outside world. For an image task that might be a number indicating the likelihood that the input is a cat (or a vector of class scores that a later step turns into a label).

Information flows from input toward output. Each hidden layer transforms the representation: edges, then textures, then object-ish patterns, in the folklore of vision nets. You do not need to memorize a biological story for those stages. You do need to know that computation happens at neurons after the input layer, and that depth is the “several layers” part of the definition of a deep network.

What one neuron actually computes

Skip the input layer. For every later neuron:

1. Take the activation values of the connected neurons in the previous layer.
2. Multiply each of those activations by that connection’s weight. Weights are independent; two neurons in the same layer do not have to share a weight to the same downstream unit.
3. Add those products — a weighted sum.
4. Add that neuron’s individual bias — a learned numeric offset. A bias lets the unit activate even when incoming activations are small, or stay quiet until the weighted sum clears a higher bar.
5. Pass the total through a nonlinear activation function (ReLU, sigmoid, tanh, and others exist; the syllabus does not require you to recite derivative tables). The result is this neuron’s activation value, which becomes an input to the next layer.

Different activation functions yield different activation values from the same weighted sum. That nonlinearity is not decoration: stacked linear maps without it would collapse into one linear map, and depth would buy you almost nothing.

Name collision you must not fail: this bias is a parameter inside the neuron. It is unrelated to dataset bias discussed with data quality (syllabus 5.1.2). Dataset bias is about unrepresentative or systematically skewed training examples. Neuron bias is ... + b in the unit’s arithmetic. If an exam stem says “bias,” read the surrounding words: weights and activations mean the numeric offset; sampling and labels mean the data-quality problem.

Initialization, forward pass, loss, and epochs

At the start of training, weights connecting neurons, and each neuron’s bias, are typically initialized to small random values. Biases are sometimes initialized to zero. Random small weights break symmetry so that hidden units do not all compute identical functions from the first step.

Then the training data is passed through the network. Each neuron after the input layer runs its activation recipe. That trip from features to outputs is the forward pass. The generated output is compared with the known correct result (the label). The difference is summarized as error or loss. That loss is fed back through the network to adjust weights and biases, shrinking the difference on subsequent passes. You do not need the calculus of backpropagation for this K2 objective; you need the control story: forward to get an output, compare, feed the loss back, nudge parameters.

As more training data is fed through, each full pass through the training dataset is an epoch. After many epochs, weights and biases have usually moved a long way from their tiny random starts. Training ends when the output is considered good enough — good enough on the validation criteria you already contrasted in 6.1, not “good enough because the training loss hit zero and we never looked at hold-out.” Stopping is a product and risk decision: continue until the network is useful, without pretending that “one more epoch” is free of overfitting.

What testers should be able to point at

When a team says they shipped a deep net, you should be able to ask:

• How many hidden layers, and are they fully connected or a different pattern (convolution, attention) that still has neurons, weights, and activations?
• What is the input (pixels, tokens, sensor channels) and what does the output mean (cat likelihood, class scores, a yes/no spam score)?
• Are we looking at a perceptron-style linear separator or a deep stack that can form nonlinear boundaries?
• Were weights and biases initialized in the usual small-random way, and was training measured in epochs with a defined stopping rule?
• When someone mentions bias, do they mean the neuron offset or dataset bias (5.1.2)?

A perceptron that draws a straight spam/not-spam line is a legitimate, testable model for linearly separable mail. A deep fully connected net that maps pixels to a cat score is a different structure: several layers, hidden widths that may differ, nonlinear activations, and iterative epoch-wise updates. AI-3.4.1 is the vocabulary and the mechanism, so later coverage measures (neuron activations, boundaries, unused layers) have somewhere to attach.

A tiny numeric sketch of one neuron (not an extra formula to memorize)

Suppose a hidden neuron has two incoming activations, 0.5 and 0.2, weights 0.8 and −0.4, and bias 0.1. The weighted sum plus bias is 0.5 × 0.8 + 0.2 × (−0.4) + 0.1 = 0.4 − 0.08 + 0.1 = 0.42. A ReLU-style activation that zeros negatives and keeps positives would emit 0.42 as this neuron’s activation. A saturating function might squash 0.42 into a smaller range. Either way, that number is what the next layer sees. Multiply this picture by every neuron in every hidden layer and you have a forward pass. Compare the final output with the label, feed the loss back, and the 0.8, −0.4, and 0.1 will shift a little before the next example or the next epoch.

Hands-on courses often show a perceptron learning a logical AND by walking weights across epochs until error hits zero. That exercise is the same loop: initialize, forward, compare, update, repeat for epochs. Depth adds layers and nonlinearities; it does not replace the loop.