CNN Concepts
A convolution slides one small set of weights over the whole input. That weight sharing is a belief about the data — the same pattern matters wherever it appears — and it is the reason a CNN needs far fewer examples than a fully-connected net on pixels.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A fully-connected network on raw pixels memorises the training images and fails on new ones. What does a convolution change, and what does it assume about the data?
A quality team on a packaging line photographs every carton and wants to flag dented or torn ones automatically. They have about twenty thousand labelled photos, and an intern's first model — a dense network on flattened pixels — reached near-perfect training accuracy and is useless on the line.
Flatten the image into a vector and feed it to a fully-connected network. Every pixel gets its own weight to every hidden unit; the network is free to learn any pattern anywhere.
That freedom is the problem. The dense network has a separate weight for "dark pixel at position 4,712" and "dark pixel at position 4,713"; it has to learn that a crease at each of thousands of positions is the same thing, from examples, one position at a time. It never sees enough of any one position, so it memorises.
- That freedom is the problem. The dense network has a separate weight for "dark pixel at position 4,712" and "dark pixel at position 4,713"; it has to learn that a crease at each of thousands of positions is the same thing, from examples, one position at a time. It never sees enough of any one position, so it memorises.
- It learns the training set's accidents instead: the shadow of the fixed camera mount, the particular lighting of the day the damaged batch was photographed. Training accuracy is perfect and the model is reading the background.
- On the line, cartons sit a few pixels further left than they did in the training photos and every learned pixel-weight is now looking at the wrong pixel. Offline validation from the same session did not show this because the offset was the same in both.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a carton image shows damage. The label is a binary inspector verdict recorded at the time of the photo; the decision is whether to divert the carton for manual check.
- The output is a probability that a threshold turns into a diversion. A false diversion costs a few seconds of a worker's time; a missed dent reaches a customer.
- One example is a 224×224 RGB image — about 150,000 numbers — with one label. Cartons appear anywhere in the frame, at slightly different angles, under lighting that changes with the time of day.
- The damage that matters is local: a crease a few pixels wide, a torn corner. The same crease is damage whether it is at the top-left or the bottom-right of the frame.
- Twenty thousand images sounds like a lot until each one is a 150,000-dimensional point. A dense first layer with 512 units would have 77 million weights fitted to 20,000 examples.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A convolutional layer holds a small kernel — 3×3 or 5×5 weights per input channel — and computes the same dot product at every position of the input, producing a feature map that says how strongly that pattern is present at each location. Weight sharing means the crease detector learned at one position is the crease detector at all positions.
- Because the same weights are applied everywhere, shifting the input shifts the feature map by the same amount: convolution is translation-equivariant. Pooling (max or average over a small window) then discards exact position, which moves the network toward translation *invariance* — "a crease is present" rather than "a crease is present at pixel 4,712".
- Each layer sees a small window, but stacking layers grows the receptive field: a unit three 3×3 layers deep is looking at a 7×7 patch of the input, and after pooling, at a much larger one. Early layers learn edges and textures; later layers combine them into parts and objects. The parameter count barely depends on the image size — it is kernels × channels — which is why the same architecture fits 20,000 examples and 20 million.
One kernel, every position
The core operation is small. A 3×3 kernel is nine weights per input channel. At each position of the input the layer multiplies the nine pixels under the kernel by those weights, sums, adds a bias and applies an activation. Slide the kernel one pixel and repeat. The output is a map of "how much does this patch look like the pattern the kernel encodes".
The consequence is what matters: because the weights are shared across positions, every example teaches the kernel something regardless of where the pattern appeared. A dense layer has to learn a crease detector once per position from the examples that happened to have a crease there. A convolution learns one detector from all of them.
1def conv2d(image, kernel):2 # image: H x W, kernel: k x k — the SAME nine weights at every position3 H, W = len(image), len(image[0]); k = len(kernel)4 out = [[0.0] * (W - k + 1) for _ in range(H - k + 1)]5 for i in range(H - k + 1):6 for j in range(W - k + 1):7 out[i][j] = sum(image[i + a][j + b] * kernel[a][b]8 for a in range(k) for b in range(k))9 return out # a map of "how much does this patch match the kernel"10 11# shifting the image by one pixel shifts out by one pixel: equivarianceCount the learned parameters: nine, for any image size. A dense layer from a 224×224 image to a single unit has 50,176. The convolution is not a cleverer function — it is a much smaller hypothesis space that happens to contain the right answer for images.
Why the dense network memorised
The intern's model did not fail because it was too weak. It failed because it was too free. With 77 million weights and 20,000 examples, there are many settings of the weights that fit the training set exactly, and gradient descent finds one of them. Nothing in a dense layer prefers the setting that says "a crease anywhere is damage" over the one that says "these particular pixels on these particular photos".
The CNN restricts the hypothesis space to functions built from local, position-shared patterns. If the truth is in that space — and for a crease it is — the restriction costs nothing and removes almost all the ways to memorise. That is what an inductive bias is: a prior over which functions are plausible, paid for in flexibility, repaid in data efficiency.
The patterns that decide the label are local and mean the same thing at every position; position itself is not informative.
holds when Damage detection, object presence, texture classification — anything where the question is "is this pattern present" and the frame is not fixed.
breaks when The target depends on where something is — a label in the wrong corner, a component misaligned by a fixed offset. Pooling has thrown away exactly the information the label needs.
respond Keep the convolutional features but remove or reduce pooling, or add coordinate channels so position is an input; do not retrain the same architecture harder.
Every pixel has its own weight to every hidden unit. A crease at position A and the same crease at position B are unrelated features. Training accuracy near-perfect; validation on a new session poor.
One kernel per pattern, applied everywhere; pooling discards exact position. Training accuracy lower, validation on a new session far higher; the same architecture size for any image resolution.
The convolution cannot express "this pixel on this photo" as cheaply as "this pattern anywhere", so gradient descent lands on the generalising solution because it is the only one available. Fewer parameters is the symptom; the matched inductive bias is the reason.
The compute shape
A convolutional layer over a 224×224 image with 64 kernels performs on the order of a hundred million multiply-adds, all independent of each other. That is the ideal workload for a GPU — thousands of identical operations on neighbouring memory — and a poor one for a CPU, which is built for a few fast sequential threads (GPU Fundamentals, CPU or GPU for Inference).
This decides infrastructure before it decides accuracy. A CNN that runs the line at 30 frames per second needs an accelerator at the edge or a batched service with a latency budget; a model that runs on a CPU at two frames per second is a different product. The architecture choice is also a serving-cost choice (Inference Cost).
| Option | Quality | Latency | Cost | Interpretability | Data needed | Operational | Note |
|---|---|---|---|---|---|---|---|
| Engineered edge features + logistic regression | Fine for a rigid part under fixed lighting; brittle when the line changes. | ||||||
| CNN trained from scratch | Needs far more labelled images than twenty thousand to learn low-level filters well. | ||||||
| Pretrained CNN backbone, fine-tuned head | The pragmatic default; the backbone carries the filters, the data carries the task. |
caveat The quality scores assume images with local, translation-invariant structure. They say nothing about which model detects the shortcut the data contains, and the latency column depends entirely on whether a GPU is available at the point of inference.
How to build it
Most important first.
- Match the architecture to the structure you believe in. Local patterns that matter regardless of position are exactly the CNN's inductive bias; use it when that is true and not otherwise (Raw Features vs Learned Representations).
- Start from a pretrained convolutional backbone rather than random weights: the edge and texture filters of the early layers transfer across almost all natural images, and twenty thousand examples is plenty to adapt the last layers (Transfer Learning).
- Augment with the transformations production will produce — shifts, small rotations, brightness — so the training distribution contains the variation the line will supply. Augmentation encodes an assumption too: rotate by 180° and you have told the model orientation does not matter, which is false for a label that says "text upright".
- Split by capture session, not by image, so the validation set tests generalisation to a new day's lighting rather than to a neighbouring frame of the same batch (Group Split).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Recall of damage at the operating threshold on a validation set from capture sessions the model never saw — the number that maps to the decision, because a missed dent is the expensive mistake.
- The gap between training and validation loss across epochs. A dense network on pixels shows it opening immediately; a CNN with augmentation shows it staying narrow (Learning Curves).
- Do not measure accuracy on a random split of frames. Neighbouring frames of the same carton in train and validation is entity leakage, and the number will be excellent (Entity Leakage).
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The patterns that decide the label are local and mean the same thing wherever they appear in the frame — the translation-equivariance assumption weight sharing encodes.
- The scale, orientation and lighting of production images fall within the range the training set, plus augmentation, covered; a camera move or a new carton size breaks this silently.
- Position information the pooling layers discarded is not part of the target. If the target changes to something position-dependent, the architecture must change with it.
- Offline: hold out entire capture sessions and check recall at threshold on them; then shift, rescale and darken the held-out images and check that recall degrades gracefully rather than collapsing.
- Online: log a sample of production images with predictions and have inspectors label them weekly, so a camera move shows up as a recall drop within days rather than as customer complaints (Ground-Truth Delay).
- Over time: monitor the distribution of a few cheap image statistics — mean brightness, edge density — at the serving boundary. A step change is a physical change to the line before it is a model problem (Data Drift).
What can go wrong
- The camera is moved ten centimetres during maintenance. Translation equivariance handles the shift; the change in scale and perspective it does not, and the receptive fields that matched a crease now match nothing.
- The model learns a shortcut correlated with the label that is still local and translation-invariant — damaged cartons in the training set were photographed after being handled, and carry fingerprints. The CNN's inductive bias does not protect against this; only the data does.
- Pooling discards position, which is what you wanted for "is there a crease" and exactly wrong for "is the label in the correct corner". The same architecture, a different target, and the bias is now working against you.
- A CNN is a commitment to locality and translation. Where the structure is different — tabular columns with no spatial neighbourhood, a global property of the whole image — the bias helps nothing and a different model is cheaper and better.
- The compute shape is many small dense operations over a large input, which is what a GPU is built for and a CPU is not; training and often inference will need an accelerator (GPU Fundamentals).
- Interpretability drops from "which column" to "which patch": saliency maps show where the model looked, not why, and a fingerprint shortcut looks like attention to the carton (Explainability).
- "Neural networks are always better than the classical approach for images." A CNN is better than a dense network on pixels for the specific reason that its inductive bias matches image structure. For a fixed camera and a rigid part, a handful of engineered edge features and a logistic regression can be adequate, faster and inspectable — the question is the data and the latency budget, not the family.
- "The model reached near-perfect training accuracy, so the architecture works." Perfect training accuracy on 20,000 images is available to any model with enough parameters. The validation number on unseen sessions is the only one that says anything.
- "Augmentation is free extra data." Augmentation is a statement of invariances. Each transformation you add tells the model a distinction does not matter, and if it does — orientation, colour — you have trained it to ignore the label.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- DATA-SPECIFICThe advantage of convolution over dense layers is specific to data with local, position-independent structure — images, audio spectrograms, some sensor grids. On tabular data the columns have no neighbourhood and a convolution over them is meaningless; a gradient-boosted tree is the stronger default there.
- SIMPLIFIEDParameter counts and receptive-field arithmetic here are for the shape of the argument — 3×3 kernels, no stride or dilation, no depthwise or separable variants. Modern vision backbones add all of those and, increasingly, attention layers; the inductive-bias argument is the same.
- CONTESTEDA serious position holds that the convolutional inductive bias is a crutch that a large enough transformer on image patches does without, and that vision transformers trained on enough data match or beat CNNs while being easier to scale and to share with language models. That is right at web scale; at twenty thousand images the CNN's prior is worth a great deal of data, and a pretrained CNN backbone remains the pragmatic default for a small labelled set.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Signal processing — convolution as a filter applied to a signal is older than neural networks, and the intuition that a kernel is a matched filter for a pattern is the same one.