Feature hierarchy

AI from scratch · 11 of a series

Feature hierarchy: lesson transcript

1. A picture with things in it

Twenty-four by twenty-four this time, and four things: two filled squares, a lone vertical bar, and an L. Suppose you want a detector that lights up on the squares and stays dark on the rest. No 3×3 kernel can do that on its own: a square is bigger than nine pixels. So we build up to it.

2. Layer 1: edges

Four edge kernels, each followed by a bend that keeps only the positive side. Together they answer four questions at every pixel: is there a left edge here, a right edge, a top, a bottom? Four maps. Every shape in the picture shows up, because every shape has edges.

3. Layer 2: corners

Now convolve the edge maps, not the picture. A top-left corner is a place where a left edge runs downward and a top edge runs rightward, so the kernel is an L: a vertical strip on the left-edge map, a horizontal strip on the top-edge map, and a threshold that only both strips together can clear. Four such kernels, four corner maps. The bar has vanished.

4. Layer 3: a square

One more layer, one more combination. A 5×5 kernel over the four corner maps, asking for a top-left corner in its top-left, a top-right corner in its top-right, and so on. It clears its threshold only where all four line up: the centre of each square. The L has corners too, but not in that arrangement, so it stays dark.

5. How much each layer sees

Pick a pixel in the square map and trace back what it depends on. Its 5×5 corner-map patch came from 7×7 of edge maps, which came from 9×9 of picture. This growing patch is the layer's *receptive field*. Layer 1 sees a corner's worth of picture; layer 3 sees a whole shape. That is why a square can only be found three layers up.

6. Change the square

The square kernel was built for seven-pixel squares. Resize square A and watch the square map: at six or eight pixels the corners no longer sit where the kernel expects them, and the response fades or splits. A hand-built detector is brittle. A learned one is tuned by gradient descent to whatever sizes and shapes the training data actually contain, and there are usually hundreds of them.

7. Name it: a feature hierarchy

Edges, then corners, then shapes: each layer builds its features out of the layer below, and each sees a wider patch than the last. Trained convolutional networks arrive at this same ladder on their own. The first layer learns edge and colour detectors, the middle layers textures and parts, the last layers whole objects. Nobody writes the kernels; the hierarchy is what gradient descent finds.

8. Did it land?

Three layers, three kernel sizes.

Check your understanding

Layer 1 uses 3×3 kernels, layer 2 uses 3×3, layer 3 uses 5×5. How wide is the patch of the original picture that one layer-3 output pixel depends on?

Each layer adds (kernel size − 1) to the width: 3, then 3 + 2 = 5, then 5 + 4 = 9. Trace the box back in the receptive-field scene and it lands on a 9×9 patch, which is why a 7-pixel square could be recognised only at layer 3.