Chapter 6 - Convolutional Neural Networks

Why do we need convolutional neural networks?#

Original intention: Images already have rich structures, which can be used by humans and machine learning models. By using these structures, the number of parameters in training can be reduced and the accuracy of the model can be improved.

For example, in the task of object detection in an image, a reasonable assumption is that the object position has spatial invariance. Convolutional neural networks systematically formalize the concept of spatial invariance, so that useful representations can be learned with fewer parameters based on this model.

Screenshot 2024-02-21 093549

The following two characteristics can be summarized:

Translation invariance: Regardless of where the detection object appears in the image, the front layers of the neural network should have similar reactions to the same image area, which is called "translation invariance".
Locality: The front layers of the neural network should only explore the local area of the input image and not overly concerned about the relationship between distant areas of the image, which is the principle of "locality". Finally, these local features can be aggregated to make predictions at the image level.

Mathematical representation#

$\mathbf{H}$ is the hidden representation of the image matrix $\mathbf{X}$ . $\mathbf{U}$ is the bias. $\mathbf{W}$ is a four-order weight tensor.

The index $(i, j)$ represents the pixel at this position. Therefore, the hidden representation in the fully connected layer can be derived as follows:

\begin{aligned} \left[\mathbf{H}\right]_{i, j} &= [\mathbf{U}]_{i, j} + \sum_k \sum_l[\mathsf{W}]_{i, j, k, l} [\mathbf{X}]_{k, l}\\ &= [\mathbf{U}]_{i, j} + \sum_a \sum_b [\mathsf{V}]_{i, j, a, b} [\mathbf{X}]_{i+a, j+b}.\end{aligned}

Where $\mathsf{W}$ to $\mathsf{V}$ is just moving the sum "center" to $(i, j)$ .

Translation invariance means that $\mathsf{U}$ and $\mathsf{V}$ do not depend on the value of $(i, j)$ . ~~This is easy to understand. At different $(i, j)$ , if $\mathbf{X}$ is the same (the object is the same), then the hidden representation obtained should also be the same.~~ Therefore:

[\mathbf{H}]_{i, j} = u + \sum_a\sum_b [\mathbf{V}]_{a, b} [\mathbf{X}]_{i+a, j+b}.

Locality means that we can mask information outside a certain range, that is, set $\mathbf{V}$ outside $[a,b]$ to 0

[\mathbf{H}]_{i, j} = u + \sum_{a = -\Delta_1}^{\Delta_1} \sum_{b = -\Delta_2}^{\Delta_2} [\mathbf{V}]_{a, b} [\mathbf{X}]_{i+a, j+b}.

The above equation is called a convolutional layer. $\mathbf{V}$ is called a convolutional kernel or filter.

Simply speaking, let's talk about the convolutional layer.
The operation expressed by the convolutional layer is actually cross-correlation.
In the convolutional layer, the input tensor and the kernel tensor produce an output tensor through cross-correlation.
The real convolution operation looks like this (but it doesn't matter):

\begin{aligned} (f * g)(\mathbf{x}) &= \int f(\mathbf{z}) g(\mathbf{x}-\mathbf{z}) d\mathbf{z}.\\ (f * g)(i) &= \sum_a f(a) g(i-a).\\ (f * g)(i, j) &= \sum_a\sum_b f(a, b) g(i-a, j-b). \end{aligned}

Screenshot 2024-02-21 093524

In fact, an image is a three-dimensional tensor composed of height, width, and color, such as containing $1024 \times 1024 \times 3$ pixels.
$[\mathsf{X}]_{i, j}\to[\mathsf{X}]_{i, j, k}\quad [\mathbf{V}]_{a,b}\to [\mathsf{V}]_{a,b,c}$

$\mathsf{H}$ also uses a three-dimensional tensor.
In other words, for each spatial position, we want to use a set of hidden representations instead of just one. This set of hidden representations can be imagined as a stack of two-dimensional grids.
Therefore, we can imagine the hidden representation as a series of channels with two-dimensional tensors.
These channels are sometimes also called feature maps, because each channel provides a set of spatialized learning features to the subsequent layers.
~~Intuitively, we can imagine that in the lower layers close to the input, some channels specialize in identifying edges, while others specialize in identifying textures.~~

To support multiple channels in the input $\mathsf{X}$ and hidden representation $\mathsf{H}$ , we can add a fourth coordinate in $\mathsf{V}$ , that is, $[\mathsf{V}]_{a, b, c, d}$ . In summary,

[\mathsf{H}]_{i,j,d} = \sum_{a = -\Delta_1}^{\Delta_1} \sum_{b = -\Delta_2}^{\Delta_2} \sum_c [\mathsf{V}]_{a, b, c, d} [\mathsf{X}]_{i+a, j+b, c},

where the index $d$ in the hidden representation $\mathsf{H}$ represents the output channel, and the subsequent output will continue to enter the next convolutional layer as a three-dimensional tensor $\mathsf{H}$ . Therefore, a convolutional layer with multiple channels can be defined, and $\mathsf{V}$ is the weight of this convolutional layer.

$(n_h-k_h+1) \times (n_w-k_w+1).$

Code Practice#

def corr2d(X, K):  
    """二维互相关运算

    Args:
        X (Tensor): 二维矩阵
        K (Tensor): Kernel

    Returns:
        Tensor: 二维互相关运算结果
    """
    h, w = K.shape
    Y = torch.zeros((X.shape[0] - h + 1, X.shape[1] - w + 1))
    for i in range(Y.shape[0]):
        for j in range(Y.shape[1]):
          # Hadamard product(Schur product, entrywise)
            Y[i, j] = (X[i:i + h, j:j + w] * K).sum() 
    return Y

Two-dimensional convolutional layer

class Conv2D(nn.Module):
    def __init__(self, kernel_size):
        super().__init__()
        self.weight = nn.Parameter(torch.rand(kernel_size))
        self.bias = nn.Parameter(torch.zeros(1))

    def forward(self, x):
        return corr2d(x, self.weight) + self.bias

Simple example

import torch
import torch.nn as nn
import torch.optim as optim

conv2d = nn.Conv2d(1, 1, kernel_size=(1, 2), bias=False)
X = X.reshape((1, 1, 10, 12))
Y = Y.reshape((1, 1, 10, 11))

lr = 1e-1  

optimizer = optim.SGD([conv2d.weight], lr=lr)

for epoch in range(100):  
    Y_hat = conv2d(X)
    l = nn.MSELoss()(Y_hat, Y)

    optimizer.zero_grad()
    l.backward()
    optimizer.step()

    print(f"epoch {epoch+1}, loss {l.item():.3f}")

In convolutional neural networks, for any element in a layer, its receptive field refers to all elements (from all previous layers) that may affect the calculation during forward propagation.

The receptive field may be larger than the actual size of the input. When any element in a feature map needs to detect input features from a wider area, we can build a deeper network.

Padding and Stride#

The output shape of convolution depends on the input shape and the shape of the convolution kernel. After applying multiple convolutions in a row, the output is much smaller than the input. The side effect is the loss of information at the edges of the image.
We can affect this by padding (redundant) information at the edges.
In addition, for redundant information with high resolution, we can improve it by stride (take multiple steps at a time).

Screenshot 2024-02-21 093454

In this way, the output dimension becomes $(n_h-k_h+1) \times (n_w-k_w+1).$ becomes
$(n_h-k_h+p_h+1)\times(n_w-k_w+p_w+1)。$

The padding and output dimensions are related by the equation $(n_h-k_h+p_h+1)\times(n_w-k_w+p_w+1)。$

The padding and output dimensions are related by the equation $$(n_h-k_h+p_h+1)\times(n_w-k_w+p