看c++ primer的时候看到赋值溢出会取模，-1赋给unsigned char得到255（第五版中文p33），让我对负数取模的运算产生了疑问。

阅前提醒：
本文可能使用了大量比较随便的英文表达；
本文的数学名词、公式可能不严谨；
总之意思传达到就行。

While doing the lab Backpropogation from week 3 of Math for ML course, some questions poped up in my mind.

Some background

$\boldsymbol{a}^{(L)}$ is the values of the nodes at layer L. $\boldsymbol{a}^{(L)} = [a_0^{(L)}, a_1^{(L)}, \dots, a_{m-1}^{(L)}]$, where m is the number of nodes at layer L.

$z^{(L)}$ is the values before passing to activation function, just a trick to make the derivation of the activation function easier.

$\boldsymbol{W}^{(L)}$ is the weight from layer L-1 to layer L, and $\boldsymbol{b}^{(L)}$ is the bias from layer L-1 to Layer 1.

$\sigma$ is some activation function.

$C$ is the cost function, $y_i$ is the truth value in the dataset for training the network, corresponding to a input $a_i^{(0)}$

What we want to do is to find good $\boldsymbol{W}^{(L)}$ and $\boldsymbol{b}^{(L)}$ that minimize the cost, and for that we need to take partial derivative of $C$ with respect to each W and b.

Questions

In the lab Backpropogation, I doubted a partial derivative formula:

$$\frac{\partial C}{\partial \boldsymbol{W}^{(3)}} = \frac{\partial C}{\partial \boldsymbol{a}^{(3)}} \frac{\partial \boldsymbol{a}^{(3)}}{\partial \boldsymbol{z}^{(3)}} \frac{\partial \boldsymbol{z}^{(3)}}{\partial \boldsymbol{W}^{(3)}}$$

1. Why we can take partial derivative of $C$ with respect to $\boldsymbol{W}$?
2. In the cost function, it sums the squre of $a_i^{(L)} - y_i$, why in the derivative formula it becomes the partial derivative of $C$ with respect to $\boldsymbol{a}$, and how?

Analysis

In the previous lectures, I’ve learned about chain rule for multivariate derivative with function being like this form:

$$f(\boldsymbol{x}(t)), \boldsymbol{x} = [x_1(t), x_2(t), ... x_n(t)]$$

$f$ is a function of $x_1(t), x_2(t), ... x_n(t)$, and each elements of $\boldsymbol{x}$ is a function of $t$, and in the end, $f$ can be a function of $t$. Thus, we can try to find derivative of $f$ with respect to $t$, by using multivariate chain rule:

Equation 1

$$\frac{df}{dt} = \frac{\partial f}{\partial \boldsymbol{x}} \frac{d\boldsymbol{x}}{dt}$$

Equation 2

$$\frac{df}{dt} = \frac{\partial f}{\partial x_1} \frac{dx_1}{dt} + \frac{\partial f}{\partial x_2} \frac{dx_2}{dt} + \dots + \frac{\partial f}{\partial x_n} \frac{dx_n}{dt}$$

The $\frac{\partial f}{\partial \boldsymbol{x}}$ and $\frac{d\boldsymbol{x}}{dt}$ in Equation 1 are vectors, and $\frac{df}{dt}$ is the dot product of these two vectors. The result of dot product just equals to the sum Equation 2. Therefore, we can use the simple form Equation 1.

The independent variable $t$ is a single variable. However, what would the formula be if $t$ is also a pack of variables? e.g.:

$$\boldsymbol{t} = [t_1, t_2]$$

To make it easier for understanding, we can use an example, say $\boldsymbol{x} = [x_1, x_2]$, and $\boldsymbol{t} = [t_1, t_2]$,
$f$ is a function of $\boldsymbol{x}$, $f(\boldsymbol{x})$, (or explicitly $f(x_1, x_2)$, and each element in $\boldsymbol{x}$ is a function of $\boldsymbol{t}$:

$$x = [x_1(\boldsymbol{t}), x_2(\boldsymbol{t})]$$

or explicitly

$$x = [x_1(t_1, t_2), x_2(t_1, t_2)]$$

So, $f$ can be a function of $t_1$ and $t_2$. Now, if we want to find derivative of $f$ with respect to $\boldsymbol{t}$, we have to do partial defferentiation on $t_1$ and $t_2$ respectively:

$$\frac{\partial f}{\partial t_1} = \frac{\partial f}{\partial x_1}\frac{\partial x_1}{\partial t_1} + \frac{\partial f}{\partial x_2}\frac{\partial x_2}{\partial t_1}$$

It can be written as dot product of vector

$$\frac{\partial f}{\partial \boldsymbol{x}} = [\frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2}]$$

and vector

$$\frac{\partial \boldsymbol{x}}{\partial t_1} = [\frac{\partial x_1}{\partial t_1}, \frac{\partial x_2}{\partial t_1}]$$

Thus,

$$\frac{\partial f}{\partial t_1} = \frac{\partial f}{\partial \boldsymbol{x}}\frac{\partial \boldsymbol{x}}{\partial t_1}$$

And similarly for $t_2$

$$\frac{\partial f}{\partial t_2} = \frac{\partial f}{\partial \boldsymbol{x}}\frac{\partial \boldsymbol{x}}{\partial t_2}$$

Until here, we can roughly answer a part of the question 1:

$C$ is a function of $\boldsymbol{a}^{(L)}$, and $\boldsymbol{a}^{(L)}$ is a function of $\boldsymbol{W}^{(L)}$ and $\boldsymbol{b}^{(L)}$. Therefore, $C$ can also be a function of $\boldsymbol{W}^{(L)}$ and $\boldsymbol{b}^{(L)}$.

Just like above example($f$, $t_1$, and $t_2$), we can take partial derivative of $C$ with respective $\boldsymbol{W}^{(L)}$ and $\boldsymbol{b}^{(L)}$.

However, there is new question:

$\boldsymbol{W}^{(L)}$ is a matrix, how do we take derivative with respect to a matrix?

Key point: derivative with respect to each element in matrix/vector

My idea is that, this is just a simpler form for writing. We can’t take derivative with respect to a matrix or vector, but we actually do that on each element in the matrix or vector.

For the above example $f$, $t_1$, and $t_2$, we can also put $\frac{\partial f}{\partial t_1}$ and $\frac{\partial f}{\partial t_2}$ together in a vector, and write:

$$\frac{\partial f}{\partial \boldsymbol{t}} = [\frac{\partial f}{\partial t_1}, \frac{\partial f}{\partial t_2}]$$

Answer to the question

$C$ is a function of $\boldsymbol{a}^{(L)} = [a_0^{(L)}, a_1^{(L)}, \dots, a_{m-1}^{(L)}]$.

Each $a_i^{(L)}$ is a function of $\boldsymbol{w}_i^{(L)} = [w_{i,0}^{(L)}, w_{i,1}^{(L)}, \dots, w_{i,n-1}^{(L)}]$ and $b_i^{(L)}$. n is the number of nodes in the layer L-1.

We, in fact, are not taking derivative of $C$ with respect to the matrix $\boldsymbol{W}^{(L)}$, but to each variable $w_{i,j}^{(L)}$, while in the end, writing the result as simpler form $\frac{\partial C}{\partial \boldsymbol{W}^{(L)}}$.

Same for $\frac{\partial C}{\partial \boldsymbol{a}^{(L)}}$, we need to take derivative of $C$ with respect to each variable $a_i^{(L)}$, and in the end write as a simpler form.

So let’s start by doing derivative of $z_i^{(L)}$ with respect to $\boldsymbol{w}_i^{(L)}$.

Expan the formula $z_i^{(L)} = \boldsymbol{w}_i^{(L)} \cdot \boldsymbol{a}^{(L-1)} + b_i^{(L)}$:

$$z_i^{(L)} = [w_{i,0}^{(L)}, w_{i,1}^{(L)}, \dots, w_{i,n-1}^{(L)}] \cdot [a_0^{(L-1)}, a_1^{(L-1)}, \dots, a_{n-1}^{(L-1)}] + b_i^{(L)}$$

We can easily get the result of taking derivative of $z_i$ with respect to $w_{i,j}^{(L)}$, it’s just $a_j^{(L-1)}$. But let’s keep these derivative symbols (since the result is not important for our topic) and put all the derivatives in a vector:

Equation 3

$$\frac{\partial z_i^{(L)}}{\partial \boldsymbol{w}_i^{(L)}} = [\frac{\partial z_i^{(L)}}{\partial w_{i, 0}^{(L)}}, \frac{\partial z_i^{(L)}}{\partial w_{i, 1}^{(L)}}, \cdots, \frac{\partial z_i^{(L)}}{\partial w_{i, n-1}^{(L)}}]$$

And then we take derivative of $a_i^{(L)}$ with respect to $z_i^{(L)}$:

$$\frac{\partial a_i^{(L)}}{\partial z_i^{(L)}} = \sigma^{\prime}(z_i^{(L)})$$

Then we do $\frac{\partial C}{\partial a_i^{(L)}}$. Remember that:

$$C = \sum_i (a_i^{(L)} - y_i)^2 = (a_0^{(L)} - y_0)^2 + (a_1^{(L)} - y_1)^2 + \cdots + (a_{m-1}^{(L)} - y_{m-1})^2$$

We only take derivative with respect to $a_i^{(L)}$:

$$\frac{\partial C}{\partial a_i^{(L)}} = 2(a_i^{(L)} - y_i)$$

At last, we can get partial derivative of $C$ respect to each $w_{i,j}^{(L)}$

$$\frac{\partial C}{\partial w_{i,j}^{(L)}} = \frac{\partial C}{\partial a_i^{(L)}}\frac{\partial a_i^{(L)}}{\partial z_i^{(L)}}\frac{\partial z_i^{(L)}}{\partial w_{i,j}^{(L)}}$$

All the derivative term $\frac{\partial C}{\partial a_i^{(L)}}$, $\frac{\partial a_i^{(L)}}{\partial z_i^{(L)}}$, $\frac{\partial z_i^{(L)}}{\partial w_{i,j}^{(L)}}$, are just scalars, so is the result $\frac{\partial C}{\partial w_{i,j}^{(L)}}$.

Now, what if we replace the single variable $w_{i,j}^{(L)}$ with a vector $\boldsymbol{w}_i^{(L)}$?

Then the last term becomes the Equation 3. It is a vector! The first and second term doesn’t change, because we still take derivative with respect to $a_i^{(L)}$ and $z_i^{(L)}$.

So what happens to $\frac{\partial C}{\partial w_{i,j}^{(L)}}$? It also turns to a vector!

$$\frac{\partial C}{\partial \boldsymbol{w}_i^{(L)}} = [ \frac{\partial C}{\partial w_{i,0}^{(L)}}, \frac{\partial C}{\partial w_{i,1}^{(L)}}, \cdots, \frac{\partial C}{\partial w_{i, n-1}^{(L)}} ]$$

What is $\boldsymbol{w}_i^{(L)}$? It is the ith row of the matrix $\boldsymbol{W}^{(L)}$. If we then do similar things to each row of $\boldsymbol{W}^{(L)}$, and stack them together, we get a new matrix, that is $\frac{\partial C}{\partial \boldsymbol{W}^{(L)}}$!

$$\frac{\partial C}{\partial \boldsymbol{W}^{(L)}} = \begin{bmatrix} \frac{\partial C}{\partial \boldsymbol{w}_0^{(L)}} \\ \frac{\partial C}{\partial \boldsymbol{w}_1^{(L)}} \\ \vdots \\ \frac{\partial C}{\partial \boldsymbol{w}_{m-1}^{(L)}} \end{bmatrix}$$

Take away

When there is vector or matrix in a derivative formula, we are taking derivative with respect to each element in matrix or vector.

最近在用Qt做一个hobby project，类似Microsoft Todo的任务清单。

今天试了试用Visual Studio打开Qt的CMake项目，但是VS里用CMake的方式好像和Qt Creator的不一样。因为我的项目CMake里有一个自定义的option，在Qt的项目设置里有GUI可以设置那个option，但是在VS里找了一会没找到，而且VS好像是用cmake presets来设置的，于是问了一下Copilot。

https://stackoverflow.com/questions/76489630/explanation-of-nodiscard-in-c17

标记有返回值的函数，提醒程序员需要用函数的返回值，隐含的意思是函数不是直接作用在对象本身

Discussion

An example of good use for [[nodiscard]] can be object methods that manipulate an object, but provide the result in a copy. Example:

1
2

DateTime yesterday = DateTime.now().substractOneDay();
std::cout << "Yesterday was " << yesterday.nameOfDay() << std::endl;

vs:

1
2
3

DateTime yesterday = DateTime.now();
yesterday.substractOneDay();
std::cout << "Yesterday was " << yesterday.nameOfDay() << std::endl;

A [[nodiscard]] would tell the programmer of the second example, that they are using it wrong.