梯度下降法
\frac{d}{dx}g(x) = 2x - 2
令导函数等于 0，求出极值点，这个点是极大值还是极小值，通过极值点左右的增减性来判断（由导函数在区间范围内的正负判断）。通常我们会绘制一个增减表。
x:=x - \eta \frac{d}{dx}g(x)
\begin{split}
x&:=3-1(2\times3 - 2) = -1 \\
x&:=-1-1(2\times-1-2) = 3 \\
x&:=3 - 1(2\times 3-2) = -1\\
&\cdots
\end{split}
\begin{split}
x&:=3-0.1\times(2\times 3 - 2) = 2.6\\
x&:=2.6-0.1\times(2\times 2.6-2) = 2.3\\
x&:=2.3 -0.1\times(2\times 2.3 -2) = 2.1\\
&\cdots
\end{split}
初始移动得比较快，慢慢地移动会变得非常慢。
现在回到我们的目标函数。
E(\theta) = \frac{1}{2}\sum_{i = 1}^{n}(y^{(i)} - f_{\theta}(x^{(i)}))^2
\theta_0 := \theta_0 - \eta\frac{\partial E}{\partial \theta_0}\\
\theta_1 := \theta_1 - \eta\frac{\partial E}{\partial \theta_1}\\
\frac{\partial E}{\partial \theta_0} = \frac{\partial E}{\partial f_{\theta}(x)}\frac{\partial f_{\theta}(x)}{\partial \theta_0}
\begin{split}
\frac{\partial E}{\partial f_{\theta}(x^{(i)})} &= \frac{\partial }{\partial f_{\theta}(x^{(i)})} \frac{1}{2}(y^{(i)}-f_{\theta}(x^{(i)}))^2 \\&= (y^{(i)} - f_{\theta}(x^{(i)}))
\end{split}
\frac{\partial E}{\partial f_{\theta}(x)} = \sum_{i=1}^{n}(y^{(i)} - f_{\theta}(x^{(i)}))
有了
\frac{\partial E}{\partial f_{\theta}(x)}，
\frac{\partial f_{\theta}(x)}{\theta_0} 的计算就比较简单了。
\frac{\partial f_{\theta}(x)}{\partial \theta_0} =\frac{\partial}{\partial \theta_0}(\theta_0 + \theta_1x)=1
\begin{split}\frac{\partial E}{\partial \theta_0} &= \frac{\partial E}{\partial f_{\theta}(x)}\frac{\partial f_{\theta}(x)}{\partial\theta_0}
\\&= \sum_{i=1}^{n}(y^{(i)} - f_{\theta}(x^{(i)})) \times 1
\\&= \sum_{i=1}^{n}(y^{(i)} - f_{\theta}(x^{(i)}))
\end{split}
同理，
\frac{\partial E}{\partial \theta_1} = \frac{\partial E}{\partial f_{\theta}(x)}\frac{\partial f_{\theta}(x)}{\partial\theta_1}
\frac{\partial E}{\partial f_{\theta}(x)} 和前面一样，所以我们只需要计算
\frac{\partial f_{\theta}(x)}{\partial \theta_1} 的导数。
\frac{\partial f_{\theta}(x)}{\partial \theta_1} =\frac{\partial}{\partial \theta_1}(\theta_0 + \theta_1x)=x
\begin{split}\frac{\partial E}{\partial \theta_1} &= \frac{\partial E}{\partial f_{\theta}(x)}\frac{\partial f_{\theta}(x)}{\partial\theta_1}
\\&= \sum_{i=1}^{n}(y^{(i)} - f_{\theta}(x^{(i)})) \times x^{(i)}
\\&= \sum_{i=1}^{n}(y^{(i)} - f_{\theta}(x^{(i)}))x^{(i)}
\end{split}
通过上述的计算，梯度下降算法的表达式如下：
\begin{split}
\theta_0 &:= \theta_0 - \eta\sum_{i=1}^{n}(f_{\theta}(x^{(i)}) - y^{(i)})\\
\theta_1 &:= \theta_1 - \eta\sum_{i=1}^{n}(f_{\theta}(x^{(i)}) - y^{(i)})x^{(i)}
\end{split}
References：
《白话机器学习的数学》
粤公网安备44030502008569号