Gradient Descent

06-11-2026

The Context

After understanding what Neural Networks are and a general idea of what it means to "learn" in a network the next step involves understanding how to actualize our conceptual understanding. Precisely, we need to figure out the correct way to tune the weights and biases in these nets.

It starts with a desired output, $y(x)$ . This is simple and is just what we want the output to be. Next, we want to know how well we are doing in reaching $y(x)$ , so we introduce the idea of a cost function:

C(w,b) = \frac{1}{2n} \sum_{x}^{}||y(x) -a||^2

Once you digest the notation, what this means is pretty intuitive. We are summing the the difference between the desired output and the actual vector of results, $a$ and averaging them. The cost function $C$ is known in the statistics world as Mean Squared Error or MSE. Now, our goal is to minimize $C$ so that we are essentially as close to what we want from the neural network as possible. The technique in which we traverse to find this minimum is known as Gradient Descent.

Gradient Descent

The idea of gradient descent is to progressively iterate towards the global minimum of a cost function. Imagine a 3-dimensional plane, where two axes are $v_1$ and $v_2$ , and then the third axis is the output of the cost function. Naively, we can compute derivatives of the cost function with respect to $v_1$ and $v_2$ to find an answer. However, this only really works on trivial cases, and in modern ML applications there are certainly more than two input variables.

Then what is the answer? Well imagine that you could, on this 3-Dimensional plane, drop a ball anywhere and gravity would carry it to the local minimum of where you dropped it. This would be an easy way to find all the local minima and therefore the global minimum. This can be modeled mathematically:

\Delta C \approx \frac{\delta C}{\delta v_{1}}\Delta v_{1} + \frac{\delta C}{\delta v_{2}} \Delta v_{2}

So now all we need to do is find a way in which to choose the changes in the inputs to make $\Delta C$ negative. To simplify this mathematically, we will define the vector of changes to be $\Delta v=(\Delta v_{1},\Delta v_2)^T$ and the gradient of $C$ is the partial derivates of this: $(\frac{\delta C}{\delta v_{1}}, \frac{\delta C}{\delta v_2})^T$ which is also known as the gradient vector: $\nabla C$ . Thus, our original equation above can be rewritten as:

\Delta C = \nabla C \cdot \Delta V

Now lets introduce another variable to the system: a parameter known as the learning rate:

\Delta v = -\eta \nabla C

And now we can derive:

\Delta C= -\eta \nabla C \cdot \nabla C = -\eta ||\nabla C||^2

Since the magnitude, or length, of the gradient vector is always nonnegative, the change in $C$ will always be negative with the help of the learning function. Now, it should be clear that when we compute a value for $\Delta V$ , the "ball" in our mental image moves by $v - \eta \nabla C$ . Thus, by iteratively moving the ball, we'll keep decreasing C until we (hopefully) reach a global minimum.

Now onto the learning rate. Is this an arbitrary number or something we learn on? Well we need a value small enough so that $\Delta C \leq 0$ while also making it large enough so that we are being efficient with our computationSince if the steps are too small we are leaving time and compute on the table. All of this work can be generalize to inputs of size $m$ , so:

(\frac{\delta C}{\delta v_{1}},\dots, \frac{\delta C}{\delta v_m})^T

Gradient descent is not a catch-all, and has its drawbacks. In fact, it turns out to be necessary to compute the second partial derivatives of $C$ , which is in and of itself an expensive operationAs it scales on the order of a few orders of magnitude more than the number of input variables. Clearly, when you are operating a million or billion parameter model, this is extremely compute-intensive. But, it still is the building block of machine learning as we know it. Of course, human ingenuity has discovered ways around this computational burden. However, back to the context of neural nets, the gradient descent update rules look like this:

w_{k} \to w_{k}' = w_k - \eta \nabla \frac{\delta C}{\delta w_{k}}

b_{1} \to b_{1}' = b_1- \eta \nabla \frac{\delta C}{\delta b_{1}}

Now that we are in the context of neural nets again, we can discuss specific challenges with gradient descent in this context. If you recall, the cost function is the MSE, or average of the length of the distance between expected versus actual across all individual training examples. This does not scale well, and when you need to calculate $C_{x} = \frac{||y(x) - a||^2}{2}$ separately across each training input $x$ it becomes entirely cumbersome. This, however, can be reduced through the idea of stochastic gradient descent, which is essentially a sampling mechanism on random $\nabla C_x$ on randomly chosen training inputs, which surprisingly yields a good estimate of the true gradient vector. Under this framework, the gradient vector becomes:

\nabla C = \frac{1}{m}\sum_{j=1}^{m} \nabla C_{x_{j}}

And the gradient descent of our weights and biases is

w_{k} \to w_{k}' = w_{k} -\frac{\eta}{m} \sum_{j} \frac{\delta C_{x_j}}{\delta w_{k}}

b_{1} \to b_{1}' = b_1- \frac{\eta}{m} \sum_{j} \frac{\delta C_{x_{j}}}{\delta b_{1}}

The last gap is a big one: how do we actually compute the gradient of the cost function? The fastest way and perhaps the biggest breakthrough in machine learning is known as Backpropagation.