Linear regression

Linear regression is the simplest Regression model: predict the output as a linear function of the inputs. For a single input feature $x$:

$f(x,\mathbf{w})=w_0+w_{1}x$

$w_0$ is the intercept — the value of $f$ when $x=0$. $w_1$ is the slope — how much $f$ changes per unit increase in $x$. The vector $\mathbf{w}=(w_0,w_1)$ contains everything the model has learned. For multiple input features $x_1,x_2,\dots ,x_m$, the model generalizes to a sum of weighted inputs:

$f(\mathbf{x},\mathbf{w})=w_0+w_1{x}_1+w_2{x}_2+\cdots +w_m{x}_m$

If the intercept is absorbed into $\mathbf{w}$ by prepending a constant 1 to $\mathbf{x}$, this is just $\hat{y}={\mathbf{w}}^T\mathbf{x}$ — the inner product of the weight vector and the input vector. The hat on $\hat{y}$ marks it as the model’s prediction, distinct from the true label $y$. Training the model means finding good values for the weights $\mathbf{w}$.

What good fit means

We pick a Loss function that measures how badly the model fits the data, and minimize it. The standard regression loss is Mean squared error:

$J(\mathbf{w})=\frac{1}{2N}{\sum }_{i=1}^N(f(x_i,\mathbf{w})-y_i{)}^2$

A small $J$ means a good fit. The training problem becomes: find $\mathbf{w}$ that minimizes $J$.

Training

For linear regression with MSE loss, there’s a closed-form solution — take partial derivatives of $J$ with respect to each $w_j$, set them to zero, solve the resulting system of equations. In matrix form, stacking the inputs (with a leading column of 1s for the intercept) into a design matrix $X$ and the targets into a vector $y$, this gives the normal equation:

${\mathbf{w}}^{\ast }=(X^{T}X{)}^{-1}{X}^{T}y$

This is the exact best $\mathbf{w}$ in one calculation — no iteration. In practice we don’t form $(X^{T}X{)}^{-1}$ explicitly; numerically stable implementations (including scikit-learn’s LinearRegression) solve the system via SVD or a least-squares routine.

For more complex models — neural networks with millions of parameters, or any model that isn’t linear in its parameters — the closed form breaks down. Solving the equations becomes intractable. The fallback is Gradient descent, which works on linear regression too and generalizes to everything else.

The gradient of $J$ with respect to $w_0$ and $w_1$:

$\frac{\partial J}{\partial w_0}=\frac{1}{N}{\sum }_{i=1}^N(f(x_i)-y_i)$

$\frac{\partial J}{\partial w_1}=\frac{1}{N}{\sum }_{i=1}^N(f(x_i)-y_i)\cdot x_i$

The factor of 2 from differentiating the square cancels the 1/2 in the loss formula — the reason that 1/2 is there in the first place.

The training loop:

1. Initialize $w_0,w_1$ — often to zero.
2. Compute predictions $f(x_i)=w_0+w_1{x}_i$ for every training example.
3. Compute the loss $J(\mathbf{w})$.
4. Compute the two gradients above.
5. Update: $w_0\leftarrow w_0-\eta \frac{\partial J}{\partial w_0}$, $w_1\leftarrow w_1-\eta \frac{\partial J}{\partial w_1}$, where $\eta$ is the Learning rate.
6. Repeat until the loss stops improving.

Limitations

Linear regression assumes the relationship between input and output is linear. Many real relationships aren’t. For curvature, use Polynomial regression. For classification, the linear function gets wrapped in a sigmoid to become Logistic regression. For non-linear decision boundaries that can’t be expressed as polynomials, use neural networks, decision trees, or kernel methods.

In scikit-learn:

from sklearn.linear_model import LinearRegression
reg = LinearRegression()
reg.fit(X_train, y_train)
y_pred = reg.predict(X_test)

This uses the closed-form solver. For very large datasets, SGDRegressor uses stochastic gradient descent.