46  Linear Regression and Linear Classifiers

Supervised learning explores problems where input observations are paired with target response labels. Given a training dataset, the objective is learning a mapping function capable of predicting responses for unseen future observations. Formally, training data are structured as:

\[ \mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^n, \]

where \(\mathbf{x}_i\in\mathbb{R}^p\) represents a feature vector and \(y_i\) represents the target response. When \(y_i\) is a continuous scalar magnitude, the problem is formulated as regression; when \(y_i\) represents a discrete categorical class, it is formulated as classification (Hastie et al. 2009; Bishop 2006).

This chapter explores two central families of deterministic models in supervised learning: linear regression (used for numerical target forecasting) and linear classifiers (used for separating classes using hyperplanes). We first develop simple and multiple linear regression, review evaluation metrics for continuous response models, and present Fisher’s Linear Discriminant and the Perceptron as foundational linear classification algorithms (Goodfellow et al. 2016; Vapnik 1998).

1 Regression: Continuous Variable Prediction

In regression problems, the target response variable takes real-valued continuous numbers. Typical examples include house prices, ambient temperatures, monthly income, or patient length of stay. The task consists of learning a mapping function \(f\) such that:

\[ \widehat{y}=f(\mathbf{x}) \]

serves as an accurate prediction of observed target \(y\).

Definition. Supervised regression estimates a mapping function \(f:\mathcal{X}\to\mathbb{R}\) from paired training instances \(\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^n\) such that predicted responses \(\widehat{y}_i=f(\mathbf{x}_i)\) approximate target values \(y_i\): \[ \widehat{y}_i=f(\mathbf{x}_i),\qquad y_i\in\mathbb{R}. \tag{1}\]

Regression can be tackled using models of varying complexity, including linear regression, polynomial regression, decision trees, random forests, and gradient boosted trees (Breiman 2001; Chen and Guestrin 2016). However, linear regression occupies a foundational position due to its analytical solvability, interpretability, and clear coefficient semantics.

1.1 Simple and Multiple Linear Regression

Simple linear regression uses a single predictor feature \(x\):

\[ \widehat{y}=\beta_0+\beta_1 x, \]

where \(\beta_0\) represents the intercept and \(\beta_1\) represents the slope (expected change in \(y\) per unit change in \(x\)).

Multiple linear regression generalizes this formulation to \(p\) predictor variables:

\[ y=\beta_0+\beta_1x_1+\beta_2x_2+\cdots+\beta_px_p+\varepsilon, \]

where \(\varepsilon\) denotes the unobserved zero-mean error term.

Definition. The multiple linear regression model expresses target response \(y_i\) as a linear combination of \(p\) predictor variables plus an error term: \[ y_i=\beta_0+\sum_{j=1}^{p}\beta_jx_{ij}+\varepsilon_i, \qquad i=1,\ldots,n. \tag{2}\]

For example, predicting a student’s final grade using study hours, attendance rate, and submitted assignments:

\[ \widehat{\text{grade}}=20+5(\text{hours})+0.3(\text{attendance})+2(\text{assignments}). \]

Here, \(\beta_1 = 5\) suggests that each additional study hour is associated with an expected 5-point increase in final grade, holding attendance and assignment counts constant.

Simple Linear Regression — Interactive Explorer
Adjust β₀ and β₁ to observe how residual errors and metrics update in real time

β₀ (intercept) = 5.00

β₁ (slope) = 0.50

2 Matrix Representation of Linear Regression

Matrix notation simplifies multiple linear regression mechanics:

\[ \mathbf{y}=\begin{bmatrix}y_1\\y_2\\\vdots\\y_n\end{bmatrix}, \qquad \mathbf{X}=\begin{bmatrix} 1 & x_{11} & x_{12} & \cdots & x_{1p}\\ 1 & x_{21} & x_{22} & \cdots & x_{2p}\\ \vdots & \vdots & \vdots & \ddots & \vdots\\ 1 & x_{n1} & x_{n2} & \cdots & x_{np} \end{bmatrix}, \qquad \boldsymbol{\beta}=\begin{bmatrix}\beta_0\\\beta_1\\\vdots\\\beta_p\end{bmatrix}. \]

Definition. Design matrix \(\mathbf{X}\in\mathbb{R}^{n\times(p+1)}\) contains one row per sample and one column per feature, plus a leading bias column of ones: \[ \mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon}. \tag{3}\]

Ordinary Least Squares (OLS) minimizes Sum of Squared Errors (SSE) (Legendre 1805; Gauss 1809):

\[ \min_{\boldsymbol{\beta}}\;\|\mathbf{y}-\mathbf{X}\boldsymbol{\beta}\|_2^2. \]

If \(\mathbf{X}^\top\mathbf{X}\) is non-singular (full column rank), the closed-form analytical OLS solution is:

\[ \widehat{\boldsymbol{\beta}}=(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}. \tag{4}\]

Predicted target responses and residual vectors are computed as:

\[ \widehat{\mathbf{y}}=\mathbf{X}\widehat{\boldsymbol{\beta}}, \qquad \widehat{\boldsymbol{\varepsilon}}=\mathbf{y}-\widehat{\mathbf{y}}. \]

import numpy as np
import plotly.graph_objects as go

XD = np.array([1,2,3,4,5,6,7,8,9,10], dtype=float)
YD = np.array([2.1,4.3,5.8,7.9,10.2,11.8,14.0,16.1,17.9,20.2])
N = len(XD)

xbar, ybar = XD.mean(), YD.mean()
b1_opt = np.dot(XD - xbar, YD - ybar) / np.dot(XD - xbar, XD - xbar)
b0_opt = ybar - b1_opt * xbar

b0_vals = np.linspace(b0_opt - 4, b0_opt + 4, 60)
b1_vals = np.linspace(b1_opt - 0.8, b1_opt + 0.8, 60)
B0, B1 = np.meshgrid(b0_vals, b1_vals)

Yhat = B0[:, :, None] + B1[:, :, None] * XD[None, None, :]
SSE = np.sum((YD[None, None, :] - Yhat) ** 2, axis=2)

fig = go.Figure()
fig.add_trace(go.Surface(
    x=B0, y=B1, z=SSE,
    colorscale="Blues", opacity=0.82,
    colorbar=dict(title="SSE"), showscale=True
))

sse_opt = float(np.sum((YD - (b0_opt + b1_opt * XD))**2))
fig.add_trace(go.Scatter3d(
    x=[b0_opt], y=[b1_opt], z=[sse_opt],
    mode="markers+text",
    marker=dict(size=8, color="#dc2626", symbol="diamond"),
    text=["OLS Minimum"], textposition="top center",
    name="OLS Minimum"
))

fig.update_layout(
    title="Loss Surface SSE(β₀, β₁)",
    scene=dict(xaxis_title="β₀", yaxis_title="β₁", zaxis_title="SSE"),
    height=500, template="plotly_white"
)
fig.show()

Figure 1: Loss surface SSE(β₀, β₁) for simple linear regression. Global minimum (red diamond) marks OLS parameter solution.

2.1 Invertibility and Multicollinearity

Analytical OLS requires \(\mathbf{X}^\top\mathbf{X}\) to be non-singular. If features are collinear, column vectors become linearly dependent, leading to rank deficiency. High multicollinearity produces large coefficient variance (Hoerl and Kennard 1970; Golub and Van Loan 2013).

3 Multiple Linear Regression in Python

import numpy as np
import pandas as pd

datos = pd.DataFrame({
    "hours": [2, 4, 6, 8],
    "attendance": [60, 70, 80, 90],
    "assignments": [4, 5, 6, 7],
    "grade": [56, 71, 86, 101]
})

X = datos[["hours", "attendance", "assignments"]].to_numpy(dtype=float)
y = datos["grade"].to_numpy(dtype=float)

X_design = np.column_stack([np.ones(len(X)), X])
beta_hat = np.linalg.pinv(X_design) @ y
print("Estimated Coefficients (β0, β1, β2, β3):", beta_hat)

4 Regression Evaluation Metrics

Definition. Sum of Squared Errors (SSE) measures cumulative squared residual discrepancies: \[ \mathrm{SSE}=\sum_{i=1}^{n}(y_i-\widehat{y}_i)^2. \tag{5}\]

• Mean Squared Error (MSE): \(\mathrm{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_i-\widehat{y}_i)^2\)
• Root Mean Squared Error (RMSE): \(\mathrm{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\widehat{y}_i)^2}\)
• Mean Absolute Error (MAE): \(\mathrm{MAE}=\frac{1}{n}\sum_{i=1}^{n}|y_i-\widehat{y}_i|\) {#eq-mae}
• Coefficient of Determination (\(R^2\)): \(R^2=1-\frac{\mathrm{SSE}}{\mathrm{SST}}\) {#eq-r2}

Regression Metrics — Outlier Sensitivity
Drag the outlier slider to compare MAE vs. RMSE sensitivity to extreme errors

Outlier error magnitude: 0.0 units

5 Linear Classifiers

Binary linear classifiers assign labels based on the sign of an affine discriminant function:

\[ g(\mathbf{x})=\mathbf{w}^\top\mathbf{x}+w_0. \]

Definition. A binary linear classifier assigns class labels via: \[ \widehat{y}=\operatorname{sign}(\mathbf{w}^\top\mathbf{x}+w_0), \qquad \widehat{y}\in\{-1,+1\}. \tag{6}\]

Linear Decision Boundary — Interactive Explorer
Adjust w₁, w₂, and w₀ to rotate and shift decision boundary g(x)=w₁x₁+w₂x₂+w₀

w₁ = 1.0

w₂ = 1.0

w₀ = -6.0

Class +1 Class −1 Misclassified

6 Fisher’s Linear Discriminant

Fisher’s Linear Discriminant finds a projection vector \(\mathbf{w}\) maximizing between-class variance relative to within-class variance (Fisher 1936).

Definition. Fisher’s criterion maximizes class mean separation relative to within-class scatter: \[ J_F(\mathbf{w})=\frac{\mathbf{w}^\top\mathbf{S}_B\mathbf{w}}{\mathbf{w}^\top\mathbf{S}_W\mathbf{w}}. \tag{7}\]

The optimal projection vector is:

\[ \mathbf{w}^*\propto \mathbf{S}_W^{-1}(\mathbf{m}_1-\mathbf{m}_2). \]

Fisher's Linear Discriminant — Interactive Explorer
Rotate projection direction vector to observe class separation changes

Projection angle θ = 0°

Class 1 Class 2 Direction w Optimal Direction

7 The Perceptron

Rosenblatt’s Perceptron (Rosenblatt 1958) learns decision weights iteratively from classification errors.

Definition. The Perceptron error-driven update rule adjusts parameter weights upon misclassifying instance \(i\): \[ \mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}+\eta y_i\mathbf{x}_i, \qquad b^{(t+1)}=b^{(t)}+\eta y_i. \tag{8}\]

Novikoff’s Perceptron Convergence Theorem guarantees convergence in finite update steps if data are linearly separable (Novikoff 1962; Minsky and Papert 1969).

Definition. Dataset \(\mathcal{D}\) is linearly separable if there exists weight vector \(\mathbf{w}\) and bias \(b\) satisfying: \[ y_i(\mathbf{w}^\top\mathbf{x}_i+b)>0, \qquad i=1,\ldots,n. \tag{9}\]

Perceptron Algorithm — Step-by-Step Animation
Observe how the perceptron updates its decision boundary whenever it encounters a misclassified point

Class +1 Class −1 Active Point

import numpy as np

class PerceptronBinario:
    def __init__(self, lr=1.0, max_epochs=100):
        self.lr = lr
        self.max_epochs = max_epochs
        self.w = None
        self.b = 0.0
        self.errors_ = []

    def fit(self, X, y):
        X = np.asarray(X, dtype=float)
        y = np.asarray(y, dtype=float)
        self.w = np.zeros(X.shape[1])
        self.b = 0.0

        for _ in range(self.max_epochs):
            errores = 0
            for xi, yi in zip(X, y):
                if yi * (np.dot(self.w, xi) + self.b) <= 0:
                    self.w += self.lr * yi * xi
                    self.b += self.lr * yi
                    errores += 1
            self.errors_.append(errores)
            if errores == 0:
                break
        return self

    def decision_function(self, X):
        return np.asarray(X) @ self.w + self.b

    def predict(self, X):
        return np.where(self.decision_function(X) >= 0, 1, -1)

8 Conceptual Comparison: Linear Regression, Fisher’s Discriminant, and Perceptron

Model	Task	Objective Function	Output
Linear Regression	Regression	Minimize SSE loss	Continuous real scalar
Fisher’s LDA	Classification	Maximize projected class variance ratio	Class assignment via threshold
Perceptron	Classification	Correct misclassified errors	Discrete binary label (\(\{-1, +1\}\))

9 Conclusions

Linear regression and linear classifiers serve as fundamental building blocks of supervised machine learning. Linear regression models continuous responses via analytical OLS estimation. Fisher’s Discriminant demonstrates optimal linear subspace projections. The Perceptron illustrates error-driven online weight updates. Understanding these foundational models prepares practitioners for advanced non-linear techniques, regularization frameworks, and deep learning architectures.

References

Bishop, Christopher M. 2006. Pattern Recognition and Machine Learning. Information Science and Statistics. Springer.

Breiman, Leo. 2001. “Random Forests.” Machine Learning 45 (1): 5–32. https://doi.org/10.1023/A:1010933404324.

Chen, Tianqi, and Carlos Guestrin. 2016. “XGBoost: A Scalable Tree Boosting System.” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–94. https://doi.org/10.1145/2939672.2939785.

Fisher, Ronald A. 1936. “The Use of Multiple Measurements in Taxonomic Problems.” Annals of Eugenics 7 (2): 179–88. https://doi.org/10.1111/j.1469-1809.1936.tb02137.x.

Gauss, Carl Friedrich. 1809. Theoria Motus Corporum Coelestium in Sectionibus Conicis Solem Ambientium. Friedrich Perthes und I.H. Besser.

Golub, Gene H., and Charles F. Van Loan. 2013. Matrix Computations. 4th ed. Johns Hopkins University Press.

Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org.

Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. Springer Series in Statistics. Springer. https://doi.org/10.1007/978-0-387-84858-7.

Hoerl, Arthur E., and Robert W. Kennard. 1970. “Ridge Regression: Biased Estimation for Nonorthogonal Problems.” Technometrics 12 (1): 55–67. https://doi.org/10.1080/00401706.1970.10488634.

Legendre, Adrien-Marie. 1805. Nouvelles méthodes Pour La détermination Des Orbites Des Comètes. Firmin Didot.

Minsky, Marvin, and Seymour Papert. 1969. Perceptrons: An Introduction to Computational Geometry. MIT Press.

Novikoff, Albert B. J. 1962. “On Convergence Proofs for Perceptrons.” Proceedings of the Symposium on the Mathematical Theory of Automata (Brooklyn, NY) 12: 615–22.

Rosenblatt, Frank. 1958. “The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain.” Psychological Review 65 (6): 386–408. https://doi.org/10.1037/h0042519.

Vapnik, Vladimir N. 1998. Statistical Learning Theory. Wiley.