The Method of Least Squares | 最小二乘法

📚 The Method of Least Squares | 最小二乘法

The method of least squares is a fundamental statistical technique used to determine the line (or curve) that best fits a set of observed data points. It works by minimising the sum of the squares of the vertical distances, or residuals, between each data point and the fitted model. This approach is essential in regression analysis, enabling us to model relationships between variables, make predictions, and test hypotheses. The "least squares" criterion ensures that no single large deviation dominates the fit, and it yields a unique solution under standard conditions.

最小二乘法是一种基础统计技术，用于确定与一组观测数据点最匹配的直线（或曲线）。其原理是最小化各数据点到拟合模型的垂直距离（即残差）的平方和。该方法在回归分析中不可或缺，使我们能够对变量间的关系建模、进行预测并检验假设。“最小二乘”准则能够避免单个较大偏差主导整个拟合结果，并且在标准条件下可以给出唯一解。

1. What is the Method of Least Squares? | 什么是最小二乘法？

The method of least squares finds the "best" all as possible. If we have a set of n data points (xᵢ, yᵢ), and we propose a linear model ŷ = a + bx, the residual for the i-th point is eᵢ = yᵢ − ŷ imise SSE = Σ eᵢ². This idea can be extended to polynomial and multiple regression models.

最小二乘法通过使残差平方和尽可能小来寻找“最佳”直线。如果我们有一组 n 个数据点 (xᵢ, yᵢ)，并假设线性模型 ŷ = a + bx，则第 i 个点的残差为 eᵢ = yᵢ − ŷᵢ。a 和 b 的最小二乘估计量就是使得 SSE = Σ eᵢ² 最小化的数值。这一思想可以推广到多项式回归和多元回归模型中。

2. Linear Regression Model | 线性回归模型

where a is the intercept, b is the slope, and ε represents random error. The goal of least squares is to find the line ŷ linear trend, even though individual points will scatter above or below the line due to ε.

在最简单的情形下，我们假设存在直线关系：y = a + bx + ε，其中 a 是截距，b 是斜率，ε 代表随机误差。最小二乘法的目标是找出直线 ŷ = a + bx，使其尽可能好地反映潜在的线性趋势，即便个别点会因 ε 而散布在直线的上方或下方。

The predicted value ŷᵢ is the point on the line corresponding to xᵢ, and the difference yᵢ − ŷᵢ is the residual for that observation. The least squares line minimises the sum of squared residuals, not the sum of absolute residuals.

预测值 ŷᵢ 是直线上对应于 xᵢ 的点，而实际值 yᵢ 与 ŷᵢ 之差即为该观测值的残差。最小二乘直线最小化的是残差的平方和，而非残差绝对值的和。

3. Residuals and Sum of Squared Errors | 残差和平方和

each data point (xᵢ, yᵢ) is eᵢ = yᵢ − (a + bxᵢ). The sum of squared errors (SSE) is then:

SSE = Σ (yᵢ − a − bxᵢ)²

对于给定的直线 y = a + bx，每个数据点 (xᵢ, yᵢ) 的残差为 eᵢ = yᵢ − (a + bxᵢ)。误差平方和（SSE）即为：

SSE = Σ (yᵢ − a − bxᵢ)²

A smaller SSE indicates that the line is closer to the data points overall. Least squares selects a and b to make SSE as small as possible, which penalises large deviations more heavily because of the squaring.

SSE 越小，表明直线总体上越接近数据点。最小二乘法选择 a 和 b 以最小化 SSE，由于平方运算，该方法会对较大的偏差施加更重的惩罚。

4. Minimising the Sum of Squares | 最小化平方和

To minimise SSE with respect to a and b, we take partial derivatives and set them equal to zero:

∂SSE/∂a = −2 Σ (yᵢ − a − bxᵢ) = 0

∂SSE/∂b = −2 Σ xᵢ (yᵢ − a − bxᵢ) = 0

要针对 a 和 b 最小化 SSE，我们需要求偏导数并令其为零。

Simplifying these equations gives a system known as the normal equations. The process ensures that we have found a stationary point, and because SSE is a convex quadratic function, the solution corresponds to the global minimum.

化简这些方程会得到一个称作正规方程组的系统。该过程确保我们找到了一个驻点，并且由于 SSE 是一个凸二次函数，该解对应全局最小值。

5. Normal Equations | 正规方程

The normal equations are derived by rearranging the partial derivative equations:

Σ yᵢ = n a + b Σ xᵢ

Σ xᵢ yᵢ = a Σ xᵢ + b Σ xᵢ²

正规方程是通过整理偏导数方程得到的。

b, provided the sum of squares of x is not degenerate. All summations run from i = 1 to n.

只要 x 的平方和不是退化的，这两个联立方程就可以解出 a 和 b。所有求和均从 i = 1 到 n。

6. Solving for Slope and Intercept | 求解斜率和截距

Solving the normal equations for b yields the least squares slope:

b = (n Σ xᵢ yᵢ − Σ xᵢ Σ yᵢ) / (n Σ xᵢ² − (Σ xᵢ)²)

对正规方程求解 b 得到最小二乘斜率。

Once b is known, the intercept a is simply:

a = ȳ − b x̄

其中 x̄ = Σ xᵢ / n，ȳ = Σ yᵢ / n。

where x̄ = Σ xᵢ / n and ȳ yᵢ / n. This shows that the least squares line always passes through the point of means (x̄, ȳ).

这表示最小二乘直线必然经过均值点 (x̄, ȳ)。

广告 / Sponsored

7. Worked Example | 计算示例

Consider the following small dataset:

考虑以下小型数据集：

x	1	2	3	4
y	2	3	5	4

14. Then Σxy = 1·2 + 2·3 + 3·5 + 4·4 = 2+6+15+16 = 39. Σx² = 1²+2²+3²+4² = 1+4+9+16 = 30.

首先计算基本求和：n = 4, Σx = 1+2+3+4 = 10, Σy = 2+3+5+4 = 14。然后 Σxy = 1·2 + 2·3 + 3·5 + 4·4 = 2+6+15+16 = 39。Σx² = 1²+2²+3²+4² = 1+4+9+16 = 30。

Now apply the formula for b:

b = (4·39 − 10·14) / (4·30 − 10²) = (156 − 140) / (120 − 100) = 16 / 20 = 0.8

现在代入 b 的公式。

Then x̄ = 10/4 = 2.5, ȳ = 14/4 = 3.5, so a = 3.5 − 0.8·2.5 = 3.5 − 2.0 = 1.5.

然后 x̄ = 10/4 = 2.5, ȳ = 14/4 = 3.5，因此 a = 3.5 − 0.8·2.5 = 3.5 − 2.0 = 1.5。

The least squares regression line is therefore ŷ x. This is the line that minimises the sum of squared residuals for the given four points.

因此最小二乘回归直线为 ŷ = 1.5 + 0.8x。这就是使得给定四个点的残差平方和最小的直线。

8. Assumptions of Least Squares | 最小二乘法的假设

When used in linear regression, the ordinary least squares method assumes: (1) linearity – the relationship between x and y is linear; (2) independence – observations are independent; (3) homoscedasticity – the variance of the errors is constant; and (4) normality – the errors are normally distributed (important for inference, less for estimating coefficients). Violating these assumptions can lead to biased or inefficient estimates.

在线性回归中使用普通最小二乘法时，需要满足以下假设：(1) 线性关系 – x 与 y 之间是线性的；(2) 独立性 – 观测值相互独立；(3) 同方差性 – 误差的方差恒定；(4) 正态性 – 误差服从正态分布（对统计推断重要，对系数估计本身影响较小）。违反这些假设可能导致估计有偏或效率低下。

Transformations or alternative methods (e.g., weighted least squares) may be applied when assumptions are not met.

当假设不满足时，可以采用变量变换或其他方法（如加权最小二乘法）。

9. Goodness of Fit: R² | 拟合优度：R²

The coefficient of determination, R², measures how well the least squares line explains the variation in y. It is defined as:

R² = 1 − SSE / SST

where SST = Σ (yᵢ − ȳ ue close to 1 indicates that most of the variability is accounted for by the linear model.

决定系数 R² 衡量最小二乘直线对 y 变异的解释程度。其定义为 R² = 1 − SSE / SST，其中 SST = Σ (yᵢ − ȳ)² 是总平方和。R² 取值范围从 0 到 1；接近 1 的值表示线性模型解释了绝大部分变异。

.5)² + (5−3.5)² + (4−3.5)² = 2.25+0.25+2.25+0.25 = 5. So R² = 1 − 1.8/5 = 0.64, meaning 64% of the variability is explained.

在计算示例中，SSE = (2−2.3)² + (3−3.1)² + (5−3.9)² + (4−4.7)² = 0.09 + 0.01 + 1.21 + 0.49 = 1.8。SST = (2−3.5)² + (3−3.5)² + (5−3.5)² + (4−3.5)² = 2.25+0.25+2.25+0.25 = 5。所以 R² = 1 − 1.8/5 = 0.64，意味着 64% 的变异被模型解释。

10. Applications and Limitations | 应用与局限性

The method of least squares is widely used in economics, natural sciences, engineering, and machine learning. It underpins ANOVA, multiple regression, and polynomial fitting. Its elegant mathematical properties make coefficient estimates unbiased under classical assumptions.

最小二乘法广泛应用于经济学、自然科学、工程和机器学习中。它是方差分析、多元回归和多项式拟合的基础。其优美的数学性质使得在经典假设下系数估计是无偏的。

However, least squares is sensitive to outliers because squared residuals give exaggerated weight to extreme points. A single outlier can significantly shift the regression line. In such cases, robust regression methods (e.g., least absolute deviations) may be preferred. Additionally, the method does not handle multicollinearity well in multiple regression; variable selection or regularisation techniques are needed.

然而，最小二乘法对异常值很敏感，因为平方残差会给予极端点过大的权重。单个异常值就可能显著偏移回归线。此时，稳健回归方法（如最小绝对偏差法）可能更为可取。此外，在多元回归中该方法难以处理多重共线性问题，需要借助变量选择或正则化技术。

In practice, always inspect residual plots, check R², and verify assumptions before relying on a least squares model for inference or prediction.

在实践中，在依赖最小二乘模型进行推断或预测之前，一定要检查残差图、关注 R² 并验证假设是否成立。

Published by TutorHao | Mathematics Revision Series | aleveler.com

更多咨询请联系16621398022（同微信）