The regression model and least squares

What regression tries to explain

Suppose six companies have very different sales growth. Regression asks: can another variable account for those differences? The thing we measure is the spread of Y around its mean, the sum of squares total (SST): $\sum (Y_i - \bar Y)^2$.

The variable whose variation we want to explain is the dependent variable (also called the explained variable), written Y. The variable we use to explain it is the independent variable (also called the explanatory variable), written X. On a scatter plot, Y goes on the vertical axis and X on the horizontal axis.

With one X the method is simple linear regression; with several X variables it is multiple regression (not in this module). The X variable must actually vary across observations: a constant cannot explain why Y differs, and neither can a variable that is pure noise.

Always ask “what is being explained by what?” The dependent variable is the thing in the question you want to predict or understand. If an analyst asks whether earnings revisions help explain share returns, returns are Y and revisions are X.

The model: intercept, slope and error

The population relationship is $Y_i = b_0 + b_1X_i + \varepsilon_i$. The intercept $b_0$ and slope $b_1$ are the regression coefficients. Not every point sits on the line; the error term $\varepsilon_i$ captures how far each true Y lies from the true population line.

We never see the population values $b_0$ and $b_1$. We estimate them from a sample, and the estimates wear hats: $\hat b_0$ and $\hat b_1$. The fitted value for observation i is $\hat Y_i = \hat b_0 + \hat b_1X_i$.

The residual is the sample counterpart of the error: $e_i = Y_i - \hat Y_i$, the vertical distance from a point to the fitted line. The error term belongs to the population relationship; the residual belongs to the line fitted on your sample. A residual is measured in the units of Y (euros, percent, days and so on).

The least squares criterion

Many lines could be drawn through a cloud of points. OLS chooses the one that minimises the sum of squares error: $SSE = \sum (Y_i - \hat Y_i)^2 = \sum e_i^2$. This is the least squares criterion, which is why the method is called ordinary least squares.

Why square the residuals? Positive and negative misses would otherwise cancel out, and squaring punishes large misses much more than small ones. In fact, for any OLS line with an intercept the plain residuals always add up to exactly zero, so their sum cannot measure fit.

Two useful by-products follow from the maths. The fitted line always passes through the point of means $(\bar X, \bar Y)$, and the average fitted value equals the average actual value $\bar Y$. The expected value of the residual is zero.

• OLS minimises squared vertical distances (in Y units), not horizontal or perpendicular distances.
• Regressing Y on X gives a different line from regressing X on Y, so the roles matter.

Computing the slope and the intercept

The slope is the covariance of X and Y divided by the variance of X. Because both have $n-1$ in the denominator, it simplifies to the sum of cross-products of deviations divided by the sum of squared X deviations.

Once you have the slope, the intercept comes from the means: $\hat b_0 = \bar Y - \hat b_1\bar X$. This is just the statement that the line goes through the point of means.

Compare the slope with the correlation, $r = \text{Cov}(X,Y)/(s_X s_Y)$. They share the numerator; only the denominator differs. Since standard deviations and variances are positive, a positive covariance gives a positive slope and a positive correlation, and a negative covariance makes both negative. Combining the two formulas gives a handy shortcut: $\hat b_1 = r \times s_Y / s_X$.