STAT 440 - Forecasting
Lecture Review #3 - Least Squares

In class we introduced the "least squares" principle in statistics, and showed how to apply that principle to estimating the parameters of a linear regression model. A copy of the handout from class is available here.

Review questions:

1) What is the least squares principle in statistics?

2) In estimating the parameters of a linear regression model, exactly what total distance are we minimizing: error in the "X" dimension? Or error in the "Y" dimension? Or error measured as closest distance between point and line? Why do we do it that way?

3) Least squares is one principle for selecting the "best" parameter estimate. But there are others. What are some of these other principles we might use? What are some advantages of least squares?

Computational exercise:

Find the correlation, slope, and intercept for the following data. (DON'T do this by hand.) Briefly describe the relationship between the "X" and "Y" variables.

Mathematical exercise:

The standard linear regression model uses both a slope and an intercept. However, occasionally it is useful to use an interceptless model — that is, simply to model the data as

Y = β₁ * X + ε,

where ε is an error term. This forces the line to pass through the origin — there is no intercept.

You are asked to derive the least squares estimate for the slope parameter β₁. DO THIS FROM FIRST PRINCIPLES. That is, actually find the derivative, etc. — don’t just make a mechanical substitution into existing formulas.

SOLUTIONS:

In case you are having special difficulty with this problem, I have provided detailed solutions here. PLEASE do not consult these until you have made a diligent effort to tackle the problem on your own. You learn by struggling with difficult material a bit before mastering it — you don't learn simply by reading someone else's solution to a problem.