Properties of least-squares estimators

Download: OLS-Properties_init.xlsx

Introduction

The most common method used to estimate regression parameters is the least-squares (LS) method. LS estimates are those that minimize the sum of squared residuals, $\text{SS}_\text{res} = \sum e_i^2$. In this screencast we will highlight these advantages and use LS estimates to construct parametric confidence intervals.

There are four main advantages for using least-squares regression, two of which apply only to linear regression models:

1. Least-squares estimates are known to be BLUE statistics: the Best Linear Unbiased Estimates of their corresponding population parameters. The attraction of unbiasedness is obvious, but “best” in what sense? They have the best efficiency (ie, lowest standard error for a given sample size) when the distribution is near-normal.
2. The Central Limit Theorem applies to LS estimates: as the sample size increases, they tend towards a normal distribution even if the measurement errors on the original data are non-normal. This provides some justification for the use of parametric confidence intervals for inferential statistics.
3. For linear regression models, the estimates can be calculated directly without using optimization or some other iterative process. This is a huge boon in terms of computational efficiency and also means that one need not provide initial estimates in order to calculate the statistics. (This latter requirement is more than just a nuisance since a poor initial estimate can lead one to a local minimum.)
4. The standard errors of the estimates can be estimated directly without the need for resampling methods. Just as the standard errors of the mean and trimmed means can be estimated from a single sample, the same is true for the standard error of LS regression estimates. For linear regression models, there are equations that can be used to obtain these estimates; these will be illustrated for the simple linear regression model in this screencast. For nonlinear models an iterative process is needed to obtain these estimates (but still: no resampling needed).

There are of course disadvantages to LS estimates. One of the biggest is that they are not robust to the presence of outliers.

Learning Goals

• Recognize the advantages of least-squares regression
• Learn to use least-squares regression estimates to create parametric confidence intervals for inferential purposes

Tasks

• Use OLS to fit a best-fit line (the SLR model) to the calibration data for the Nernstian electrode
• Manually calculate the standard errors of the LS parameter estimates from the equations
• Manually calculate the parametric 95% CIs of the population slope ($\beta_1$) and intercept ($\beta_0$) values. Is the LS slope estimate $b_1$ significantly different from the theoretical value of $-59.2$ mV/decade?
• Assume the electrode is Nernstian and calculate a parametric 95% CI for the temperature of the solution.

After the screencast your spreadsheet should look something like the image below.