Regression Knowledge-Based Study

Study notes

1. Understanding Bivariate Data

   When collecting data, we often record two quantitative characteristics for each individual or item. This type of information is called bivariate numerical data.

   To examine the relationship between two variables, we assign them specific roles:

   • Explanatory Variable (Independent Variable, $x$): The variable that is suspected of causing, predicting, or explaining changes in the other. It is conventionally plotted on the horizontal axis.
   • Response Variable (Dependent Variable, $y$): The variable whose outcome or variation is measured or predicted. It is conventionally plotted on the vertical axis.

   A scatter plot displays each observation as a single point $(x, y)$ on the Cartesian coordinate plane. Examining a scatter plot reveals three fundamental features of the relationship:

   1. Direction: A relationship is positive if larger values of $x$ tend to accompany larger values of $y$. It is negative if larger values of $x$ correspond to smaller values of $y$.
   2. Form: The pattern can be linear (following roughly a straight path) or non-linear (displaying curvature or clusters).
   3. Strength: How closely the points gather around a recognizable path or line.

   Defining the Line of Best Fit

   When a scatter plot shows a linear trend, we can model the relationship using a straight line called a regression line or line of best fit. The standard equation for this line is:

   $$\hat{y} = b_1x + b_0$$

   where $\hat{y}$ denotes the predicted value of the response variable for a given value of $x$, $b_1$ represents the slope, and $b_0$ represents the $y$-intercept.

   What is a Residual?

   Real-world data points rarely lie perfectly on a single line. For any observed data point $(x_i, y_i)$, the vertical distance between the observed value and the line's predicted value is known as a residual (error):

   $$\text{Residual} = y_i - \hat{y}_i = \text{Observed } y - \text{Predicted } y$$

   • If a point lies above the regression line, $y_i > \hat{y}_i$, resulting in a positive residual.
   • If a point lies below the regression line, $y_i < \hat{y}_i$, resulting in a negative residual.
   • If an observed value matches the model prediction exactly, the residual is zero.

   The Least-Squares Criterion

   The unique line that minimizes the sum of all squared residuals, $\sum (y_i - \hat{y}_i)^2$, is called the ordinary least-squares (OLS) regression line. Squaring the residuals eliminates cancellation between positive and negative errors and penalizes larger discrepancies more heavily.

   Contextual Meaning of Model Parameters

   A regression equation is valuable only when its coefficients are interpreted accurately within the context of the underlying investigation.

   The Slope ($b_1$)

   The slope measures the rate of change of the predicted response per unit change in the explanatory variable:

   • For each increase of $1$ unit in $x$, the predicted response $\hat{y}$ changes by $b_1$ units.
   • If $b_1 > 0$, the relationship is positive; as $x$ increases, $\hat{y}$ increases.
   • If $b_1 < 0$, the relationship is negative; as $x$ increases, $\hat{y}$ decreases.

   The Intercept ($b_0$)

   The $y$-intercept is the predicted value of $y$ when $x = 0$:

   $$\hat{y} = b_1(0) + b_0 = b_0$$

   Caution: Interpreting the intercept is only physically meaningful if:

   1. The explanatory variable $x$ can plausibly equal zero in the given context.
   2. The collected data include values of $x$ near zero.

   If values near zero fall far outside the observed range of $x$, the intercept serves only as an anchoring constant for the linear geometry of the line and must not be given a literal physical interpretation.

   Using the Model to Make Predictions

   To predict the value of the response variable, substitute a specific value $x^*$ into the regression equation $\hat{y} = b_1x^* + b_0$.

   Interpolation

   Interpolation occurs when you make a prediction for an explanatory value that lies within the range of the observed values of $x$:

   $$\min(x) \le x^* \le \max(x)$$

   Interpolation is generally considered reliable as long as the underlying linear pattern is strong and well-supported by the sample points.

   Extrapolation

   Extrapolation occurs when you use the model to predict $y$ for an explanatory value outside the domain of observed data:

   $$x^* < \min(x) \quad \text{or} \quad x^* > \max(x)$$

   Extrapolation is hazardous and should be treated with extreme caution because we have no observational evidence that the linear relationship continues beyond the studied range. Patterns frequently level off, reverse direction, or change form entirely outside the measured interval.

   Comprehensive Worked Example

   A study recorded the study time in hours ($x$) and the subsequent exam score out of 100 ($y$) for a group of students whose study times ranged from $2$ to $10$ hours. The calculated least-squares regression line is:

   $$\hat{y} = 6.5x + 38.0$$

   Step-by-Step Analysis

   1. Interpreting the Slope: The slope is $6.5$. This indicates that for each additional hour spent studying, the predicted exam score increases by $6.5$ points.
   2. Interpreting the Intercept: The intercept is $38.0$. For a student who studies $0$ hours, the predicted exam score is $38.0$ points (provided studying zero hours is plausible).
   3. Predicting within the Domain: For a student studying $6$ hours:$$\hat{y} = 6.5(6) + 38.0 = 39.0 + 38.0 = 77.0 \text{ points}$$Because $6$ hours lies between $2$ and $10$ hours, this is an interpolation.
   4. Calculating a Residual: Suppose a student who studied $6$ hours actually achieved an exam score of $81$ points:$$\text{Residual} = y - \hat{y} = 81 - 77.0 = +4.0$$The positive residual indicates that the student scored $4.0$ points higher than predicted by the model (the model underpredicted the performance).
   5. Evaluating Extrapolation: Using the equation to predict the score of someone studying $20$ hours yields $\hat{y} = 6.5(20) + 38.0 = 168$, which is impossible on a 100-point exam. This illustrates why extrapolation beyond the collected domain ($2$ to $10$ hours) produces invalid conclusions.