Inference for regression Knowledge-Based Study

Study notes

1. When analyzing bivariate quantitative data, simple linear regression models the relationship between an explanatory variable $x$ and a response variable $y$. In a population, this relationship is expressed as:

   $$y = \alpha + \beta x + \epsilon$$

   where $\alpha$ is the population intercept, $\beta$ is the population slope, and $\epsilon$ represents random error. Sample data provide estimates $a$ (sample intercept) and $b$ (sample slope), yielding the estimated regression line $\hat{y} = a + bx$.

   Valid statistical inference about $\beta$ requires five key conditions (often remembered by the acronym LINER):

   • Linearity: The actual relationship between $x$ and the mean of $y$ is linear. Verified using a scatterplot or a residual plot showing no distinct curved pattern.
   • Independence: Individual observations or errors are independent of one another.
   • Normality: For any given value of $x$, the response values $y$ are normally distributed around the true regression line. Verified using a histogram or normal probability plot of the residuals.
   • Equal Variance: The standard deviation of $y$ (denoted $\sigma$) is constant across all values of $x$. Verified by checking that the vertical spread of points in the residual plot remains roughly uniform across all fitted values.
   • Random: Data originate from a well-designed random sample or randomized experiment.

   Because sample slope $b$ varies from sample to sample, it possesses a sampling distribution. When the regression conditions are satisfied:

   • The mean of the sampling distribution of $b$ is equal to the true population slope: $\mu_b = \beta$.
   • The true standard deviation of $b$ is given by $\sigma_b = \frac{\sigma}{\sigma_x \sqrt{n - 1}}$, where $\sigma$ is the standard deviation of the residuals and $\sigma_x$ is the standard deviation of the explanatory variable.

   In practice, the population standard deviation of the residuals $\sigma$ is unknown and is estimated by the standard error of the residuals, $s$:

   $$s = \sqrt{\frac{\sum (y - \hat{y})^2}{n - 2}}$$

   Replacing $\sigma$ with $s$ yields the standard error of the sample slope, $SE_b$:

   $$SE_b = \frac{s}{\sqrt{\sum (x - \bar{x})^2}}$$

   Because two parameters ($\alpha$ and $\beta$) are estimated to determine the regression line, inference procedures rely on the $t$-distribution with $df = n - 2$ degrees of freedom.

   A confidence interval estimates the plausible values for the true rate of change in $y$ per unit increase in $x$. A $C\%$ confidence interval for the population slope $\beta$ is calculated as:

   $$b \pm t^* \cdot SE_b$$

   where:

   • $b$ is the point estimate of the slope from the sample.
   • $t^*$ is the critical value from the $t$-distribution with $df = n - 2$ corresponding to the desired confidence level.
   • $SE_b$ is the standard error of the slope.

   Interpretation: We are $C\%$ confident that for each unit increase in the explanatory variable $x$, the mean of the response variable $y$ changes by an amount within the calculated interval.

   To assess whether a statistically significant linear relationship exists between $x$ and $y$, we perform a significance test for the population slope $\beta$.

   The standard hypotheses are:

   • $H_0: \beta = 0$ (There is no linear relationship between $x$ and $y$.)
   • $H_a: \beta \neq 0$, $H_a: \beta > 0$, or $H_a: \beta < 0$

   The test statistic follows a $t$-distribution with $df = n - 2$:

   $$t = \frac{b - \beta_0}{SE_b} = \frac{b - 0}{SE_b}$$

   The $P$-value is obtained by comparing the calculated $t$-statistic to the $t$-distribution with $n - 2$ degrees of freedom. If the $P$-value is less than the significance level $\alpha$, we reject $H_0$ and conclude that there is convincing evidence of a linear association between the two variables.

   Statistical software typically displays regression results in a standard table. Consider the following output format:

   • Predictor / Term: Lists "Constant" (or Intercept) and the explanatory variable name.
   • Coef: The estimated coefficients. The value beside the variable name is the sample slope $b$; the value beside "Constant" is $a$.
   • SE Coef: The standard error of each estimate. The entry in the variable row represents $SE_b$.
   • t-ratio (T): The test statistic computed as $t = \frac{\text{Coef}}{\text{SE Coef}}$.
   • P-value (P): The two-sided $P$-value for testing whether that specific parameter equals zero.
   • $s$: The standard error of the residuals (spread around the line).
   • $R^2$: The coefficient of determination, representing the proportion of variation in $y$ accounted for by the linear model with $x$.