15.3 Scatterplots, Correlation Coefficient r, Linear Regression & Residuals

15.3 Scatterplots, Correlation Coefficient r, Linear Regression & Residuals

Bivariate Quantitative Data and Scatterplot Diagnostics

Bivariate data analysis investigates the relationship between two quantitative variables measured on the same individual entities. In most bivariate settings, researchers distinguish between:

• The explanatory variable ($x$, independent variable), which accounts for, explains, or predicts changes in the response.
• The response variable ($y$, dependent variable), which measures the primary outcome under investigation.

A scatterplot provides the primary graphical display for bivariate numerical data, plotting values of the explanatory variable on the horizontal axis and values of the response variable on the vertical axis. When analyzing a scatterplot on the FTCE Mathematics 6-12 examination, you must systematically evaluate four key attributes:

1. Direction: An association is positive if larger values of $x$ correspond to larger values of $y$ (the scatter drifts upward from left to right). An association is negative if larger values of $x$ correspond to smaller values of $y$ (the scatter drifts downward from left to right).
2. Form: Describes the geometric trajectory of the data points. Form is classified as linear if points cluster along a straight line, or non-linear (curvilinear, exponential, power, or periodic) if the trajectory bends.
3. Strength: Measures how tightly clustered the observations are around the underlying form. Strong relationships exhibit narrow dispersion around a central curve, whereas weak associations display broad, diffuse scatter.
4. Unusual Features: Includes isolated outliers (individual points that deviate substantially from the overall bivariate pattern) and clusters/gaps in the data space.

---

Pearson Correlation Coefficient $r$ and Coefficient of Determination $r^2$

The Pearson product-moment correlation coefficient $r$ provides a standardized numerical index measuring the direction and strength of the linear relationship between two quantitative variables. Mathematically, $r$ is the average product of the standardized scores ($z$-scores) of $x$ and $y$:

$$r = \frac{1}{n - 1} \sum_{i=1}^n \left(\frac{x_i - \bar{x}}{s_x}\right) \left(\frac{y_i - \bar{y}}{s_y}\right)$$

Essential Mathematical Properties of $r$

• Bounded Range: The correlation coefficient is strictly bounded: $-1 \le r \le 1$.
  • $r = 1$ indicates a perfect positive linear relationship.
  • $r = -1$ indicates a perfect negative linear relationship.
  • $r = 0$ indicates the absence of any linear association.
• Dimensionless (Unitless): Because $r$ is calculated using standardized $z$-scores, it has no units of measurement. Converting measurements (e.g., changing height from inches to centimeters) leaves $r$ completely unchanged.
• Scale Invariance: Adding a constant or multiplying by a positive scalar ($x' = c_1 x + d_1, y' = c_2 y + d_2$ with $c_1, c_2 > 0$) leaves $r$ invariant. Multiplying by a negative scalar reverses the sign of $r$.
• Symmetry: Variable roles are mathematically symmetric: $r(x, y) = r(y, x)$. Swapping which variable is assigned to the $x$-axis and $y$-axis has zero effect on $r$.
• Linearity Restriction: The coefficient $r$ measures linear strength only. A bivariate relationship can exhibit a perfect deterministic curved relationship (such as $y = x^2$ symmetric over $[-3, 3]$) while yielding $r = 0$.
• Non-Resistant Statistic: Like the mean and standard deviation, $r$ is strongly influenced by extreme outliers.

The Coefficient of Determination $r^2$

The coefficient of determination $r^2$ is the square of Pearson's correlation coefficient. It quantifies the proportion of total variation in the response variable $y$ that is accounted for by the linear relationship with explanatory variable $x$:

$$r^2 = \frac{\text{SS}_{\text{regression}}}{\text{SS}_{\text{total}}} = 1 - \frac{\text{SS}_{\text{residuals}}}{\text{SS}_{\text{total}}}$$

• Contextual Interpretation: 'Approximately $r^2 \times 100\%$ of the variation in the response variable ($y$) is explained by the linear relationship with the explanatory variable ($x$).'
• The remaining $(1 - r^2) \times 100\%$ represents residual variation attributable to unmeasured lurking variables or random noise.

---

The Least-Squares Regression Line (LSRL)

When a scatterplot displays a reasonably linear form, we model the relationship using a straight line. The mathematical model is expressed as:

$$\hat{y} = a + bx$$

where $\hat{y}$ (y-hat) represents the predicted value of the response variable for a given value of $x$, $b$ is the slope, and $a$ is the y-intercept.

The least-squares regression line (LSRL) is the unique straight line that minimizes the sum of the squared vertical deviations (residuals) between the observed data points and the line:

$$\text{Minimize } \sum_{i=1}^n (y_i - \hat{y}_i)^2$$

Formulating the Regression Equation from Summary Statistics

When given summary statistics (means $\bar{x}, \bar{y}$, standard deviations $s_x, s_y$, and correlation $r$), the parameters of the LSRL are calculated directly:

1. Slope ($b$): $$b = r \left(\frac{s_y}{s_x}\right)$$ Because standard deviations are strictly positive, the sign of the slope $b$ is identical to the sign of the correlation coefficient $r$.
2. Y-Intercept ($a$): $$a = \bar{y} - b\bar{x}$$ This algebraic formula guarantees that the regression line always passes through the centroid (point of averages), $(\bar{x}, \bar{y})$.

Contextual Interpretation of Regression Parameters

• Slope ($b$): 'For each 1-unit increase in the explanatory variable $x$, the model predicts that the response variable $y$ will increase (or decrease) by approximately $|b|$ units.'
• Y-Intercept ($a$): 'The predicted value of the response variable $y$ when the explanatory variable $x$ equals 0.' The intercept is physically meaningful only if $x = 0$ is plausible within the context and lies within or near the domain of observed data.

---

Residual Analysis and Linear Appropriateness

A residual $e_i$ is the directed vertical distance between an observed data point and the regression line:

$$\text{Residual } = \text{Observed } y - \text{Predicted } y = y_i - \hat{y}_i$$

• A positive residual ($e_i > 0$) means the observed point lies above the line; the model underestimated the actual value.
• A negative residual ($e_i < 0$) means the observed point lies below the line; the model overestimated the actual value.
• A fundamental algebraic property of the LSRL is that the sum and mean of the residuals are always zero: $\sum e_i = 0$ and $\bar{e} = 0$.

A residual plot is a scatterplot displaying explanatory values $x_i$ (or predicted values $\hat{y}_i$) on the horizontal axis and residuals $e_i$ on the vertical axis.

• Linear Model is Appropriate: The residual plot displays a random, uniform, horizontal band of scatter around the reference line $e = 0$, with no discernible pattern and approximately constant vertical spread (homoscedasticity).
• Linear Model is Inappropriate: The residual plot exhibits a clear curved or U-shaped pattern, indicating that the true relationship is non-linear and requiring a polynomial or non-linear transformation.
• Heteroscedasticity: A fan-shaped or megaphone pattern (where vertical spread expands as $x$ increases) indicates non-constant variance.

---

Outliers, High Leverage, Influential Points & Extrapolation

• Regression Outlier: A data point with an unusually large residual that deviates far vertically from the fitted line.
• High-Leverage Point: A data point whose explanatory coordinate $x$ is substantially smaller or larger than the mean $\bar{x}$. High-leverage points possess the physical potential to act as a fulcrum, heavily tilting the regression line.
• Influential Point: An observation that substantially alters the slope, y-intercept, or correlation coefficient when removed from the dataset. High-leverage points that deviate from the linear trend of the remaining data are strongly influential.
• The Danger of Extrapolation: Extrapolation is the practice of using a regression model to predict values of $y$ for explanatory values far outside the domain of observed $x$-data. Extrapolation is highly unreliable because researchers have no empirical evidence that the linear pattern persists beyond the observed range.

---

Worked Exemplar: Deriving the LSRL from Summary Statistics

Problem: An educational psychologist investigates the relationship between weekly time spent reading for pleasure in hours ($x$) and vocabulary assessment scores on a 100-point scale ($y$) among a random sample of 30 middle school students. The sample summary statistics are:

• Mean reading hours: $\bar{x} = 5.0\text{ hours}$, Standard deviation: $s_x = 2.0\text{ hours}$
• Mean vocabulary score: $\bar{y} = 70.0\text{ points}$, Standard deviation: $s_y = 8.0\text{ points}$
• Pearson correlation coefficient: $r = 0.75$

1. Compute the Regression Slope ($b$):

   $$b = r \left(\frac{s_y}{s_x}\right) = 0.75 \left(\frac{8.0}{2.0}\right) = 0.75 \times 4.0 = 3.0\text{ points per hour}$$

2. Compute the Y-Intercept ($a$):

   $$a = \bar{y} - b\bar{x} = 70.0 - (3.0)(5.0) = 70.0 - 15.0 = 55.0\text{ points}$$

3. State the Regression Equation:

   $$\hat{y} = 55.0 + 3.0x$$

4. Evaluate the Coefficient of Determination ($r^2$):

   $$r^2 = (0.75)^2 = 0.5625 = 56.25\%$$

   Interpretation: Exactly $56.25\%$ of the variation in vocabulary scores is explained by the linear relationship with weekly reading hours.

5. Predict Score and Calculate Residual: A student in the sample reads $7.0$ hours per week and achieves an observed vocabulary score of $79.0$ points.

   • Predicted score: $\hat{y} = 55.0 + 3.0(7.0) = 55.0 + 21.0 = 76.0\text{ points}$.
   • Residual: $e = y - \hat{y} = 79.0 - 76.0 = +3.0\text{ points}$.
   • Interpretation: Because the residual is positive ($+3.0$), the observed score lies above the regression line; the model underestimated the student's actual performance by $3.0$ points.

---

Regression Parameters and Bivariate Diagnostics Guide