10.2 Simple Linear Regression & Coefficient of Determination (R^2)

10.2 Simple Linear Regression & Coefficient of Determination (R^2)

Quick Summary: While correlation analysis quantifies the strength of association between two continuous variables, Simple Linear Regression establishes an operational transfer function—a mathematical equation expressing $Y = f(X)$. Using the Ordinary Least Squares (OLS) method, regression derives a best-fit straight line that minimizes the sum of squared vertical deviations between actual observed data and model predictions. Six Sigma Green Belts must be proficient in reading software regression outputs, interpreting slope ($b_1$) and intercept ($b_0$) parameters, deconstructing the ANOVA table, and validating the four foundational assumptions—LINE (Linearity, Independence, Normality, Equal Variance)—via residual analysis before deploying any model for process control or optimization.

---

The Transition from Correlation to Regression

In DMAIC projects, correlation answers: "Are $X$ and $Y$ linearly related?" Regression takes the next operational step and answers:

• "By how many units does $Y$ change when we adjust $X$ by 1 unit?"
• "What is the expected value of $Y$ if we operate $X$ at a specific target setting?"
• "Can we mathematically control the process output by dialing in the input?"

                      Correlation vs. Regression

   CORRELATION (Bidirectional)                 REGRESSION (Directional Transfer Function)
   ┌─────────────────────────┐                 ┌─────────────────────────┐
   │ Variable X ◀──▶ Variable Y│                 │   Input Predictor (X)   │
   └─────────────────────────┘                 └────────────┬────────────┘
   • Non-directional (r_xy = r_yx)                          │ Transfer Function
   • Scaled from -1.0 to +1.0                               ▼ Y = b0 + b1(X)
   • Measures strength/association             ┌─────────────────────────┐
   • No cause-and-effect equation              │   Output Response (Y)   │
                                               └─────────────────────────┘
                                               • Directional (X predicts Y)
                                               • Retains physical measurement units
                                               • Quantifies magnitude of change (slope)

---

The Mathematical Model of Simple Linear Regression

The Theoretical Population Model

The theoretical population model assumes that for any specific value of the independent predictor variable $X$, the corresponding values of the dependent response variable $Y$ are normally distributed around a linear mean line:

$Y = \beta_0 + \beta_1 X + \epsilon$

Where:

• $Y$ is the dependent response variable (the process output / CTQ),
• $X$ is the independent predictor variable (the process input factor),
• $\beta_0$ (beta-zero) is the true population intercept (the value of $Y$ when $X = 0$),
• $\beta_1$ (beta-one) is the true population slope (the rate of change in $Y$ per unit change in $X$),
• $\epsilon$ (epsilon) is the random unobserved error term, assumed to be independent and identically distributed as $N(0, \sigma^2)$.

The Sample Estimated Regression Line

Because true population parameters $(\beta_0, \beta_1)$ are unobservable, Green Belts calculate sample estimates $(b_0, b_1)$ from experimental sample data, yielding the sample prediction equation:

$\hat{y} = b_0 + b_1 x$

Where:

• $\hat{y}$ (y-hat) is the fitted (predicted) value of $Y$ for a given value of $x$,
• $b_0$ is the sample estimate of the intercept,
• $b_1$ is the sample estimate of the slope.

---

The Ordinary Least Squares (OLS) Method

How do we determine which straight line among infinitely many possibilities represents the "best fit"? The standard methodology is Ordinary Least Squares (OLS).

Defining the Residual (Error)

For each observed data point $(x_i, y_i)$, the vertical distance between the actual observed value ($y_i$) and the predicted value on the regression line ($\hat{y}_i$) is called the residual ($e_i$):

$\text{Residual } e_i = y_i - \hat{y}_i = y_i - (b_0 + b_1 x_i)$

                     The Ordinary Least Squares Concept

   Y ▲
     │                                            (x_i, y_i) Actual Point
     │                                               █
     │                                               │ ▲
     │                                               │ │ Residual e_i = y_i - ŷ_i
     │                                               │ ▼ (Vertical Distance)
     │                             ▲                 ○ ŷ_i (Point on Line)
     │                           ╱ │ e_j
     │                         ╱   ▼ █
     │                       ╱
     │                     ╱   Regression Line: ŷ = b0 + b1(x)
     │       Intercept b0 ╱    (Minimizes Sum of Squared e_i²)
     │─────────────○─────
     │
     └─────────────────────────────────────────────────────────────► X
     0

The Minimization Criterion

If we simply summed the raw residuals ($\sum e_i$), positive deviations above the line would cancel out negative deviations below the line, resulting in zero for countless arbitrary lines. To eliminate signs and penalize larger errors, OLS squares every residual and minimizes the Sum of Squared Errors ($SSE$):

$\text{Minimize } SSE = \sum_{i=1}^n e_i^2 = \sum_{i=1}^n (y_i - \hat{y}_i)^2 = \sum_{i=1}^n [y_i - (b_0 + b_1 x_i)]^2$

The OLS Formulas for Slope and Intercept

Using calculus (taking partial derivatives with respect to $b_0$ and $b_1$ and setting them to zero), the unique least-squares estimators are derived:

$b_1 = \frac{SS_{xy}}{SS_{xx}} = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} = r \left(\frac{s_y}{s_x}\right)$

$b_0 = \bar{y} - b_1 \bar{x}$

Key Mathematical Properties of the OLS Line

1. The Centroid Property: The regression line always passes through the point of sample means $(\bar{x}, \bar{y})$.
2. Zero Sum of Residuals: The sum (and arithmetic mean) of the OLS residuals is exactly zero: $\sum e_i = 0$.
3. Relationship to Correlation: The slope $b_1$ and Pearson's $r$ always share the same algebraic sign ($b_1 > 0 \iff r > 0$). However, while $r$ is bounded by $\pm 1.0$, the slope $b_1$ has no bounds and reflects the actual units of $Y / X$.

---

Reading Software Regression Output & The ANOVA Table

In modern Six Sigma practice, regression is computed using statistical software (Minitab, JMP, or R). Green Belts must be fluent in deconstructing software output tables.

Sample Software Output: Curing Oven Analysis

Consider an industrial study evaluating heating time ($X$, minutes) versus resin hardness ($Y$, Shore D units) across $n = 20$ production batches:

Regression Analysis: Hardness (Y) versus Heating Time (X)

The regression equation is:
Hardness = 22.41 + 3.125 Heating Time

Predictor        Coef     SE Coef          T        P
Constant       22.410       2.150      10.42   <0.001
Time (X)        3.125       0.285      10.96   <0.001

S = 2.4512    R-Sq = 87.0%    R-Sq(adj) = 86.2%

Analysis of Variance (ANOVA)
Source           DF        SS        MS        F        P
Regression        1    721.25    721.25   120.12   <0.001
Error (Residual) 18    108.08      6.00
Total            19    829.33

Deconstructing the Parameter Estimates Table

1. Coefficients (Coef):
   • Constant ($b_0 = 22.410$): The estimated hardness when heating time is zero minutes. (Caution: Only physically meaningful if $X = 0$ is within the operational range).
   • Slope ($b_1 = 3.125$): For every additional 1.0 minute of heating time, resin hardness is predicted to increase by $3.125\text{ Shore D units}$.
2. Standard Error of Coefficient (SE Coef): Quantifies the sampling uncertainty of the parameter estimate. Smaller standard errors indicate greater precision.
3. $T$-Value and $P$-Value:
   • Tests the hypothesis: $H_0: \beta_1 = 0$ (no linear effect) vs $H_a: \beta_1 \ne 0$.
   • Test statistic: $t = \frac{b_1 - 0}{SE(b_1)} = \frac{3.125}{0.285} = 10.96$.
   • Since $p < 0.001 \le 0.05$, Reject $H_0$. Heating time has a statistically significant linear effect on hardness.

Deconstructing the Model Summary Statistics

• $S = 2.4512$ (Standard Error of the Regression): The estimated standard deviation of the residual errors ($\sqrt{MS_{error}}$). Represents the typical distance that observed data points fall from the regression line, measured in the units of $Y$.
• $R\text{-Sq} = 87.0%$ ($R^2$): $87.0%$ of the total variance in resin hardness is explained by heating time.
• $R\text{-Sq(adj)} = 86.2%$ (Adjusted $R^2$): Modified version of $R^2$ that accounts for sample size and the number of predictors. In multiple regression, adding useless predictors always inflates raw $R^2$, but causes $R^2_{adj}$ to decrease.

Deconstructing the ANOVA Table

The Analysis of Variance (ANOVA) partitions the total sum of squares into two distinct components:

$SS_{total} = SS_{regression} + SS_{error}$

$\sum (y_i - \bar{y})^2 = \sum (\hat{y}_i - \bar{y})^2 + \sum (y_i - \hat{y}_i)^2$

Exam Relationship: In simple linear regression (one predictor), the ANOVA $F$-statistic is exactly equal to the square of the slope's $t$-statistic: $F = t^2 \implies 120.12 \approx (10.96)^2$

---

Residual Analysis & The LINE Assumptions

A high $R^2$ and a significant $p$-value do not prove that a regression model is valid. The mathematical validity of OLS hypothesis tests, confidence intervals, and prediction bands hinges strictly upon satisfying four core assumptions, known by the acronym LINE:

                      The Four LINE Assumptions

   L — LINEARITY           Relationship between X and Y is strictly linear.
   I — INDEPENDENCE        Residual errors are independent (no autocorrelation).
   N — NORMALITY           Residual errors follow a Gaussian normal distribution.
   E — EQUAL VARIANCE      Residual variance is constant across all values of X.
                           (Homoscedasticity)

To audit these assumptions, practitioners generate a diagnostic Four-in-One Residual Plot:

                  The Four-in-One Residual Diagnostic Panel

    1. NORMAL PROBABILITY PLOT (Normality)     2. RESIDUALS VS. FITTED (Linearity & Equal Var)
   Residuals ▲                                Residuals ▲
             │            █                             │       █   █     █   █
             │        █ █                               │     █   █     █       █
             │      █ █                                 ├───┼───┼───┼───┼───┼───┼───► Fitted (ŷ)
             │    █ █                                   │       █   █     █   █
             │  █                                       │     █       █     █
             └─────────────────────────►                └─────────────────────────►
               Standard Normal Quantiles                  Points bounce randomly in a band

    3. HISTOGRAM OF RESIDUALS (Normality)      4. RESIDUALS VS. ORDER (Independence)
   Frequency ▲                                Residuals ▲
             │        ┌───┐                             │   █       █       █
             │     ┌──┤   ├──┐                          │     █   █   █   █   █   █
             │  ┌──┤  │   │  ├──┐                       ├───┼───┼───┼───┼───┼───┼───► Run Order
             │──┴──┴──┴───┴──┴──┴──►                    │       █       █       █
                Residual Bins (Centered at 0)             No cyclic, drifting, or runs patterns

1. Linearity ($L$)

• How to verify: Examine the Residuals versus Fitted Values plot.
• Desirable Pattern: Residuals bounce randomly around the horizontal zero line within a parallel horizontal band, showing no discernable shape.
• Violation Pattern: A distinct curvature (e.g., U-shape or inverted U-shape) indicates that the true relationship is non-linear. The linear model systematically overpredicts in the middle and underpredicts at the tails. Correct by adding polynomial terms (e.g., $X^2$) or transforming variables.

2. Independence ($I$)

• How to verify: Examine the Residuals versus Order (Run Order) plot.
• Desirable Pattern: Random scatter across time, showing no sequences or trends.
• Violation Pattern: Residuals that drift upward over time, oscillate cyclically, or exhibit long runs above or below zero indicate autocorrelation (time-dependent lurking variables such as tool wear, ambient temperature, or shift fatigue). Standard OLS standard errors become severely biased.

3. Normality ($N$)

• How to verify: Examine the Normal Probability Plot of Residuals and conduct an Anderson-Darling test.
• Desirable Pattern: Residual points fall tightly along the straight diagonal reference line ($p > 0.05$).
• Violation Pattern: Severe S-shaped curves (indicating heavy or light tails) or severe bow shapes (indicating skewness). If non-normal, hypothesis tests ($p$-values) lose validity, especially for small sample sizes ($n < 30$).

4. Equal Variance / Homoscedasticity ($E$)

• How to verify: Examine the Residuals versus Fitted Values plot.
• Desirable Pattern: Constant vertical spread of residuals across all levels of $\hat{y}$.
• Violation Pattern (Heteroscedasticity): A distinct funnel (megaphone) shape, where residual spread widens dramatically as fitted values increase. Standard errors become invalid. Remediate using variance-stabilizing transformations (e.g., $\ln(Y)$, $\sqrt{Y}$, or Box-Cox transformation) or weighted least squares.

---

The Danger of Extrapolation

A critical analytical boundary in Six Sigma engineering is avoiding extrapolation:

Extrapolation is the practice of using a regression equation to predict output values ($Y$) for input values ($X$) that lie outside the experimental range used to calibrate the model ($x_{new} < x_{min}$ or $x_{new} > x_{max}$).

                     The Peril of Extrapolation

   Y ▲                                              EXTRAPOLATION ZONE
     │                                              (Unknown, Hazardous)
     │                       INTERPOLATION ZONE      ? ? ?
     │                       (Valid Model Domain)    ?       
     │                            ┌──────────┐     ╱
     │                            │    █  █  │   ╱ (Model assumes line continues)
     │                            │  █  █    │ ╱
     │                            │█  █      │█ (Physical Reality Crashes: Phase Change)
     │                            └──────────┘  ╲
     │                             x_min  x_max   ╲ █
     └──────────────────────────────────────────────┴────────────────► X
     0

• Interpolation (Safe): Predicting within $[x_{min}, x_{max}]$ is mathematically and physically valid because empirical data supports the model throughout this domain.
• Extrapolation (Dangerous): Outside the observed window, physical systems frequently experience regime changes: chemical catalysts saturate, metal structures hit yield points, fluids boil or freeze, and electronic components enter thermal runaway. A line that fit brilliantly between $50^\circ\text{C}$ and $100^\circ\text{C}$ may fail completely at $150^\circ\text{C}$.

---

Step-by-Step Worked Calculation Example

Problem: A packaging line engineer monitors sealing temperature ($X$, in $^\circ\text{C}$) and seal rupture force ($Y$, in Newtons). A calibration study yields the following summary statistics for $n = 10$ test pouches:

• $\bar{x} = 150.0^\circ\text{C}$, $s_x = 10.0^\circ\text{C}$, $SS_{xx} = 900.0$
• $\bar{y} = 85.0\text{ N}$, $s_y = 12.0\text{ N}$, $SS_{yy} = 1296.0$
• $SS_{xy} = 972.0$

1. Calculate the slope $b_1$ and intercept $b_0$.
2. State the regression equation.
3. Predict the rupture force for a temperature of $160.0^\circ\text{C}$.
4. If a test pouch sealed at $160.0^\circ\text{C}$ exhibits an actual rupture force of $98.0\text{ N}$, compute the residual.

Step 1: Calculate Slope ($b_1$) and Intercept ($b_0$)

$b_1 = \frac{SS_{xy}}{SS_{xx}} = \frac{972.0}{900.0} = 1.08\text{ N}/^\circ\text{C}$ $b_0 = \bar{y} - b_1 \bar{x} = 85.0 - (1.08 \times 150.0) = 85.0 - 162.0 = -77.0\text{ N}$

Step 2: Formulate the Prediction Equation

$\hat{y} = -77.0 + 1.08x$ Interpretation: For each $1.0^\circ\text{C}$ increase in sealing temperature, seal rupture force is predicted to increase by $1.08\text{ N}$. (Note: The intercept $-77.0\text{ N}$ has no physical meaning because a temperature of $0^\circ\text{C}$ is far outside the operating window).

Step 3: Predict Rupture Force at $x = 160.0^\circ\text{C}$

$\hat{y} = -77.0 + 1.08(160.0) = -77.0 + 172.8 = 95.8\text{ N}$

Step 4: Calculate the Residual ($e$)

$\text{Residual } e = y_{actual} - \hat{y}_{predicted} = 98.0 - 95.8 = +2.2\text{ N}$ The actual seal was $2.2\text{ N}$ stronger than predicted by the regression line.

---

Critical Exam Traps to Avoid

• Trap 1: Extrapolating Beyond the Observed Range of $X$ — Questions will present a model built on temperatures between $100^\circ\text{C}$ and $200^\circ\text{C}$ and ask for a prediction at $450^\circ\text{C}$. The correct answer is to flag extrapolation as statistically hazardous.
• Trap 2: Declaring a Model Valid Based on High $R^2$ Alone — A regression model can boast $R^2 = 95%$ while suffering from severe heteroscedasticity, strong autocorrelation, or severe non-linearity. High $R^2$ never excuses omitting residual diagnostic plots.
• Trap 3: Misinterpreting the Intercept ($b_0$) — In many physical processes, $b_0$ has no realistic operational interpretation if $X = 0$ is physically impossible (e.g., zero human height, zero absolute pressure, or zero absolute temperature in Celsius).
• Trap 4: Inverting the Residual Formula — Remember that residual is always Actual minus Predicted ($e = y - \hat{y}$). Inverting it as $\hat{y} - y$ will reverse all diagnostic signs.