7.2 Linear Regression and Residual Analysis

Regression and correlation analysis are core statistical modeling tools utilized in the Analyze phase to quantify relationships between continuous process inputs ($X$) and continuous output metrics ($Y$). They enable Black Belts to establish empirical transfer functions $Y = f(X)$ for process optimization and control.

---

Simple Linear Regression Modeling

Simple linear regression fits an empirical straight line to predict a continuous response variable $Y$ from a single continuous predictor variable $X$:

$Y = \beta_0 + \beta_1 X + \epsilon$

Where:

• $\beta_0$: Intercept parameter (value of $Y$ when $X = 0$).
• $\beta_1$: Slope parameter (change in $Y$ per unit increase in $X$).
• $\epsilon$: Random error term, assumed $\epsilon \sim N(0, \sigma^2)$.

Least Squares Estimation Formulas

$\hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} = r \left( \frac{s_y}{s_x} \right)$

$\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}$

---

Model Evaluation Metrics: $R^2$ and Adjusted $R^2$

Coefficient of Determination ($R^2$)

Represents the proportion of total variation in the response variable $Y$ explained by the regression model:

$R^2 = \frac{\text{SS}_{\text{regression}}}{\text{SS}_{\text{total}}} = 1 - \frac{\text{SS}_{\text{residual}}}{\text{SS}_{\text{total}}}$

Adjusted $R^2$ ($R^2_{\text{adj}}$)

Penalizes the model for adding unnecessary predictor variables that do not significantly improve fit:

$R^2_{\text{adj}} = 1 - \left[ \frac{(1 - R^2)(n - 1)}{n - p - 1} \right]$

Where $p$ is the number of predictor terms in the model.

---

Residual Analysis & Regression Assumptions Verification

Valid linear regression modeling requires checking four residual diagnostic assumptions (LINE):

1. Linearity: Residuals plot vs. fitted values shows a random scatter around zero without curvature.
2. Independence: Residuals show no autocorrelation across time or observation order (verified via Durbin-Watson statistic).
3. Normality: Residuals follow a Normal distribution (verified via Anderson-Darling test, $p > 0.05$).
4. Equal Variance (Homoscedasticity): Residual spread remains constant across all predicted values of $\hat{Y}$.

---

Multiple Linear Regression & Multicollinearity

Multiple linear regression incorporates $p \ge 2$ continuous or dummy predictor variables:

$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p + \epsilon$

Multicollinearity & Variance Inflation Factor (VIF)

Multicollinearity occurs when predictor variables ($X_i, X_j$) are highly correlated with each other, inflating parameter standard errors. Multicollinearity is evaluated using the Variance Inflation Factor (VIF):

$\text{VIF}_i = \frac{1}{1 - R_i^2}$

• VIF $< 5$: Acceptable low collinearity.
• VIF $\ge 5 - 10$: High collinearity; model parameters are unstable. Predictors must be centered or removed.

---

Hypothesis Testing for Regression Parameters

Once a linear regression model is fitted, statistical hypothesis tests evaluate whether regression parameters ($\beta_0, \beta_1$) differ significantly from zero.

1. $t$-Test for Slope Significance ($\beta_1$)

Evaluates $H_0: \beta_1 = 0$ (no linear relationship) vs. $H_a: \beta_1 eq 0$:

$t_0 = \frac{\hat{\beta}_1}{\text{SE}(\hat{\beta}_1)}$

Where standard error of slope is:

$\text{SE}(\hat{\beta}_1) = \frac{s_e}{\sqrt{\sum (x_i - \bar{x})^2}} = \frac{\sqrt{\text{MS}_{\text{residual}}}}{\sqrt{\text{SS}_{xx}}}$

• Decision Rule: If $|t_0| > t_{\alpha/2, n-2}$ ($p < \alpha$), reject $H_0$. Predictor $X$ significantly influences response $Y$.

2. Analysis of Variance (ANOVA) for Regression Model Significance

Evaluates overall model utility ($H_0: \beta_1 = \beta_2 = \dots = \beta_p = 0$):

---

Worked Simple Linear Regression Example

An chemical engineer models reactor yield $Y$ (%) against reaction temperature $X$ ($^\circ\text{C}$). Data from $n = 10$ pilot runs yields:

• $\bar{x} = 150.0^\circ\text{C}$, $\bar{y} = 82.0%$
• $\text{SS}_{xx} = \sum (x_i - \bar{x})^2 = 2,000.0$
• $\text{SS}_{xy} = \sum (x_i - \bar{x})(y_i - \bar{y}) = 3,600.0$
• $\text{SS}_{yy} = \sum (y_i - \bar{y})^2 = 6,800.0$

1. Calculate Regression Slope & Intercept

$\hat{\beta}_1 = \frac{\text{SS}_{xy}}{\text{SS}_{xx}} = \frac{3,600.0}{2,000.0} = 1.80\%/^\circ\text{C}$

$\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} = 82.0 - (1.80 \times 150.0) = 82.0 - 270.0 = -188.0\%$

• Fitted Regression Model: $\hat{Y} = -188.0 + 1.80 X$

2. Calculate Coefficient of Determination ($R^2$)

$\text{SS}_{\text{reg}} = \hat{\beta}_1 \text{SS}_{xy} = 1.80 \times 3,600.0 = 6,480.0$

$R^2 = \frac{\text{SS}_{\text{reg}}}{\text{SS}_{\text{yy}}} = \frac{6,480.0}{6,800.0} = 0.9529 \implies 95.29\%$

• Interpretation: $95.29%$ of the variation in reactor yield is explained by reaction temperature.

---

Prediction Intervals vs. Confidence Intervals

When predicting $Y$ at a specific predictor setting $X_0$:

• Confidence Interval for Mean Response: Quantifies uncertainty in estimating the expected mean response $E(Y|X_0)$: $\hat{y}_0 \pm t_{\alpha/2, n-2} \, s_e \sqrt{\frac{1}{n} + \frac{(x_0 - \bar{x})^2}{\text{SS}_{xx}}}$
• Prediction Interval for Single Future Observation: Quantifies uncertainty in predicting a single future individual value $Y_0$. It is significantly wider because it incorporates individual observation error: $\hat{y}_0 \pm t_{\alpha/2, n-2} \, s_e \sqrt{1 + \frac{1}{n} + \frac{(x_0 - \bar{x})^2}{\text{SS}_{xx}}}$