10.2 Simple Linear Regression & Prediction

Simple Linear Regression & Prediction

Quick Answer: Simple linear regression develops a predictive mathematical equation—$\hat{Y} = b_0 + b_1 X$—quantifying how a continuous dependent response ($Y$) changes when an independent predictor ($X$) is varied. Using Ordinary Least Squares (OLS), regression calculates the unique straight line that minimizes the sum of squared vertical residuals between data points and the line. The coefficient of determination ($R^2$) reflects the percentage of total variation explained by the model. Regression provides reliable predictions within the experimental data range, but extrapolating beyond observed boundaries creates severe operational risk. Independent CSSYB study guide by OpenExamPrep.

Purpose of Simple Linear Regression

Once scatter plots and correlation confirm a linear association between process variables, Six Sigma teams must quantify that link: How much does output $Y$ change per unit change in input $X$? What setting of $X$ achieves our target $Y$?

While correlation measures the degree of association, simple linear regression constructs a functional mathematical transfer function ($Y = f(X)$). It establishes an explicit relationship between:

• One Independent Predictor ($X$): The controllable process setting, raw material parameter, or operational condition.
• One Dependent Response ($Y$): The output quality characteristic, dimensional measurement, or cycle time metric targeted for improvement.

This predictive model enables teams to evaluate process sensitivity, establish operating tolerances, and design data-driven solutions.

---

The Linear Regression Model

Regression differentiates between the true population relationship and the sample model estimated from empirical data.

Theoretical Population Model

The theoretical relationship across the entire process population is expressed as:

$$Y = \beta_0 + \beta_1 X + \epsilon$$

Where $\beta_0$ is the population intercept, $\beta_1$ is the population slope, and $\epsilon$ represents independent, normally distributed random error ($\epsilon \sim N(0, \sigma^2)$).

Fitted Sample Regression Equation

Using sample data, practitioners compute the fitted regression line:

$$\hat{Y} = b_0 + b_1 X$$

Where:

• $\hat{Y}$ ("Y-hat"): The predicted mean response for a specified value of $X$.
• $b_1$ (Slope): The estimated change in response $Y$ for each 1-unit increase in predictor $X$. The slope represents the rate of change ($\Delta Y / \Delta X$). A positive slope indicates $Y$ increases as $X$ increases, whereas a negative slope indicates an inverse relationship.
• $b_0$ ($Y$-Intercept): The estimated value of $\hat{Y}$ when $X = 0$. If $X = 0$ is outside operating reality (such as machine speed at 0 RPM during production), $b_0$ serves purely as a coordinate anchor without physical meaning.

---

The Method of Ordinary Least Squares (OLS)

To select the single best-fitting line among infinite candidates, statisticians use the Method of Ordinary Least Squares (OLS).

Residuals and Minimization

For every observation $(X_i, Y_i)$, the vertical distance between the actual observed value ($Y_i$) and the fitted value ($\hat{Y}_i$) is the residual ($e_i$):

$$e_i = Y_i - \hat{Y}_i = Y_i - (b_0 + b_1 X_i)$$

The sum of raw residuals always equals zero because positive and negative deviations cancel. OLS eliminates cancellation and penalizes large discrepancies by minimizing the Sum of Squared Errors (SSE):

$$\text{SSE} = \sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2$$

Minimizing SSE produces the closed-form OLS formulas:

$$b_1 = \frac{\sum (X_i - \bar{X})(Y_i - \bar{Y})}{\sum (X_i - \bar{X})^2} = r \left( \frac{s_Y}{s_X} \right)$$ $$b_0 = \bar{Y} - b_1 \bar{X}$$

The fitted OLS line always passes through the point of averages $(\bar{X}, \bar{Y})$.

---

Coefficient of Determination ($R^2$)

To assess model adequacy, Six Sigma practitioners evaluate the Coefficient of Determination ($R^2$).

Regression partitions the total variation in the response variable into two additive components:

$$\text{Total Sum of Squares (SST)} = \text{Regression Sum of Squares (SSR)} + \text{Error Sum of Squares (SSE)}$$

Where SST measures total variation around $\bar{Y}$, SSR measures variation explained by the regression line, and SSE measures unexplained residual noise. The coefficient of determination is:

$$R^2 = \frac{\text{SSR}}{\text{SST}} = 1 - \frac{\text{SSE}}{\text{SST}}$$

Key Properties and Interpretation

• Range: $0.0 \le R^2 \le 1.0$ (or $0\%$ to $100\%$).
• Relation to Pearson $r$: In simple linear regression, $R^2 = (r)^2$. If $r = 0.90$, then $R^2 = 0.81$.
• Practical Interpretation: An $R^2$ of $0.85$ means that $85\%$ of the total variation in response $Y$ is explained by the linear relationship with predictor $X$. The remaining $15\%$ represents unexplained common-cause variation or unmeasured variables.

---

Prediction: Interpolation vs. Extrapolation Risks

A validated regression equation allows practitioners to predict response values, but Yellow Belts must recognize the danger of extrapolation.

• Interpolation (Statistically Valid): Predicting $\hat{Y}$ for predictor values within the observed data range ($X_{\min} \le X \le X_{\max}$). Interpolated predictions are statistically valid and supported by empirical evidence.
• Extrapolation (Severe Risk): Predicting $\hat{Y}$ for predictor values outside the observed data range ($X < X_{\min}$ or $X > X_{\max}$).

Extrapolating assumes the linear relationship continues indefinitely. In reality, physical processes experience saturation, material degradation, or structural failure outside tested operating ranges, yielding misleading or impossible predictions.

---

Step-by-Step Worked Numerical Example

A Six Sigma team in a medical device facility evaluates sealing bar pressure in psi ($X$) versus package seal strength in Newtons ($Y$).

1. Data and Summary Values

Five test samples yield:

• $\bar{X} = 50\text{ psi}$, $\bar{Y} = 80\text{ N}$
• $\sum (X_i - \bar{X})^2 = 1,000$
• $\sum (X_i - \bar{X})(Y_i - \bar{Y}) = 1,550$
• Experimental range: $30\text{ psi} \le X \le 70\text{ psi}$

2. Computing Slope and Intercept

$$b_1 = \frac{1,550}{1,000} = 1.55\text{ N/psi}$$ $$b_0 = 80 - (1.55)(50) = 80 - 77.5 = 2.5\text{ N}$$

The fitted prediction model is:

$$\hat{Y} = 2.5 + 1.55 X$$

Interpretation: Each additional psi of sealing pressure increases predicted seal strength by $1.55\text{ N}$.

3. Model Evaluation and Prediction

Software output for the same five samples reports $r = 0.992$, so $R^2 = (0.992)^2 = 0.984$ ($98.4\%$ of seal strength variation is explained by pressure).

• Valid Interpolation: At $X = 55\text{ psi}$ (within the 30–70 psi range):

$$\hat{Y} = 2.5 + 1.55(55) = 87.75\text{ N}$$

• Extrapolation Danger: At $X = 200\text{ psi}$, the equation predicts $\hat{Y} = 312.5\text{ N}$. In production, 200 psi would crush the blister pack, causing catastrophic seal failure.