7.3 Multivariate Tools: Factor Analysis, Discriminant Analysis, and MANOVA

Why multivariate methods exist

Two problems motivate them.

Error rate inflation. Testing 10 responses separately at $\alpha = 0.05$ gives a family-wise error rate of $1 - 0.95^{10} = 0.40$. Roughly two chances in five of at least one false positive, before any real effect exists.

Missed joint structure. Two responses may each show no significant difference between groups, while their combination separates the groups cleanly. Univariate tests examined one variable at a time cannot see a relationship that lives in the correlation between them.

Factor analysis

Factor analysis assumes that a set of observed, correlated variables is driven by a smaller number of unobserved latent factors plus variable-specific error. Each observed variable is modelled as a weighted sum of the factors:

$X_j = \lambda_{j1}F_1 + \lambda_{j2}F_2 + \cdots + \lambda_{jm}F_m + e_j$

The weights $\lambda_{jk}$ are loadings, and a variable's communality is the share of its variance explained by the common factors.

Uses in Six Sigma:

• Survey and VOC reduction. Twenty customer satisfaction questions frequently reduce to three or four underlying dimensions -- reliability, responsiveness, value -- which then become the CTQs.
• Finding common causes of variation. If eight process measurements load heavily on one factor, they are being driven by a shared source such as ambient conditions or a common utility supply, and chasing them individually is wasted effort.

Interpretation is helped by rotation, most commonly varimax, which redistributes loadings so each variable loads strongly on one factor and weakly on the rest without changing the total variance explained.

Factor analysis versus principal component analysis

They are often confused, and the distinction is worth knowing:

Use PCA when the goal is dimension reduction for downstream modelling; use factor analysis when the goal is to identify an interpretable underlying cause.

Discriminant analysis

Discriminant analysis finds the linear combination of predictors that maximally separates two or more known groups, and then uses it to classify new observations.

$D = w_1X_1 + w_2X_2 + \cdots + w_pX_p$

The weights are chosen to maximize the ratio of between-group to within-group variation on $D$.

Six Sigma applications:

• Classifying incoming material lots as likely-good or likely-problem from a panel of measured properties.
• Separating warranty-return units from survivors using in-process measurements, which points at the measurements worth controlling.
• Deciding which of several failure modes a unit exhibits from sensor data.

Key contrast: discriminant analysis has known group labels and predicts membership; cluster analysis has no labels and discovers groups. Confusing them is a standard exam distractor. Logistic regression addresses a similar problem to two-group discriminant analysis but makes weaker distributional assumptions and is usually preferred when the predictors are not multivariate normal.

MANOVA

MANOVA extends ANOVA to several dependent variables at once. Instead of asking whether group means differ on one response, it asks whether the vector of means differs.

• $H_0$: the mean vectors of all groups are equal.
• $H_1$: at least one group's mean vector differs.

Test statistics include Wilks' lambda (the most commonly reported), Pillai's trace, the Hotelling-Lawley trace, and Roy's largest root. Pillai's trace is the most robust to violations of assumptions.

When to use MANOVA rather than several ANOVAs:

1. When the responses are conceptually related -- tensile strength, elongation, and hardness of the same specimen.
2. When the responses are correlated, so separate tests both waste information and inflate the error rate.
3. When a difference may exist in the combination of responses rather than in any one of them.

The usual workflow is MANOVA first as an omnibus test; only if it is significant do you examine the individual responses, which protects the family-wise error rate.

Assumptions: multivariate normality of the responses within each group, homogeneity of the variance-covariance matrices (tested with Box's M), and independent observations. Sample size must exceed the number of dependent variables in every group, and comfortably so.

Choosing among them

Two cautions apply to all of them. They need considerably more data than univariate methods -- rules of thumb range from 5 to 10 observations per variable as a minimum. And they are tagged Understand in the Body of Knowledge: the exam expects you to select the right tool for a described situation, not to compute Wilks' lambda by hand.