3.5 Extensions of Multiple Regression: Influence Analysis, Qualitative Variables & Logistic Regression

3.5 Extensions of Multiple Regression: Influence Analysis, Qualitative Variables & Logistic Regression

Blueprint note: this is a distinct learning module in the Level II Quantitative Methods topic, separate from the misspecification module. Its three learning outcomes are influence analysis, qualitative independent variables, and logistic regression. Vignettes present regression output tables and diagnostic statistics and ask you to interpret them, not to run the estimation.

---

1. Influence Analysis: Which Observations Are Driving the Fit?

A single data point can dominate an estimated relationship. Influence analysis separates two distinct problems.

Leverage measures how unusual an observation is in the independent variable space. For observation i, the leverage $h_{ii}$ is the i-th diagonal element of the hat matrix, and it satisfies $0 \le h_{ii} \le 1$ with the leverages summing to $k+1$, where k is the number of independent variables and n the sample size. Because the average leverage is therefore $(k+1)/n$, the standard screening rule is:

An observation is potentially influential when $h_{ii} > 3,(k+1)/n$.

Outliers are unusual in the dependent variable: the model fits them badly. The scale-free measure is the studentized residual, computed by deleting observation i, re-estimating the model on the remaining $n-1$ observations, and standardising the resulting prediction error. A common screen flags $|t_i^*| > 3$, though the formal test compares the studentized residual to a critical value from the t distribution with $n - k - 2$ degrees of freedom.

The distinction matters because the two problems have different consequences:

Cook's distance combines both. It measures how much all fitted values move when observation i is deleted:

$D_i = \frac{e_i^2}{k \times MSE} \left[ \frac{h_{ii}}{(1 - h_{ii})^2} \right]$

where $e_i$ is the residual and MSE is the mean squared error of the full regression. Interpretation rules used at Level II:

• $D_i > 0.5$ — the observation may be influential and deserves investigation;
• $D_i > 1.0$ — the observation is highly likely to be influential;
• an alternative screen compares $D_i$ to $\sqrt{k/n}$.

What to do about an influential point is a judgement question that vignettes love. Deleting data because it is inconvenient is not acceptable. The defensible sequence is: verify the observation is not a data error; if it is genuine, consider whether the model is misspecified for that regime (a crisis quarter may require a dummy variable); report results with and without the observation; and disclose the sensitivity. Note the link to Ethics Standard V(A) — silently dropping the observation that reverses your conclusion is not a reasonable and adequate basis.

---

2. Qualitative Independent Variables

A dummy variable takes the value 1 when a condition holds and 0 otherwise, letting a regression absorb categorical information such as industry, exchange listing, credit-rating band, quarter, or a policy regime.

The dummy variable trap

If a qualitative characteristic has n mutually exclusive categories, include exactly n − 1 dummies when the model has an intercept. The omitted category becomes the base (reference) case and is captured by the intercept. Including all n dummies plus an intercept creates an exact linear dependence — perfect multicollinearity — and the regression cannot be estimated.

Example with four rating buckets (AAA–AA, A, BBB, high yield) and high yield as the base:

$\text{Spread}_i = b_0 + b_1 D_{AAA/AA,i} + b_2 D_{A,i} + b_3 D_{BBB,i} + b_4 \text{Leverage}_i + \varepsilon_i$

• $b_0$ is the estimated spread for a high-yield issuer with zero leverage;
• $b_1$ is the estimated spread difference between an AAA–AA issuer and a high-yield issuer, holding leverage constant;
• a t-test on $b_1$ tests whether that difference is statistically significant, not whether AAA–AA spreads are non-zero.

Intercept dummies versus slope (interaction) dummies

An intercept dummy shifts the level of the line. An interaction dummy, formed by multiplying the dummy by a continuous regressor, changes the slope:

$R_i = b_0 + b_1 D_i + b_2 X_i + b_3 (D_i \times X_i) + \varepsilon_i$

• When $D_i = 0$: intercept $b_0$, slope $b_2$.
• When $D_i = 1$: intercept $b_0 + b_1$, slope $b_2 + b_3$.

So $b_1$ answers "is the level different?" and $b_3$ answers "is the sensitivity different?" A vignette showing a market-beta regression with a recession dummy and a recession-times-market interaction is asking whether beta itself rises in recessions ($b_3$), not merely whether average returns are lower ($b_1$).

Seasonal dummies are the same tool applied to time: three quarterly dummies with the fourth quarter as the base, or eleven monthly dummies.

3. Logistic Regression: Modelling a Binary Outcome

When the dependent variable is binary — default or no default, acquired or not acquired, breach of covenant or not — linear regression is inappropriate. A linear probability model can predict probabilities below 0 or above 1, its errors are heteroskedastic by construction, and its errors cannot be normally distributed.

Logistic regression solves this by modelling the log odds as a linear function of the regressors:

$\ln\!\left(\frac{p}{1-p}\right) = b_0 + b_1 X_1 + b_2 X_2 + \dots + b_k X_k$

where $p$ is the probability that the event occurs. Solving back for the probability gives the logistic (sigmoid) function:

$p = \frac{1}{1 + e^{-(b_0 + b_1 X_1 + \dots + b_k X_k)}}$

which is bounded strictly between 0 and 1 and is S-shaped, so extreme values of the regressors move the fitted probability by progressively less.

What the coefficients mean

• $b_j$ is the change in the log odds for a one-unit increase in $X_j$, holding other variables constant.
• $e^{b_j}$ is the odds ratio: the multiplicative change in the odds. A coefficient of 0.40 means the odds are multiplied by $e^{0.40} = 1.49$, a 49% increase in the odds — not a 49% increase in the probability.
• The effect on the probability is non-linear and depends on the starting point, which is why vignettes ask for the fitted probability at stated regressor values rather than a marginal effect.

Estimation and model fit

Logistic regression is estimated by maximum likelihood, not ordinary least squares, so there is no $R^2$ in the OLS sense and no F-statistic. Instead:

• Likelihood ratio (LR) test: $LR = -2,(\text{log-likelihood of the restricted model} - \text{log-likelihood of the unrestricted model})$, compared against a chi-square distribution. It plays the role the F-test plays in linear regression.
• Individual coefficients are tested with a z-statistic, coefficient divided by its standard error.
• A pseudo-$R^2$ may be reported but is not comparable across model types and should not be read as explained variance.

Worked interpretation

A credit analyst estimates a one-year default model on 900 issuers using two regressors, net debt to EBITDA and interest coverage:

$\ln\!\left(\frac{p}{1-p}\right) = -4.60 + 0.55\,(\text{Net debt/EBITDA}) - 0.30\,(\text{Interest coverage})$

For an issuer with net debt to EBITDA of 5.0 times and interest coverage of 2.0 times:

• Log odds $= -4.60 + 0.55(5.0) - 0.30(2.0) = -4.60 + 2.75 - 0.60 = -2.45$
• Odds $= e^{-2.45} = 0.0863$
• Probability $= 0.0863 / (1 + 0.0863) = 0.0794$, about 7.9%

Raising net debt to EBITDA by one turn to 6.0 times moves the log odds to $-1.90$, odds to $e^{-1.90} = 0.1496$, and the probability to 13.0%. The odds rose by the factor $e^{0.55} = 1.73$ exactly, while the probability rose by 5.1 percentage points — a figure that would be different if the starting probability were different. That asymmetry is the single most-tested idea in the module.

Level II traps in this learning module

1. Reading a logistic coefficient as a change in probability rather than in log odds.
2. Including all n dummies with an intercept and then being asked why the software dropped one.
3. Treating a high-leverage point as an outlier, or vice versa — they are measured on different axes and have different remedies.
4. Assuming Cook's distance has a t-distribution; it does not, and the exam uses the 0.5 and 1.0 rules of thumb.
5. Applying an F-test to a logistic model; the analogous test is the likelihood ratio test.