Regression, Logistic Regression, and Classification Trees

Regression, Logistic Regression, and Classification Trees

Quick Answer: Match the model to the outcome scale and the decision. Use linear regression for continuous targets, logistic regression for binary events framed as probabilities/odds, and trees / random forests when nonlinear splits and interactions matter—trading interpretability for complexity. CBDA scenarios emphasize fit-for-purpose choice and accuracy validation, not coding fluency.

This section deepens competency 3.4 model-family awareness into decision-ready judgment. You will see the same business theme (churn, demand, claim risk) solved differently depending on how the research question is measured.

Linear Regression: Continuous Outcomes

Linear regression estimates a continuous dependent variable Y from one or more predictors X. In multiple regression, several predictors enter the model together. Conceptually:

• The intercept is the expected Y when predictors are at their baseline (often zero or centered—context matters).
• A coefficient for predictor Xⱼ is the estimated average change in Y associated with a one-unit increase in Xⱼ, holding other predictors in the model constant.
• The model assumes an approximately linear relationship (or linear after transformation), additive effects (unless interactions are added), and error behavior that makes ordinary estimates reasonable.

Business interpretation examples

• “Each additional year of tenure is associated with $12 higher monthly spend, holding plan type and region constant.”
• “Each extra day in stage 3 is associated with 0.4 fewer closed deals per quarter in this segment model.”

Interpretation caveats CBDA cares about:

• Association ≠ causation unless design (experiment, strong quasi-experiment, credible causal identification) supports causal claims.
• Coefficients depend on which other variables are included (omitted variable bias conceptually).
• Scaling matters: a coefficient on income in dollars looks tiny next to a coefficient on a 0–1 flag; standardize or communicate units carefully.
• Outliers and leverage points can dominate fits—link back to Domain 2 outlier investigation rather than silent deletion.

When linear regression is a strong choice: forecasting continuous KPIs, estimating magnitude of drivers for continuous metrics, establishing transparent baselines before more complex models, and settings where stakeholders must audit coefficients.

When it struggles: highly nonlinear relationships, strong interactions that are painful to hand-specify, heavy-tailed outcomes, or when the true decision needs a probability of an event rather than a continuous level.

Logistic Regression: Binary Outcomes and Odds Framing

Many business events are binary: churned or not, defaulted or not, converted or not, failed inspection or not. Feeding 0/1 outcomes into plain linear regression can produce probabilities outside [0, 1] and awkward residual behavior. Logistic regression models the log-odds (logit) of the event as a linear function of predictors, then maps back to a probability between 0 and 1.

Odds framing without heavy math

• Probability p = chance the event occurs (e.g., 0.20 = 20% chance of churn).
• Odds = p / (1 − p). If p = 0.20, odds = 0.25 (“1 to 4”).
• A logistic coefficient relates to change in log-odds; practitioners often communicate odds ratios (exponentiated coefficients): “Holding other factors constant, customers on monthly plans have 1.8× the odds of churning versus annual plans.”

You do not need to derive the likelihood. You do need to:

• State that outputs are probabilities (or scores monotone with risk), not destiny.
• Choose a threshold (e.g., contact if p > 0.35) based on business costs of false positives vs false negatives, not a default 0.50 if costs are asymmetric.
• Watch class imbalance: if only 2% churn, a naive model predicting “no” is 98% accurate and useless; use appropriate metrics (precision/recall, lift, ROC concepts—next section) and sampling awareness from Domain 2.

Logistic regression is often the CBDA “default respectable classifier” when explainability, governance, and moderate tabular data dominate.

Decision Trees and Random Forests: Rules vs Ensembles

Single decision trees

A classification tree (or regression tree) recursively splits features to improve purity or reduce error. The result is a flowchart of business-readable rules:

• IF tickets_last_90d ≥ 4 AND tenure_months < 6 THEN high_churn_risk
• ELSE IF contract = monthly AND discount = none THEN medium_risk
• ELSE low_risk

Strengths: interpretability, natural handling of nonlinearities and interactions, minimal scaling of inputs, easy stakeholder walkthroughs.

Weaknesses: high variance (small data changes → different trees), tendency to overfit deep trees, discontinuous predictions at split boundaries.

Pruning, minimum leaf size, and maximum depth are conceptual controls against over-complexity.

Random forests

A random forest builds many trees on bootstrap samples with random feature subsets and aggregates their votes or averages. Benefits: usually better predictive accuracy and stability than one tree; partial resistance to overfitting relative to an unpruned single tree. Costs: the full ensemble is harder to explain case-by-case; computation and monitoring overhead rise; stakeholders may still demand global feature importance and local explanations.

CBDA judgment call: if a compliance officer must approve every automated decline with a human-readable rule path, a constrained single tree or logistic model may beat a forest. If the decision is internal prioritization of outreach and lift on holdout data is the KPI, a forest may be justified—with monitoring and documented limits.

When to Choose Which for Business Scenarios

Use this decision pattern on exam stems:

1. What is the outcome type? Continuous → regression family; binary/multi-class → classification (logistic, trees, forests, etc.).
2. How will the answer be used? Board narrative and policy → favor interpretable models. High-volume ranking with monitoring → ensembles may be acceptable.
3. What data reality exists? Sparse labels, small n, messy features → simpler models. Large n, complex interactions proven in EDA → trees/ensembles.
4. What constraints exist? Regulation, ethics, real-time latency, skill in the org (Domain 6 readiness themes).
5. What is already known? If descriptive baselines and diagnostics are missing, do not start with the most complex classifier.

Validating Predictive Model Accuracy (IIBA Sample Flavor)

IIBA-aligned CBDA thinking treats validation as part of analysis, not a postscript. Two complementary ideas appear in professional practice and sample-style scenarios:

Statistical / holdout validation (often “regression” and classification metrics)

• Fit on training data; score held-out cases.
• For continuous outcomes: error metrics (MAE, RMSE concepts), R² as variance explained (with humility—high R² can still be decision-useless; low R² can still rank well).
• For classifiers: confusion-matrix rates, precision/recall tradeoffs, ROC/AUC awareness, calibration (“do predicted 30% events happen ~30% of the time?”).

Decision-process or simulation-style validation

Accuracy in the abstract is not enough. Ask: if we had used this model last quarter to allocate offers, what would have happened? That may involve:

• Backtesting the ranking policy on historical periods.
• Comparing model-guided allocation to business-as-usual on a pilot or A/B test when feasible.
• Stressing the model under seasonality, new product mix, or policy change.
• Checking that predicted scores still align with the research question population (no silent filter that drops the customers leadership cares about).

Simulation versus regression in exam language is not “pick a software product.” It is recognizing when you need to replay the decision (policy simulation, what-if, backtest of actions) versus when you need to estimate a relationship (regression coefficients, predicted values). A demand regression might forecast units well yet fail a capacity simulation if constraints, lead times, or service levels are ignored. A churn logistic model might have solid AUC yet fail a retention simulation if the offer changes the customers’ behavior in ways not in the training data (intervention bias).

CBDA-quality close: report accuracy and decision usefulness, with assumptions.

Practical Workflow Snapshot

1. Lock outcome definition, time horizon, and population (Domains 1–2).
2. Choose family by outcome and constraints.
3. Engineer only features available at decision time (no leakage).
4. Train with validation discipline; keep a final holdout or time holdout.
5. Interpret coefficients/rules/importances with business SMEs.
6. Validate metrics and a simple decision simulation or pilot plan.
7. Document limits before Domain 4 storytelling.

Master this workflow and most “which model?” stems become straightforward.