1.2 Sampling, Hypothesis Testing & Regression Analysis

1.2 Sampling, Hypothesis Testing & Regression Analysis

1. From Population to Sample: Sampling Distributions and Standard Error

Every statistic an investment consultant quotes — a manager's average excess return, an asset class's volatility, a factor loading — is a sample estimate drawn from an unobservable population. The CIMA exam repeatedly tests whether a candidate understands that an estimate is a random variable with its own distribution, not a fact.

The sampling distribution of the sample mean has a standard deviation called the standard error of the mean:

$SE_{\bar{x}} = \frac{s}{\sqrt{n}}$

where $s$ is the sample standard deviation and $n$ is the number of observations. Because $n$ enters under a square root, precision improves slowly: to halve the standard error you must quadruple the sample. This single fact explains why performance track records are so weak as evidence.

The Central Limit Theorem (CLT) states that the sampling distribution of the mean approaches a normal distribution as $n$ grows, regardless of the shape of the underlying population. In practice, $n \geq 30$ is the conventional threshold. The CLT is what licenses the use of normal-based confidence intervals on mean returns even when the return series itself is skewed and fat-tailed.

When the population standard deviation is unknown — always the case in practice — the correct reference distribution is Student's t with $n - 1$ degrees of freedom, not the standard normal. The t-distribution has fatter tails, so it produces wider (more honest) confidence intervals. The two converge as $n$ rises above roughly 120.

A confidence interval for the mean is:

$\bar{x} \pm t_{\alpha/2,\, n-1} \times \frac{s}{\sqrt{n}}$

2. The Hypothesis-Testing Framework

A hypothesis test formalizes the question "could this result plausibly be luck?" The structure is fixed:

1. State the null hypothesis $H_0$ — the position of no effect, e.g. "the manager's true alpha is zero."
2. State the alternative $H_a$ — e.g. "true alpha is not zero" (two-tailed) or "true alpha is positive" (one-tailed).
3. Choose a significance level $\alpha$, conventionally 0.05.
4. Compute the test statistic, e.g. $t = (\bar{x} - \mu_0) / SE$.
5. Compare the statistic to the critical value, or compare the p-value to $\alpha$.

The p-value is the probability of observing a result at least as extreme as the one obtained if the null were true. It is not the probability that the null is true — a distinction the exam tests directly.

Two errors are possible:

Lowering $\alpha$ to reduce false positives mechanically raises $\beta$ and reduces power. Only a larger sample improves both simultaneously.

Worked Example: Is This Manager's Alpha Real?

A consultant reviews a large-cap manager with 36 monthly excess returns. The sample mean monthly excess return is +0.90% with a sample standard deviation of 4.20%.

1. Standard error: $SE = 4.20% / \sqrt{36} = 4.20% / 6 = 0.70%$
2. Test statistic: $t = (0.90% - 0%) / 0.70% = 1.286$
3. Critical value: two-tailed, $\alpha = 0.05$, df = 35 $\rightarrow$ 2.030
4. Decision: $1.286 < 2.030$, so fail to reject $H_0$.

Despite an annualized excess return of roughly 10.8%, three years of data cannot distinguish this manager from luck. Solving for the sample size that would be needed at this signal-to-noise ratio requires $t \geq 2$, i.e. $n \geq (2 \times 4.20 / 0.90)^2 \approx 87$ months — more than seven years. This is the quantitative justification for the consulting convention of requiring full market-cycle track records.

3. Simple Linear Regression: Estimating Beta and Alpha

The single-index model regresses a fund's excess return on the benchmark's excess return:

$R_{p,t} - R_{f,t} = \alpha + \beta (R_{m,t} - R_{f,t}) + \varepsilon_t$

Ordinary least squares (OLS) chooses $\alpha$ and $\beta$ to minimize the sum of squared residuals. The slope has a closed form:

$\beta = \frac{Cov(R_p, R_m)}{Var(R_m)} = \rho_{p,m} \times \frac{\sigma_p}{\sigma_m}$

• The intercept $\alpha$ is Jensen's alpha — return unexplained by benchmark exposure.
• The slope $\beta$ is systematic risk relative to the benchmark.
• R-squared is the proportion of the fund's return variance explained by the benchmark; in a single-factor regression $R^2 = \rho^2$.
• The standard error of the estimate measures typical residual size; residual (non-systematic) risk is the volatility of $\varepsilon$.

Worked Example: Beta and R-Squared

A fund has $\sigma_p = 18.0%$, the benchmark has $\sigma_m = 16.0%$, and their correlation is $\rho = 0.90$.

$\beta = 0.90 \times \frac{18.0}{16.0} = 0.90 \times 1.125 = 1.0125$

$R^2 = 0.90^2 = 0.81$

81% of the fund's return variation is benchmark-driven; the remaining 19% is idiosyncratic. An $R^2$ this high indicates the benchmark is appropriate — a critical input to the benchmark-selection process covered in the manager-monitoring material.

4. Multiple Regression and Model Diagnostics

Multi-factor attribution extends the model to several explanatory variables:

$R_p - R_f = \alpha + \beta_1 F_1 + \beta_2 F_2 + \dots + \beta_k F_k + \varepsilon$

Because adding any regressor mechanically raises $R^2$, multiple regression is judged on adjusted R-squared, which penalizes the addition of variables that do not earn their degrees of freedom:

$R^2_{adj} = 1 - (1 - R^2)\frac{n-1}{n-k-1}$

Adjusted $R^2$ can fall when a useless factor is added — that decline is the signal to drop it.

OLS inference relies on assumptions, and violations are a favorite exam target because each one has a different consequence:

The practical consulting warning: serial correlation is pervasive in private real estate, private credit, and fund-of-funds returns because valuations are appraisal-based and smoothed. Smoothed returns understate true volatility, understate beta, and inflate Sharpe ratios. A Durbin–Watson statistic materially below 2.0 (positive autocorrelation) should trigger an unsmoothing adjustment before those returns enter a mean-variance optimizer.