6.10 Probability, Conditional Probability & Bayes' Theorem

Classical Definition and Sample Spaces

For an experiment with a finite set of equally likely outcomes, the probability of an event $A$ is

$P(A) = \frac{\text{number of favourable outcomes}}{\text{total number of outcomes}}$

with $0 \leq P(A) \leq 1$. The complement satisfies $P(\bar{A}) = 1 - P(A)$, and the complement route is very often the shorter one — "at least one" problems are almost always solved as $1 - P(\text{none})$.

Common sample space sizes worth having ready: one die has 6 outcomes; two dice have 36; a standard deck has 52 cards, of which 26 are red, 13 per suit, 12 face cards and 4 aces; tossing $n$ coins has $2^n$ outcomes.

The Addition Rule

For any two events,

$P(A \cup B) = P(A) + P(B) - P(A \cap B)$

The subtraction removes the double count of the overlap. When the events are mutually exclusive they cannot occur together, so $P(A \cap B) = 0$ and the rule simplifies to a plain sum.

For three events the pattern extends by inclusion-exclusion:

$P(A\cup B\cup C) = \sum P(A) - \sum P(A\cap B) + P(A\cap B\cap C)$

Independence Versus Mutual Exclusivity

These two ideas are constantly confused, and CIL-style papers reward telling them apart.

Note the consequence: two events with non-zero probabilities cannot be both mutually exclusive and independent. If they are mutually exclusive, knowing that $A$ occurred tells you with certainty that $B$ did not — the strongest possible dependence.

Conditional Probability and the Multiplication Rule

The probability of $A$ given that $B$ has occurred is

$P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \qquad P(B) > 0$

Rearranging gives the multiplication rule $P(A\cap B) = P(B),P(A\mid B)$. For independent events this collapses to $P(A)P(B)$, which is precisely the definition of independence.

Worked example — sampling without replacement. A bin holds 6 conforming and 4 defective bearings. Two are drawn without replacement. The probability that both are defective is

$P = \frac{4}{10}\times\frac{3}{9} = \frac{12}{90} = \frac{2}{15}$

The second fraction uses 3 and 9 because the first draw is not replaced — this is conditional probability in action. Had the draw been with replacement, the events would be independent and the answer would be $(4/10)^2 = 4/25$.

Total Probability and Bayes' Theorem

Suppose the events $B_1, B_2, \ldots, B_n$ partition the sample space — they are mutually exclusive and collectively exhaustive. The law of total probability gives

$P(A) = \sum_{i=1}^{n} P(B_i)\,P(A\mid B_i)$

Bayes' theorem then reverses the conditioning:

$P(B_k \mid A) = \frac{P(B_k)\,P(A\mid B_k)}{\displaystyle\sum_{i=1}^{n} P(B_i)\,P(A\mid B_i)}$

In words: the prior probability $P(B_k)$ is updated by the evidence $A$ into the posterior $P(B_k\mid A)$. Bayes questions are recognisable by their phrasing — a defect is found, and you are asked which machine most likely produced it.

Worked example: which machine made the defective part?

A CIL central workshop runs three machines producing identical bushes:

A bush drawn at random is found defective. What is the probability it came from $M_3$?

First the total probability of a defect:

$P(D) = 0.50(0.03) + 0.30(0.04) + 0.20(0.05) = 0.015 + 0.012 + 0.010 = 0.037$

Then Bayes:

$P(M_3 \mid D) = \frac{0.010}{0.037} = 0.270$

So although $M_3$ makes only 20% of the output, it accounts for 27% of the defectives — the posterior is shifted upward because its defect rate is the worst. Note also $P(M_1\mid D) = 0.015/0.037 = 0.405$: machine 1 still produces the largest number of defectives simply because it makes the most parts. Distinguishing "highest defect rate" from "most defectives produced" is the conceptual point such items test.

Odds and Expectation

Odds in favour of an event are the ratio of favourable to unfavourable outcomes, so odds of $a{:}b$ correspond to $P = a/(a+b)$. The mathematical expectation of a random variable taking values $x_i$ with probabilities $p_i$ is

$E(X) = \sum p_i x_i$

A game or contract is called fair when its expectation is zero. Expectation is developed further in the next section, where it becomes the mean of a probability distribution.