9.2 Discrete-Time Markov Chains: Transition Matrices and Steady-State Probabilities

Discrete-time Markov chains (DTMCs) model dynamic stochastic systems that transition between countable operational states over sequential discrete time steps. In industrial and systems engineering, Markov chains are indispensable tools for modeling machine wear and degradation, predicting preventative maintenance intervals, analyzing warranty claims, evaluating inventory replenishment policies, and forecasting customer brand-switching market shares. The NCEES FE Reference Handbook outlines core Markovian definitions under its Industrial and Systems Engineering section. Mastery of transition matrices, Chapman-Kolmogorov equations, state classification, and steady-state vector solutions is essential for solving stochastic process problems on the FE exam.

---

1. The Markov Property and Stochastic Processes

A stochastic process is a mathematical collection of random variables ${X_n, n \in T}$ indexed by time or step parameter $n$. In a discrete-time stochastic process with discrete state space $S = {0, 1, 2, \dots, M}$, $X_n$ represents the state of the system at step $n$.

The Markov Property (Memoryless Property)

A discrete-time stochastic process possesses the Markov property if the conditional probability distribution of the future state $X_{n+1}$ depends solely upon the present state $X_n$ and is completely conditionally independent of the past states $X_{n-1}, X_{n-2}, \dots, X_0$:

$P(X_{n+1} = j \mid X_n = i, X_{n-1} = i_{n-1}, \dots, X_0 = i_0) = P(X_{n+1} = j \mid X_n = i) = p_{ij}$

Informally: Given the present, the future is conditionally independent of the past. Historical trajectories that led the system to its current state provide no additional predictive power regarding its next transition.

Time-Homogeneity

A Markov chain is time-homogeneous (or stationary) if the transition probabilities between states remain constant across all time steps $n$:

$P(X_{n+1} = j \mid X_n = i) = P(X_1 = j \mid X_0 = i) = p_{ij} \quad \text{for all } n \ge 0$

All standard FE exam Markov problems assume time-homogeneous chains unless explicitly noted otherwise.

---

2. State Space, Transition Matrices, and Transition Diagrams

The Transition Probability Matrix ($P$)

For a Markov chain with $M$ discrete states $S = {1, 2, \dots, M}$, the transition probabilities $p_{ij}$ are arranged into a square $M \times M$ transition probability matrix $P$:

$P = \begin{bmatrix} p_{11} & p_{12} & \dots & p_{1M} \\ p_{21} & p_{22} & \dots & p_{2M} \\ \vdots & \vdots & \ddots & \vdots \\ p_{M1} & p_{M2} & \dots & p_{MM} \end{bmatrix}$

Where $p_{ij}$ represents the conditional probability of transitioning from current state $i$ to next state $j$ in one step:

$p_{ij} = P(X_{n+1} = j \mid X_n = i)$

Fundamental Properties of Matrix $P$

1. Non-negativity: Every element is a valid probability: $p_{ij} \ge 0$ for all $i, j$.
2. Row-Stochastic Condition: Each row $i$ represents a complete conditional probability distribution over all mutually exclusive and collectively exhaustive destination states. Therefore, each row must sum to exactly 1.0: $\sum_{j=1}^M p_{ij} = 1 \quad \text{for each row } i = 1, 2, \dots, M$ (Note: Columns do not need to sum to 1.0; a matrix where both rows and columns sum to 1.0 is doubly stochastic).

State Transition Diagrams

A Markov chain can be visualized as a directed graph where:

• Nodes represent the distinct physical states ${1, 2, \dots, M}$.
• Directed Arcs from node $i$ to node $j$ represent permissible transitions with non-zero probability $p_{ij} > 0$.
• Self-Loops from node $i$ back to node $i$ represent the probability $p_{ii}$ of remaining in the current state.

                         ┌──────────── p12 ────────────┐
                         │                             ▼
                   ┌───────────┐                 ┌───────────┐
        p11 ↺ ─── │  State 1  │ ◄─────────────── │  State 2  │ ─── ↻ p22
                   └───────────┘       p21       └───────────┘
                         │                             ▲
                         │             p23             │
                         └──────────────┬──────────────┘
                                        ▼
                                  ┌───────────┐
                       p33 ↺ ──── │  State 3  │ ───> p31 (to State 1)
                                  └───────────┘

---

3. Multi-Step Transitions and Chapman-Kolmogorov Equations

To predict the state of the system multiple steps into the future, we evaluate the $n$-step transition probability, denoted $p_{ij}^{(n)}$:

$p_{ij}^{(n)} = P(X_{m+n} = j \mid X_m = i)$

Chapman-Kolmogorov Equations

The probability of transitioning from state $i$ to state $j$ in $n + m$ steps equals the sum of probabilities across all possible intermediate states $k$ occupied after $n$ steps:

$p_{ij}^{(n+m)} = \sum_{k=1}^M p_{ik}^{(n)} p_{kj}^{(m)}$

In matrix notation, this relationship reduces to ordinary matrix multiplication:

$P^{(n+m)} = P^{(n)} P^{(m)}$

By induction, the matrix of $n$-step transition probabilities is simply the transition matrix $P$ raised to the $n$-th power:

$P^{(n)} = P^n = \underbrace{P \times P \times \dots \times P}_{n \text{ times}}$

Unconditional State Probability Propagation

Let $\pi^{(0)} = [\pi_1^{(0)}, \pi_2^{(0)}, \dots, \pi_M^{(0)}]$ be a row vector representing the initial probability distribution over states at time $n = 0$. The probability distribution across states at time step $n$ is given by:

$\pi^{(n)} = \pi^{(0)} P^n$

Where $\pi_j^{(n)} = P(X_n = j)$ is the unconditional probability that the system occupies state $j$ at step $n$.

---

4. Classification of States in Markov Chains

Understanding the structural classification of states is necessary to determine whether a Markov chain reaches a unique steady-state equilibrium.

Exam Key: If any diagonal entry $p_{ii} > 0$ (a state has a self-loop), that state is automatically aperiodic ($d = 1$). Furthermore, if an irreducible Markov chain has at least one aperiodic state, the entire chain is aperiodic.

---

5. Stationary and Steady-State Probability Distributions

In long-run industrial operations, engineers are rarely interested in transient fluctuations from a specific starting state; they require the long-run proportion of time the process spends in each state.

Mathematical Definition of Stationary Distribution

A row vector $\pi = [\pi_1, \pi_2, \dots, \pi_M]$ is called a stationary (steady-state) distribution if it satisfies:

$\pi P = \pi$ $\sum_{i=1}^M \pi_i = 1, \quad \text{with } \pi_i \ge 0 \text{ for all } i$

In expanded linear system form:

$\pi_j = \sum_{i=1}^M \pi_i p_{ij} \quad \text{for } j = 1, 2, \dots, M$ $\pi_1 + \pi_2 + \dots + \pi_M = 1$

Physical and Long-Run Interpretation

For an irreducible, aperiodic, finite Markov chain (an ergodic chain):

$\lim_{n \to \infty} P^n = \begin{bmatrix} \pi_1 & \pi_2 & \dots & \pi_M \\ \pi_1 & \pi_2 & \dots & \pi_M \\ \vdots & \vdots & \ddots & \vdots \\ \pi_1 & \pi_2 & \dots & \pi_M \end{bmatrix}$

Every row of $P^n$ converges to the exact same steady-state vector $\pi$. This implies:

1. $\lim_{n \to \infty} P(X_n = j \mid X_0 = i) = \pi_j$, completely independent of the starting state $i$.
2. $\pi_j$ represents the long-run fraction of time the system spends in state $j$.
3. The mean recurrence time (expected number of steps between consecutive visits to state $j$) is: $\mu_{jj} = \frac{1}{\pi_j}$

Systematic Solution Procedure for the FE Exam

The matrix equation $\pi P = \pi$ can be rewritten as the homogeneous linear system $\pi(P - I) = 0$. Because the rows of $P$ sum to $1.0$, the matrix $(P - I)$ is singular (its determinant is zero), meaning the system $\pi(P - I) = 0$ has infinitely many dependent solutions. To find the unique solution:

1. Write out the $M$ linear equations from $\pi P = \pi$.
2. Discard any one redundant equation from the set.
3. Replace the discarded equation with the normalization equation: $\sum_{i=1}^M \pi_i = 1$.
4. Solve the resulting system of $M$ independent linear equations for $\pi_1, \pi_2, \dots, \pi_M$.

---

6. Industrial Applications of Markov Chains

Machine Reliability and Preventative Maintenance

Equipment condition degrades stochastically over time. A common model discretizes machine health into:

• State 1: Operational / Excellent (new tool, tight tolerances)
• State 2: Degraded / Acceptable (surface finish drifting, minor chatter)
• State 3: Failed / Out of Specification (broken tool, machine halted)

By evaluating the steady-state probability vector $\pi$, reliability engineers compute:

• Long-run machine availability: $A = \pi_1 + \pi_2$
• Long-run machine downtime: $D = \pi_3$
• Average hourly maintenance cost: $\text{Cost} = \sum_{i} \pi_i C_i$

Customer Brand Switching

Marketing and operations teams model consumer brand loyalty using Markov chains. If state $1$ represents purchasing Company A's product and state $2$ represents purchasing Competitor B's product, the steady-state probabilities $\pi_1$ and $\pi_2$ dictate long-run equilibrium market shares.

Inventory Replenishment Policies

In $(s, S)$ inventory control, when inventory drops to reorder point $s$, an order is triggered to restore stock to target $S$. Modeling stock levels as a Markov chain allows exact calculation of stockout probabilities $\pi_0$ and average holding costs.

---

7. Step-by-Step Worked Engineering Examples

Worked Example 9.2.1: Three-State CNC Degradation and Availability Analysis

Problem: An automated CNC mill is inspected at the start of each 8-hour shift and categorized into one of three states:

• State 1: Optimal operating condition
• State 2: Minor tool wear (operates with acceptable tolerances)
• State 3: Failed (shut down; requires emergency maintenance)

The one-shift transition probability matrix is:

$P = \begin{bmatrix} 0.80 & 0.15 & 0.05 \\ 0.00 & 0.70 & 0.30 \\ 0.90 & 0.00 & 0.10 \end{bmatrix}$

1. Verify that $P$ is a valid transition probability matrix.
2. Formulate the steady-state equations and determine the long-run probability distribution $\pi = [\pi_1, \pi_2, \pi_3]$.
3. What is the long-run availability of the CNC mill (percentage of shifts it can operate)?
4. What is the mean time between failures (MTBF in shifts)?

Solution:

1. Verify Matrix Properties:
   • All entries $p_{ij} \ge 0$.
   • Row 1: $0.80 + 0.15 + 0.05 = 1.00$
   • Row 2: $0.00 + 0.70 + 0.30 = 1.00$
   • Row 3: $0.90 + 0.00 + 0.10 = 1.00$
   • All rows sum to $1.00$. $P$ is a valid row-stochastic matrix.
2. Formulate and Solve Steady-State Equations:
   • Set up $\pi P = \pi$: $[\pi_1, \pi_2, \pi_3] \begin{bmatrix} 0.80 & 0.15 & 0.05 \\ 0.00 & 0.70 & 0.30 \\ 0.90 & 0.00 & 0.10 \end{bmatrix} = [\pi_1, \pi_2, \pi_3]$
   • Multiply components: $(1) \quad \pi_1 = 0.80 \pi_1 + 0.90 \pi_3$ $(2) \quad \pi_2 = 0.15 \pi_1 + 0.70 \pi_2$ $(3) \quad \pi_3 = 0.05 \pi_1 + 0.30 \pi_2 + 0.10 \pi_3$ $(4) \quad \pi_1 + \pi_2 + \pi_3 = 1 \quad (\text{Normalization})$
   • Express all variables in terms of $\pi_1$:
     • From equation (2): $\pi_2 - 0.70 \pi_2 = 0.15 \pi_1 \implies 0.30 \pi_2 = 0.15 \pi_1 \implies \pi_2 = \frac{0.15}{0.30} \pi_1 = 0.50 \pi_1$
     • From equation (1): $\pi_1 - 0.80 \pi_1 = 0.90 \pi_3 \implies 0.20 \pi_1 = 0.90 \pi_3 \implies \pi_3 = \frac{0.20}{0.90} \pi_1 = \frac{2}{9} \pi_1$
   • Substitute into the normalization equation (4): $\pi_1 + 0.50 \pi_1 + \frac{2}{9} \pi_1 = 1$ $\pi_1 \left( 1 + \frac{1}{2} + \frac{2}{9} \right) = 1 \implies \pi_1 \left( \frac{18 + 9 + 4}{18} \right) = 1 \implies \pi_1 \left( \frac{31}{18} \right) = 1$ $\pi_1 = \frac{18}{31} \approx 0.5806 \quad (58.06\%)$
   • Compute $\pi_2$ and $\pi_3$: $\pi_2 = 0.50 \left( \frac{18}{31} \right) = \frac{9}{31} \approx 0.2903 \quad (29.03\%)$ $\pi_3 = \frac{2}{9} \left( \frac{18}{31} \right) = \frac{4}{31} \approx 0.1290 \quad (12.90\%)$
   • Verification: $18/31 + 9/31 + 4/31 = 31/31 = 1.0000$.
3. Compute Long-Run Machine Availability:
   • The mill operates when it is in State 1 or State 2: $\text{Availability} = \pi_1 + \pi_2 = \frac{18}{31} + \frac{9}{31} = \frac{27}{31} \approx 0.8710 \quad (87.10\%)$
4. Compute Mean Recurrence Time for Failure (MTBF): $\mu_{33} = \frac{1}{\pi_3} = \frac{1}{4/31} = \frac{31}{4} = 7.75 \text{ shifts}$

Worked Example 9.2.2: Two-Step Transition and State Propagation

Problem: An automated assembly tool exhibits two operational states: State 1 (Calibrated) and State 2 (Out of Calibration). The transition matrix between production batches is:

$P = \begin{bmatrix} 0.90 & 0.10 \\ 0.40 & 0.60 \end{bmatrix}$

1. If the tool is known to be Calibrated at batch 0 ($\pi^{(0)} = [1.0, 0.0]$), find the probability that the tool is Out of Calibration at batch 2 ($p_{12}^{(2)}$).
2. Find the long-run steady-state probability distribution $\pi$.

Solution:

1. Compute Two-Step Transition Matrix $P^2$: $P^2 = P \times P = \begin{bmatrix} 0.90 & 0.10 \\ 0.40 & 0.60 \end{bmatrix} \begin{bmatrix} 0.90 & 0.10 \\ 0.40 & 0.60 \end{bmatrix}$ $p_{11}^{(2)} = (0.90)(0.90) + (0.10)(0.40) = 0.81 + 0.04 = 0.85$ $p_{12}^{(2)} = (0.90)(0.10) + (0.10)(0.60) = 0.09 + 0.06 = 0.15$ $p_{21}^{(2)} = (0.40)(0.90) + (0.60)(0.40) = 0.36 + 0.24 = 0.60$ $p_{22}^{(2)} = (0.40)(0.10) + (0.60)(0.60) = 0.04 + 0.36 = 0.40$ $P^2 = \begin{bmatrix} 0.85 & 0.15 \\ 0.60 & 0.40 \end{bmatrix}$
   • The probability that the tool is in State 2 after 2 batches given it started in State 1 is: $P(X_2 = 2 \mid X_0 = 1) = p_{12}^{(2)} = 0.15 \quad (15.0\%)$
2. Compute Steady-State Probabilities:
   • $\pi P = \pi \implies [\pi_1, \pi_2] \begin{bmatrix} 0.90 & 0.10 \ 0.40 & 0.60 \end{bmatrix} = [\pi_1, \pi_2]$
   • Equation 1: $\pi_1 = 0.90 \pi_1 + 0.40 \pi_2 \implies 0.10 \pi_1 = 0.40 \pi_2 \implies \pi_1 = 4 \pi_2$
   • Normalization: $\pi_1 + \pi_2 = 1 \implies 4 \pi_2 + \pi_2 = 1 \implies 5 \pi_2 = 1$ $\pi_2 = \frac{1}{5} = 0.20 \quad (20.0\%), \quad \pi_1 = 0.80 \quad (80.0\%)$
   • In the long run, the tool is calibrated $80%$ of the time and out of calibration $20%$ of the time.

---

8. NCEES Reference Handbook Tips & Realistic Exam Traps

• Row Vector vs. Column Vector Trap: The steady-state system is $\pi P = \pi$, where $\pi$ is a row vector multiplying $P$ from the left. Do not write $P \pi = \pi$; that formulation treats $\pi$ as a column vector and will multiply columns incorrectly unless you transpose $P$ ($P^T \pi^T = \pi^T$).
• Omitting the Normalization Equation: Attempting to solve $\pi(P - I) = 0$ directly using matrix inversion ($[P - I]^{-1}$) will result in a divide-by-zero error on your calculator because $(P - I)$ is singular. You must manually replace one of the dependent rows with the normalization equation $\sum \pi_i = 1$.
• Row Sum Verification: Before doing any Markov calculations, always sum the elements of each row in the given matrix. If a row does not sum to $1.0$, you may have transcribed a column as a row, or the problem statement defines transitions differently.
• Absorbing State Distractor: A state with $p_{ii} = 1.0$ is absorbing: probability that reaches it never leaves. If a chain has exactly one absorbing state reachable from every transient state, its long-run probability is $1.0$. If the chain has two or more absorbing states (as in the 4-state chain earlier in this section), the long-run distribution splits among them according to the starting state and requires absorption-probability analysis, not the ergodic $\pi P = \pi$ method.