2.5 Markov Chains: Transition Matrices, Steady-State Probabilities & Absorbing States

2.5 Markov Chains

Markov chains model systems that move among a set of states at regular time steps, such as a machine that is good, degraded, or failed each shift. The specification lists Markov chains as a modeling technique, and they also support maintenance and reliability decisions.

---

1. The Markov Property and the Transition Matrix

A discrete-time Markov chain satisfies the Markov property: the probability of the next state depends only on the current state, not on how the system got there.

$P(X_{t+1} = j \mid X_t = i, X_{t-1}, \dots, X_0) = P(X_{t+1} = j \mid X_t = i) = p_{ij}$

The probabilities $p_{ij}$ form the transition matrix $\mathbf{P}$. Row $i$ lists where the system goes from state $i$, so every row sums to 1.

Machine-condition example. A packaging machine is inspected each shift and classified as Good (G), Degraded (D), or Failed (F). A failed machine is repaired during the next shift and returns to Good.

---

2. n-Step Probabilities

If $\boldsymbol{\pi}^{(0)}$ is the row vector of starting probabilities, the distribution after $n$ steps is:

$\boldsymbol{\pi}^{(n)} = \boldsymbol{\pi}^{(n-1)} \mathbf{P} = \boldsymbol{\pi}^{(0)} \mathbf{P}^n$

Starting from a newly repaired machine, $\boldsymbol{\pi}^{(0)} = [1, 0, 0]$:

• After one shift: $\boldsymbol{\pi}^{(1)} = [0.80, 0.15, 0.05]$.
• After two shifts:
  • $P(G) = 0.80(0.80) + 0.15(0) + 0.05(1.00) = 0.69$
  • $P(D) = 0.80(0.15) + 0.15(0.70) + 0.05(0) = 0.225$
  • $P(F) = 0.80(0.05) + 0.15(0.30) + 0.05(0) = 0.085$

The three values sum to 1, which is a quick check on the arithmetic. On the exam, multiply row vector by matrix one step at a time rather than computing $\mathbf{P}^n$.

---

3. Steady-State (Long-Run) Probabilities

For a chain in which every state can reach every other state and the chain is not periodic, the distribution settles to a unique steady state $\boldsymbol{\pi}$ regardless of the starting state:

$\boldsymbol{\pi} = \boldsymbol{\pi}\mathbf{P}, \qquad \sum_i \pi_i = 1$

Write one balance equation per state, drop any one of them (they are redundant), and add the normalization equation.

Machine example:

• From the D column: $\pi_D = 0.15\pi_G + 0.70\pi_D$, so $0.30\pi_D = 0.15\pi_G$ and $\pi_D = 0.5\pi_G$.
• From the F column: $\pi_F = 0.05\pi_G + 0.30\pi_D = 0.05\pi_G + 0.15\pi_G = 0.2\pi_G$.
• Normalize: $\pi_G(1 + 0.5 + 0.2) = 1$, so $\pi_G = 0.588$, $\pi_D = 0.294$, and $\pi_F = 0.118$.

Check with the G column: $0.80(0.588) + 1.00(0.118) = 0.588$. ✓

Using the result.

• The machine is failed about 11.8% of shifts, and the mean recurrence time of the failed state is $1/\pi_F = 8.5$ shifts between failures.
• With costs of $1,000 per degraded shift (scrap) and $5,000 per failed shift (repair and lost output), the long-run expected cost is $0.294(1{,}000) + 0.118(5{,}000) \approx \text{\textdollar}882$ per shift.
• A preventive policy, such as repairing whenever the machine is found degraded, changes the matrix. Compare policies by recomputing $\boldsymbol{\pi}$ and the expected cost.

---

4. Absorbing Markov Chains

A state is absorbing if the chain can never leave it ($p_{ii} = 1$). Arrange the matrix with transient states first:

$\mathbf{P} = \begin{bmatrix} \mathbf{Q} & \mathbf{R} \\ \mathbf{0} & \mathbf{I} \end{bmatrix}$

where $\mathbf{Q}$ holds transitions among transient states and $\mathbf{R}$ holds transitions from transient to absorbing states.

• Fundamental matrix: $\mathbf{N} = (\mathbf{I} - \mathbf{Q})^{-1}$. Entry $n_{ij}$ is the expected number of visits to transient state $j$ when starting in transient state $i$.
• Expected steps before absorption: $\mathbf{t} = \mathbf{N}\mathbf{1}$ (row sums of $\mathbf{N}$).
• Absorption probabilities: $\mathbf{B} = \mathbf{N}\mathbf{R}$.

Rework loop example. Parts leave machining (state P) and go to Good stock 85% of the time, to Rework (state W) 10%, or to Scrap 5%. Reworked parts become Good 70% of the time, are scrapped 20%, and need rework again 10%.

$\mathbf{Q} = \begin{bmatrix} 0 & 0.10 \\ 0 & 0.10 \end{bmatrix}, \quad \mathbf{R} = \begin{bmatrix} 0.85 & 0.05 \\ 0.70 & 0.20 \end{bmatrix}, \quad \mathbf{N} = (\mathbf{I} - \mathbf{Q})^{-1} = \begin{bmatrix} 1 & 0.111 \\ 0 & 1.111 \end{bmatrix}$

For a part starting in machining:

• $P(\text{Good}) = 1(0.85) + 0.111(0.70) = 0.928$
• $P(\text{Scrap}) = 1(0.05) + 0.111(0.20) = 0.072$
• Expected rework operations per part $= n_{PW} = 0.111$

For 10,000 parts started, expect about 9,278 good parts, 722 scrapped, and 1,111 rework operations. That rework load is what the capacity plan must cover.

---

5. Common Applications and Exam Traps

Traps:

• Columns, not rows, are used when writing balance equations for $\pi_j$, because $\pi_j = \sum_i \pi_i p_{ij}$ sums down column $j$.
• Probabilities must be per time step; convert rates to per-shift or per-period probabilities first.
• Steady-state results do not apply to absorbing chains, because those chains end in an absorbing state.