Discrete Time Markov Chains, Limiting Distribution and Classification
Discrete Time Markov Chains, Limiting Distribution and Classification
Discrete Time Markov Chains, Limiting Distribution and Classification
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
State classification by f (n)ii<strong>Classification</strong> of <strong>Markov</strong> chains◮ A state is recurrent (persistent) if f ii(= ∑ )∞n=1 f (n)ii= 1◮ A state is positive or non-null recurrent if E(Ti ) < ∞.E(T i ) = ∑ ∞ (n)n=1nfii= µ i◮ A state is null recurrent if E(Ti ) = µ i = ∞◮ A state is transient if f ii < 1.In this case we define µ i = ∞ for later convenience.◮ A peridoic state has nonzero p ii (nk) for some k.◮ A state is ergodic if it is positive recurrent <strong>and</strong> aperiodic.◮ We can identify subclasses of states with the sameproperties◮ All states which can mutually reach each other will be ofthe same type◮ Once again the formal analysis is a little bit heavy, but try tostick to the fundamentals, definitions (concepts) <strong>and</strong> resultsBo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Properties of sets of intercommunicating states◮ (a) i <strong>and</strong> j has the same period◮ (b) i is transient if <strong>and</strong> only if j is transient◮ (c) i is null persistent (null recurrent) if <strong>and</strong> only if j is nullpersistentA set C of states is called◮ (a) Closed if p ij = 0 for all i ∈ C, j /∈ C◮ (b) Irreducible if i ↔ j for all i, j ∈ C.TheoremDecomposition Theorem The state space S can be partitioneduniquely asS = T ∪ C 1 ∪ C 2 ∪ . . .where T is the set of transient states, <strong>and</strong> the C i are irreducibleclosed sets of persistent states□LemmaIf S is finite, then at least one state is persistent(recurrent) <strong>and</strong>all persistent states are non-null (positive recurrent) □Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>
The state probabilities<strong>Limiting</strong> distributionFor an irreducible aperiodic chain, we have that1p (n)ij→ 1 µ jas n → ∞, for all i <strong>and</strong> j0.90.80.70.60.50.40.30.20.100 10 20 30 40 50 60 70Three important remarks◮ If the chain is transient or null-persistent (null-recurrent)p (n)ij→ 0◮ If the chain is positive recurrent p (n)ij→ 1 µ j◮ The limiting probability of X n = j does not depend on thestarting state X 0 = ip (n)jBo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>The stationary distributionStationary distribution◮ A distribution that does not change with n◮ The elements of p (n) are all constant◮ The implication of this is p (n) = p (n−1) P = p (n−1) by ourassumption of p (n) being constant◮ Expressed differently π = πPDefinitionThe vector π is called a stationary distribution of the chain if πhas entries (π j : j ∈ S) such that◮ (a) π j ≥ 0 for all j, <strong>and</strong> ∑ j π j = 1◮ (b) π = πP, which is to say that π j = ∑ i π ip ij for all j.□VERY IMPORTANTAn irreducible chain has a stationary distribution π if <strong>and</strong> only ifall the states are non-null persistent (positive recurrent);in thiscase, π is the unique stationary distribution <strong>and</strong> is given byπ i = 1 µ ifor each i ∈ S, where µ i is the mean recurrence time ofi.Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>
The solution of π = πP continued◮ The vector π is a left eigenvector of P.◮ The main theorem says that there is a unique eigenvectorassociated with the eigenvalue 1 of P◮ In practice this means that the we can only solve but anormalising condition◮ But we have the normalising condition by ∑ j π j = 1 thiscan expressed as π1 = 1. Where⎛ ⎞111 = ⎜ ⎟⎝ . ⎠1Roles of the solution to π = πPFor an irreducible <strong>Markov</strong> chain, (the condition we need toverify)◮ The stationary solution. If p (0) = π then p (n) = π for all n.◮ The limiting distribution, i.e. p (n) → π for n → ∞ (the<strong>Markov</strong> chain has to be aperiodic too). Also p (n)ij→ π j .◮ The mean recurrence time for state i is µ i = 1 π i.◮ The mean number of visits in state j between twosuccessive visits to state i is π jπ i.◮ The long run average probability of finding the <strong>Markov</strong>1∑chain in state i is π i . π i = lim nn→∞ n k=1 p(k) ialso true forperiodic chains.Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Example (null-recurrent) chain⎡P =⎢⎣p 1 p 2 p 3 p 4 p 5 . . .1 0 0 0 0 . . .0 1 0 0 0 . . .0 0 1 0 0 . . .0 0 0 1 0 . . .0 0 0 0 1 . . .. . . . . . . . . . . . . . . . . .For p j > 0 the chain is obviously irreducible.The main theorem tells us that we can investigate directly forπ = πP.π 1 = π 1 p 1 + π 2 π 2 = π 1 p 2 + π 3 π j = π 1 p j + π j+1⎤⎥⎦π 1 = π 1 p 1 + π 2 π 2 = π 1 p 2 + π 3 π j = π 1 p j + π j+1we getπ 2 = (1−p 1 )π 1 π 3 = (1−p 1 −p 2 )π 1 π j = (1−p 1 · · ·−p j−1 )π 1⎛ ⎞∑j−1∑∞π j = (1 − p 1 · · · − p j−1 )π 1 ⇔ π j = π 1⎝1 − p i⎠ ⇔ π j = π 1i=1i=jNormalisation∞∑∞∑ ∑∞ ∑∞ i∑ ∑∞π j = 1 π 1 p i = π 1 p i = π 1 ip ij=1j=1 i=ji=1 j=1i=1p iBo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>
Reversible <strong>Markov</strong> chainsp 11p121 2p21p22P =[ ]p11 p 12p 21 p 22◮ Solve sequence of linear equations instead of the wholesystem◮ Local balance in probability flow as opposed to globalbalance◮ Nice theoretical constructionBo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Local balance equationsπ i = ∑ jπ j p jiTerm for term we getπ i · 1 = ∑ jπ j p ji∑π i p ij = ∑jjπ i p ij = π j p jiπ j p ji∑π i p ij = ∑jjπ j p jiIf they are fulfilled for each i <strong>and</strong> j, the global balance equationscan be obtained by summation.Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>
Why reversible?Another look at a similar questionP{X n−1 = i ∩ X n = j} = P{X n−1 = i}P{X n = j|X n−1 = i}<strong>and</strong> for a stationary chain= P{X n−1 = i}p ijπ i p ijFor a reversible chain (local balance) this is π i p ij = π j p ji =P{X n−1 = j}P{X n = i|X n−1 = j} = P{X n−1 = j ∩ X n = i} thereversed sequence.P{X n−1 = j|X n = i} = P{X n−1 = j ∩ X n = i}P{X n = i}= P{X n−1 = j}P{X n = i|X n−1 = j}P{X n = i}For a stationary chain we getπ j p jiπ i= P{X n−1 = j}p jiP{X n = i}The chain is reversible if P{X n−1 = j|X n = i} = p ij leading tothe local balance equationsp ij = π jp jiπ iBo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Exercise 10 (16/12/91 ex.1)In connection with an examination of the reliability of somesoftware intended for use in control of modern ferries one isinterested in examining a stochastic model of the use of acontrol program.The control program works as " state machine " i.e. it can be ina number of different levels, 4 are considered here. The levelsdepend on the physical state of the ferry. With every shift oftime unit while the program is run, the program will change fromlevel j to level k with probability p jk .Two possibilities are considered:The program has no errors <strong>and</strong> will run continously shifting between the fourlevels.The program has a critical error. In this case it is possible that the error isfound, this happens with probality q i , i = 1, 2, 3, 4 depending on the level.The error will be corrected immediately <strong>and</strong> the program will from then on bewithout faults.Alternatively the program can stop with a critical error (the ferry will continueto sail, but without control). This happens with probability r i , i = 1, 2, 3, 4.In general q i + r i < 1, a program with errors can thus work <strong>and</strong> the error isnot nescesarily discovered. It is assumed that detection of an error, as wellas the apperance of a fault happens coincidently with shift between levels.The program starts running in level 1, <strong>and</strong> it is known that the programcontains one critical error.Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>
Solution: Question 1Question 1 - continuedFormulate a stochastic process ( <strong>Markov</strong> chain) in discretetime describing this system.The model is a discrete time <strong>Markov</strong> chain. A possibledefinition of states could be0: The programme has stopped.1-4: The programme is operating safely in level i.5-8: The programme is operating in level i-4, the criticalerror is not detected.The transition matrix A is⎡A = ⎣1 ⃗ 0 ⃗ 0⃗ 0 P 0⃗r Diag(q i )P Diag(S i )P⎤⎦⃗r =The model is a discrete time <strong>Markov</strong> chain. Where P = {p ij }⎡⎢⎣⎤r 1r 2r 3r 4⎥⎦ Diag(S i ) =⎡⎢⎣Diag(q i ) =⎤S 1 0 0 00 S 2 0 0⎥0 0 S 3 0 ⎦0 0 0 S 4⎡⎢⎣⎤q 1 0 0 00 q 2 0 0⎥0 0 q 3 0 ⎦0 0 0 q 4S i = 1 − r i − q iBo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Question 1 - continuedSolution question 2⎡A =⎢⎣Or without matrix notation:⎤1 0 0 0 0 0 0 0 00 p 11 p 12 p 13 p 14 0 0 0 00 p 21 p 22 p 23 p 24 0 0 0 00 p 31 p 32 p 33 p 34 0 0 0 00 p 41 p 42 p 43 p 44 0 0 0 0r 1 q 1 p 11 q 1 p 12 q 1 p 13 q 1 p 14 S 1 p 11 S 1 p 12 S 1 p 13 S 1 p 14r 2 q 2 p 21 q 2 p 22 q 2 p 23 q 2 p 24 S 2 p 21 S 2 p 22 S 2 p 23 S 2 p 24 ⎥r 3 q 3 p 31 q 3 p 32 q 3 p 33 q 3 p 34 S 3 p 31 S 3 p 32 S 3 p 33 S 3 p ⎦ 34r 4 q 4 p 41 q 4 p 42 q 4 p 43 q 4 p 44 S 4 p 41 S 4 p 42 S 4 p 43 S 4 p 44Characterise the states in the <strong>Markov</strong> chain.With reasonable assumptions on P (i.e. irreducible) we getState 0 Absorbing1 Positive recurrent2 Positive recurrent3 Positive recurrent4 Positive recurrent5 Transient6 Transient7 Transient8 TransientBo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>
Solution question 3We now consider the case where the stability of the system hasbeen assured, i.e. the error has been found <strong>and</strong> corrected, <strong>and</strong>the program has been running for long time without errors. Theparameters are as follows.P i,i+1 = 0.6 i = 1, 2, 3 P i,i−1 = 0.2 i = 2, 3, 4P i,j = 0 |i − j| > 1 q i = 10 −3i r i = 10 −3i−5Characterise the stochastic proces, that describes thestable system.The system becomes stable by reaching one of the states 1-4.The process is ergodic from then on. The procces is areversible ergodic <strong>Markov</strong> chain in discrete time.Solution question 4For what fraction of time will the system be in level 1.We obtain the following steady state equationsπ i = 3 i−1 π 14∑3 i−1 π 1 = 1 ⇔ 40π 1 = 1i=1π 1 = 140The sum ∑ 4i=1 3i−1 can be obtained by using∑ 4i=1 3i−1 = 1−341−3 = 40.4∑i=13 i−1 π 1 = 1 ⇔ 1 − 341 − 3 π 1 = 1Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>Bo Friis Nielsen<strong>Limiting</strong> <strong>Distribution</strong> <strong>and</strong> <strong>Classification</strong>