MARKOV CHAINS
MARKOV CHAINS Andrey Markov 1856-1922
Markov Chains General Description We want to describe the behavior of a system as it moves (makes transiaons) probabilisacally from state to state. Basic or Markov AssumpAon The future depends only on the present (current state) and not on the past. That is, the future depends on the state we are in, not on how we arrived at this state.
Example 1 - Brand loyalty or Market Share For ease, assume that all cola buyers purchase either Coke or Pepsi in any given week. That is, there is a duopoly. Assume that if a customer purchases Coke in one week there is a 90% chance that the customer will purchase Coke the next week (and a 10% chance that the customer will purchase Pepsi). Similarly, 80% of Pepsi drinkers will repeat the purchase from week to week.
Example 1 - Developing the Markov Matrix States State 1 - Coke was purchased State 2 - Pepsi was purchased (note: states are qualitaave) Markov (transiaon or probability) Matrix From\To Coke Pepsi Coke 0.9 0.1 Pepsi 0.2 0.8
Graphical DescripAon 1 The States From\To Coke Coke.9.1 Pepsi.2.8 Pepsi
Graphical DescripAon 2 TransiAons from Coke.9.1 From\To Coke Coke.9.1 Pepsi.2.8 Pepsi
Graphical DescripAon 3 All transiaons.9.8.1.2 From\To Coke Coke.9.1 Pepsi.2.8 Pepsi
Example 1 - StarAng CondiAons Percentages IdenAfy probability of (percentage of shoppers) starang in either state (We will assume a 50/50 starang market share in our example that follows.) Assume we start in one specific state (by se`ng one probability to 1 and the remaining probabiliaes to 0) Counts (numbers) IdenAfy number of shoppers starang in either state
Example 1 From\To Coke Pepsi Coke 0.9 0.1 Pepsi 0.2 0.8 StarAng ProbabiliAes = 50% (or 50 people) each QuesAons What will happen in the short run (next 3 periods)? What will happen in the long run? Do starang probabiliaes influence long run?
Graphical Solution After 1 Transition.9(50)=45.8(50)=40.1(50)=5 (50)Coke(55) (50)Pepsi(45).2(50)=10 From\To Coke Coke.9.1 Pepsi.2.8 Pepsi
Graphical Solution After 2 Transitions.9(55)=49.5.8(45)=36.1(55)=5.5 (55)Coke(58.5) (45)Pepsi(41.5).2(45)=9 From\To Coke Coke.9.1 Pepsi.2.8 Pepsi
Graphical Solution After 3 Transitions.9(58.5)=52.65.8(41.5)=33.2.1(58.5)=5.85 (58.5)Coke(60.95) (41.5)Pepsi(39.05).2(41.5)=8.3
Analyzing Markov Chains Using QM for Windows Module Markov Chains Number of states 2 Number of transiaons - 3
Example 1 Ager 3 transiaons n- step TransiAon probabiliaes End of Period 1 Coke Pepsi Coke 0.8999 0.1000 Pepsi 0.2000 0.8000 End prob (given iniaal) 0.5500 0.4500 End of Period 2 Coke Pepsi Coke 0.8299 0.1700 Pepsi 0.3400 0.6600 End prob (given iniaal) 0.5849 0.4150 End of Period 3 Coke Pepsi Coke 0.7809 0.2190 Pepsi 0.4380 0.5620 End prob (given iniaal) 0.6094 0.3905 1 step transition matrix 2 step transition matrix 3 step transition matrix
Example 1 - Results (3 transitions, start =.5,.5) From\To Coke Pepsi Coke 0.78100 0.21900 Pepsi 0.43800 0.56200 Ending probability 0.6095 0.3905 Steady State probability 0.6666 0.3333 Note: We end up alternaang between Coke and Pepsi 3 step transition matrix Depends on initial conditions Independent of initial conditions
Example 2 - Student Progression Through a University States Freshman Sophomore Junior Senior Dropout Graduate (note: again, states are qualitaave)
Example 2 - Student Progression Through a University - States Freshman Sophomore Junior Senior Drop out Graduate Note that eventually you must end up in Grad or Drop-out.
Classification of states Absorbing Those states such that once you are in you never leave. Graduate, Drop Out Recurrent Those states to which you will always both leave and return at some time. Coke, Pepsi Transient States that you will eventually never return to Freshman, Sophomore, Junior, Senior
Example 3 - Diseases States no disease pre-clinical (no symptoms) clinical death (note: again states are qualitative) Purpose Transition probabilities can be different for different testing or treatment protocols
Example 4 - Customer Bill paying States State 0: Bill is paid in full State i: Bill is in arrears for i months, i= 1,2,,11 State 12: Deadbeat
Example 5 HIV infecaons Based on Can Difficult- to- Reuse Syringes Reduce the Spread of HIV among InjecAon Drug Users Caulkins, et. al. Interfaces, Vol 28, No. 3, May- June 1998, pp 23-33 State State 0 Syringe is uninfected State 1 Syringe is infected Notes: P(0, 1) =.14 14% of drug users are infected with HIV P(1, 0) =.33+.05 5% of the Ame the virus dies; 33% of the Ame it is killed by bleaching
Example 6 - Baseball States State 0 - no outs, bases empty State 1 - no outs, runner on first State 2 - no outs, runner on second State 3 - no outs, runner on third State 4 - no outs, runners on first, second State 5 - no outs, runners on first, third State 6 - no outs, runners on second, third State 7 - no outs, runners on first, second, third. Repeat for 1 out and 2 outs for a total of 24 states Moneyball by Michael Lewis, p 134