Stochastic Processes

Content

Objective

Uncertainty exists in most real life problems, and for a better decision making, it is important to take into account the uncertainties. The use of stochastic variables is a common way to represent the uncertainty in quantities of interest such as customer demands, production lead times, product price, repair times, service times etc. that can be encountered in the decision problems in both service sector and industry. The primary aim of this course is to teach how to formulate and solve the stochastic decision problems using stochastic processes. The objectives of this course can be listed as follows.
1. To make the students be aware of the stochastic nature of most real life problems.
2. To provide students an insight of how to analyze the stochastic systems.
3. To make the students develop the necessary skills to identify, formulate and solve the stochastic problems.

Course Content

Week 1. Review on probability concepts (Ross, Chapter 1)
Week 2. Random variables: discrete and continuous, expectation, variance (Ross, Chapter 2)
Week 3. Random variables (cont.): Jointly distributed random variables, variance and covariance of sum of random variables, moment generating functions, limit theorems (Ross, Chapter 2)
Week 4. Conditional Probability, Conditional Expectation : Conditional distribution functions, Use of conditioning for calculating probability, expectation and variance (Ross, Chapter 3)
Week 5. Markov Chain: Definition of Markov chain, Chapman-Kolmogorov Equations, Calculation of state probabilities (Ross, Chapter 4)
Week 6. Markov Chain (Cont.): Classification of states, Limiting state probabilities (Ross, Chapter 4)
Week 7. Discrete-Time Markov Process: State probability calculations, Numerical examples (Howard, Chapters 1-2)
Week 8. Discrete-Time Markov Process with Reward (Howard, Chapter 3)
Week 9. Midterm Exam
Week 10. Discrete-Time Markov Decision Process : Solution algorithms: Value iteration technique for finite horizon, policy iteration algorithm for infinite horizon (Howard, Chapter 4)
Week 11. Exponential Distribution : definition, properties and use of exponential distribution (Ross, Chapter 5)
Week 12. Poisson Process: definition, properties of Poisson Processes, Non-homogenous and Compound Poisson Processes (Ross, Chapter 5)
Week 13. Continuous-Time Markov Chain (Ross, Chapter 6)
Week 14. Project presentations on the application of MDP to research problems

Course Learning Outcomes

ÖÇ 1: be able to define the basic probability concepts.
ÖÇ 2: be able to use the conditioning to calculate the probability of a random event or the expectation and variance of stochastic variables.
ÖÇ 3: be able to identify the stochastic problems.
ÖÇ 4: be able to classify the stochastic problems into appropriate categories such as Poisson or Markov Processes.
ÖÇ 5: be able to formulate and solve the stochastic problems as Poisson Processes.
ÖÇ 6: be able to analyze Markov chains.
ÖÇ 7: be able to formulate the stochastic decision problems as discrete-time Markov decision processes (MDP)
ÖÇ 8: be able to use an appropriate algorithm to solve the finite and infinite horizon MDP problems
ÖÇ 9: be able to formulate and solve continuous-time Markov Processes.
ÖÇ 10: be able to write a report on the analysis of a stochastic problem and do an oral presentation.

References

1. Ross, S., “Introduction to Probability Models”, 9th edition, Academic Press, Inc.,2007.
2. Howard, R.A., “Dynamic Programming and Markov Processes”, MIT Press, 1960.
3. Winston, W.L., “Introduction to Probability Models - Operations Research: Volume 2”, Duxbury Resource Center, 2003.

Weekly Contents

Assessment System

Relation of Proficiency

ECTS