## Stochastic Processes(IND 621)

Course Code Course Name Semester Theory Practice Lab Credit ECTS
IND 621 Stochastic Processes 1 3 0 0 3 8
 Language of Instruction English Course Type Compulsory Course Level Doctoral Degree Course Instructor(s) EBRU ANGÜN ebru.angun@gmail.com (Email) Assistant Objective Uncertainty exists in most real life problems, and for a better decision making, it is important to take into account the uncertainties. The use of stochastic variables is a common way to represent the uncertainty in quantities of interest such as customer demands, production lead times, product price, repair times, service times etc. that can be encountered in the decision problems in both service sector and industry. The primary aim of this course is to teach how to formulate and solve the stochastic decision problems using stochastic processes. The objectives of this course can be listed as follows.1. To make the students be aware of the stochastic nature of most real life problems.2. To provide students an insight of how to analyze the stochastic systems.3. To make the students develop the necessary skills to identify, formulate and solve the stochastic problems. Content Week 1. Review on probability concepts (Ross, Chapter 1)Week 2. Random variables: discrete and continuous, expectation, variance (Ross, Chapter 2)Week 3. Random variables (cont.): Jointly distributed random variables, variance and covariance of sum of random variables, moment generating functions, limit theorems (Ross, Chapter 2)Week 4. Conditional Probability, Conditional Expectation : Conditional distribution functions, Use of conditioning for calculating probability, expectation and variance (Ross, Chapter 3)Week 5. Markov Chain: Definition of Markov chain, Chapman-Kolmogorov Equations, Calculation of state probabilities (Ross, Chapter 4)Week 6. Markov Chain (Cont.): Classification of states, Limiting state probabilities (Ross, Chapter 4)Week 7. Discrete-Time Markov Process: State probability calculations, Numerical examples (Howard, Chapters 1-2)Week 8. Discrete-Time Markov Process with Reward (Howard, Chapter 3)Week 9. Midterm ExamWeek 10. Discrete-Time Markov Decision Process : Solution algorithms: Value iteration technique for finite horizon, policy iteration algorithm for infinite horizon (Howard, Chapter 4)Week 11. Exponential Distribution : definition, properties and use of exponential distribution (Ross, Chapter 5)Week 12. Poisson Process: definition, properties of Poisson Processes, Non-homogenous and Compound Poisson Processes (Ross, Chapter 5)Week 13. Continuous-Time Markov Chain (Ross, Chapter 6)Week 14. Project presentations on the application of MDP to research problems Course Learning Outcomes ÖÇ 1: be able to define the basic probability concepts.ÖÇ 2: be able to use the conditioning to calculate the probability of a random event or the expectation and variance of stochastic variables.ÖÇ 3: be able to identify the stochastic problems.ÖÇ 4: be able to classify the stochastic problems into appropriate categories such as Poisson or Markov Processes.ÖÇ 5: be able to formulate and solve the stochastic problems as Poisson Processes.ÖÇ 6: be able to analyze Markov chains.ÖÇ 7: be able to formulate the stochastic decision problems as discrete-time Markov decision processes (MDP) ÖÇ 8: be able to use an appropriate algorithm to solve the finite and infinite horizon MDP problemsÖÇ 9: be able to formulate and solve continuous-time Markov Processes.ÖÇ 10: be able to write a report on the analysis of a stochastic problem and do an oral presentation. Teaching and Learning Methods References 1. Ross, S., “Introduction to Probability Models”, 9th edition, Academic Press, Inc.,2007.2. Howard, R.A., “Dynamic Programming and Markov Processes”, MIT Press, 1960.3. Winston, W.L., “Introduction to Probability Models - Operations Research: Volume 2”, Duxbury Resource Center, 2003.
###### Theory Topics
Week Weekly Contents
1 Review on probability concepts (Ross, Chapter 1)
2 Random variables: discrete and continuous, expectation, variance (Ross, Chapter 2)
3 Random variables (cont.): Jointly distributed random variables, variance and covariance of sum of random variables, moment generating functions, limit theorems (Ross, Chapter 2)
4 Conditional Probability, Conditional Expectation : Conditional distribution functions, Use of conditioning for calculating probability, expectation and variance (Ross, Chapter 3)
5 Markov Chain: Definition of Markov chain, Chapman-Kolmogorov Equations, Calculation of state probabilities (Ross, Chapter 4)
6 Markov Chain (Cont.): Classification of states, Limiting state probabilities (Ross, Chapter 4)
7 Discrete-Time Markov Process: State probability calculations, Numerical examples (Howard, Chapters 1-2)
8 Discrete-Time Markov Process with Reward (Howard, Chapter 3)
9 Midterm Exam
10 Discrete-Time Markov Decision Process : Solution algorithms: Value iteration technique for finite horizon, policy iteration algorithm for infinite horizon (Howard, Chapter 4)
11 Exponential Distribution : definition, properties and use of exponential distribution (Ross, Chapter 5)
12 Poisson Process: definition, properties of Poisson Processes, Non-homogenous and Compound Poisson Processes (Ross, Chapter 5)
13 Continuous-Time Markov Chain (Ross, Chapter 6)
14 Project presentations on the application of MDP to research problems
###### Practice Topics
Week Weekly Contents
Number Contribution
Contribution of in-term studies to overall grade 0 60
Contribution of final exam to overall grade 0 40
Toplam 0 100
###### In-Term Studies
Number Contribution
Assignments 4 20
Presentation 0 0
Midterm Examinations (including preparation) 1 30
Project 1 10
Laboratory 0 0
Other Applications 0 0
Quiz 0 0
Term Paper/ Project 0 0
Portfolio Study 0 0
Reports 0 0
Learning Diary 0 0
Thesis/ Project 0 0
Seminar 0 0
Other 0 0
Toplam 6 60
No Program Learning Outcomes Contribution
1 2 3 4 5
1 X
2 X
3 X
4 X
5 X
6
7
8 X
9
10
Class Hours 14 3 42
Working Hours out of Class 13 5 65
Assignments 4 10 40
Presentation 0 0 0
Midterm Examinations (including preparation) 1 10 10
Project 1 40 40
Laboratory 0 0 0
Other Applications 0 0 0
Final Examinations (including preparation) 1 17 17
Quiz 0 0 0
Term Paper/ Project 0 0 0
Portfolio Study 0 0 0
Reports 0 0 0
Learning Diary 0 0 0
Thesis/ Project 0 0 0
Seminar 0 0 0
Other 0 0 0