Markov Decision Processes in Practice 1st Edition by Richard Boucherie, Nico van Dijk ISBN 3319477641 9783319477640

Here’s an overview of the concepts and topics related to Markov Decision Processes (MDPs), Dynamic Programming, and Queueing Systems from your input, structured in a more organized way:

1. Value Function Approximation in Complex Queueing Systems

• Key Focus: Applying value function approximation techniques in queueing systems where traditional methods may not scale efficiently.
• Computational Aspects: These systems often have large state spaces, making it computationally expensive to use exact methods. Approximation techniques can help in reducing computational cost while maintaining reasonable accuracy.

2. Approximate Dynamic Programming by Practical Examples

• Key Focus: Demonstrating how Approximate Dynamic Programming (ADP) methods can be applied to real-world problems.
• Computational Aspects: Practical examples showcase how to balance the accuracy and speed of solving dynamic optimization problems, particularly in high-dimensional spaces.

3. Server Optimization of Infinite Queueing Systems

• Key Focus: Optimizing server performance in systems with infinite queues, often applicable in network traffic or manufacturing systems.
• Computational Aspects: Techniques like queuing theory and ADP are used to find policies that minimize waiting time or maximize throughput.

4. Structures of Optimal Policies in MDPs with Unbounded Jumps: The State of Our Art

• Key Focus: Analyzing and understanding the structure of optimal policies in MDPs where state transitions may involve unbounded jumps, a complex scenario in stochastic decision-making.
• Computational Aspects: The exploration of computationally feasible methods to deal with unbounded jumps and ensure stability and optimality.

5. Markov Decision Processes for Screening and Treatment of Chronic Diseases

• Key Focus: Applying MDPs to healthcare problems, specifically in the screening and treatment of chronic diseases.
• Computational Aspects: This involves modeling patient transitions through different health states and determining optimal treatment policies under uncertainty.

6. Stratified Breast Cancer Follow-Up Using a Partially Observable MDP

• Key Focus: Using a partially observable MDP (POMDP) to model and optimize follow-up decisions for breast cancer patients, accounting for both observable and hidden states.
• Computational Aspects: POMDPs introduce additional computational complexity due to the need to estimate unobservable states, requiring advanced methods like Monte Carlo simulations or particle filtering.

7. Advance Patient Appointment Scheduling

• Key Focus: Optimizing patient appointment scheduling to minimize waiting times and increase the efficiency of healthcare systems.
• Computational Aspects: This involves developing policies using queueing theory and MDPs to dynamically allocate appointment slots based on patient demand patterns.

8. Optimal Ambulance Dispatching

• Key Focus: The use of dynamic programming and MDPs for optimizing ambulance dispatch decisions to reduce response time and improve patient outcomes.
• Computational Aspects: This requires real-time decision-making under uncertainty, considering factors like traffic, ambulance availability, and patient condition.

9. Blood Platelet Inventory Management

• Key Focus: Managing the inventory of blood platelets in hospitals, ensuring that supply meets the demand while minimizing wastage.
• Computational Aspects: MDPs are used to model inventory systems where decisions are made on when to restock and how much inventory to keep.

10. Stochastic Dynamic Programming for Noise Load Management

• Key Focus: Managing and optimizing the noise load in complex systems (e.g., communication networks, manufacturing systems).
• Computational Aspects: Stochastic dynamic programming (SDP) is applied to handle the uncertainty and optimize system performance under varying noise conditions.

11. Allocation in a Vertical Rotary Car Park

• Key Focus: Optimizing car allocation in vertical rotary car parks, which is a classic application of scheduling and queueing theory.
• Computational Aspects: This involves modeling the car park system using discrete event simulation or MDPs to minimize waiting times and optimize car retrieval processes.

12. Dynamic Control of Traffic Lights

• Key Focus: Using dynamic programming and MDPs to control traffic lights and optimize traffic flow in urban environments.
• Computational Aspects: This involves real-time decision-making based on traffic patterns, aiming to minimize congestion and waiting times at intersections.

13. Smart Charging of Electric Vehicles

• Key Focus: Optimizing the charging schedules of electric vehicles (EVs) in smart grids, considering the time-of-use electricity prices and EV battery constraints.
• Computational Aspects: MDPs can help in making real-time decisions about charging to balance grid load and minimize costs.

14. Analysis of a Stochastic Lot Scheduling Problem with Strict Due-Dates

• Key Focus: Optimizing lot scheduling in manufacturing systems with strict due dates, aiming to minimize tardiness or production delays.
• Computational Aspects: Stochastic dynamic programming or simulation models are often used to account for variability in production times.

15. Optimal Fishery Policies

• Key Focus: Developing optimal harvesting policies for fisheries using MDPs to ensure sustainable practices while maximizing long-term profit.
• Computational Aspects: Stochastic models are used to account for fish population dynamics, environmental changes, and policy constraints.

16. Near-Optimal Switching Strategies for a Tandem Queue

• Key Focus: Developing switching strategies for optimizing the flow of items in a tandem queue system (a series of queues where items pass from one queue to another).
• Computational Aspects: Methods such as Approximate Dynamic Programming or other heuristic approaches can be used to find near-optimal policies in systems with long queues.

17. Wireless Channel Selection with Restless Bandits

• Key Focus: Solving the problem of channel selection in wireless networks under the restless bandit framework, where multiple actions are possible, and the system states evolve dynamically.
• Computational Aspects: Restless bandit problems are a type of MDP where actions influence future states in a complex manner, requiring sophisticated algorithms like Q-learning or other reinforcement learning methods.

18. Flexible Staffing for Call Centers with Non-Stationary Arrival Rates

• Key Focus: Optimizing staffing levels in call centers where call arrival rates vary over time.
• Computational Aspects: MDPs can be used to model the call center environment and determine optimal staffing policies in real-time, adjusting for fluctuations in call volume.

19. MDP for Query-Based Wireless Sensor Networks

• Key Focus: Using MDPs to manage wireless sensor networks, where decisions must be made on where and when to query sensors based on available data and battery life.
• Computational Aspects: This involves optimizing data collection and sensor querying while considering energy consumption and network constraints.

20. Optimal Portfolios and Pricing of Financial Derivatives Under Proportional Transaction Costs

• Key Focus: Applying MDPs to portfolio optimization and pricing of financial derivatives, where transaction costs affect decision-making.
• Computational Aspects: Proportional transaction costs complicate the standard financial models, requiring specialized algorithms like Stochastic Dynamic Programming or Reinforcement Learning to find optimal strategies.