Abstract
Memoryless and finite-memory policies offer a practical alternative for solving partially observable Markov decision processes (POMDPs), as they operate directly in the output space rather than in the high-dimensional belief space. However, extending classical methods such as policy iteration to this setting remains difficult; the output process is non-Markovian, making policy-improvement steps interdependent across stages. We introduce a new family of monotonically improving policy-iteration algorithms that alternate between single-stage output-based policy improvements and policy evaluations according to a prescribed periodic pattern. We show that this family admits optimal patterns that maximize a natural computational-efficiency index, and we identify the simplest pattern with minimal period. Building on this structure, we further develop a model-free variant that estimates values from data and learns memoryless policies directly. We discuss the applicability of the proposed framework to the optimal static output-feedback problem. In particular, when specialized to the linear–quadratic–Gaussian (LQG) setting, the method yields two coupled but dual Riccati equations: one governing the state covariance and the other characterizing the value function. We analyze their convergence properties.
Across several POMDPs examples, our method achieves significant computational speedups over policy-gradient baselines and recent specialized algorithms in both model-based and model-free settings.
Biographical Information
Duarte J. Antunes received the Ph.D. degree (cum laude) in Automatic Control from the Institute for Systems and Robotics, Instituto Superior Técnico, Lisbon, in 2011, in collaboration with the University of California, Santa Barbara, USA. From 2011 to 2013, he was a Postdoctoral Researcher at Eindhoven University of Technology (TU/e), where he became an Assistant Professor in 2013. He is currently an Associate Professor in the Department of Mechanical Engineering at TU/e. His research interests include networked control systems, stochastic control, approximate dynamic programming, and robotics.