Finite state Markov Decision Processes (MDP) for process control are considered. MDP provide robust tools to perform optimization in closed-loop, and their finite state description enables an easy implementation of Bayesian state estimation. An approach to tackle the curse of dimensionality problem, yet retaining the benefits of the finite state MDP in control and estimator design, is proposed. The suggested approach uses iterative re-discretization based on clustering of closed-loop data. An efficient modification of the k-means clustering technique is proposed. The performance of the approach is demonstrated using a challenging benchmark from chemical engineering, the van der Vusse continuous stirred tank reactor control problem. It is shown that the requirements of the benchmark are met, and that the suggested iterated clustering significantly improves the performance. It is concluded that the finite state MDP approach is a viable alternative for small-to-medium scale problems of practical process control and state estimation. (C) 2016 Elsevier Ltd. All rights reserved.