Imperfect and incomplete understanding of reaction kinetics compounded with uncontrollable variations not only prevent achieving an optimal operation of batch and semi-batch reactors, but also give rise to potential risks of violating product end-use properties, ecological or safety constraints. This paper proposes a sequential experiment design strategy based on reinforcement learning to accomplish the specific goal of modeling for optimization in batch reactors by making the most effective use of cumulative data and an approximate model. Reactor operating condition is incrementally improved over runs by integrating together estimation of a probabilistic measure of success using an imperfect model and a gradient-based approach so as to trade off exploitation with exploration. An improved operating policy is found by incrementally shrinking the region of interest for policy parameters. The solution strategy focuses on 'learning by doing' using a value function that accounts for endpoint performance and feasibility. Simulation results reveal the robustness of reinforcement learning to parametric and structural modeling errors.