Due to the electricity systems' increasing need for flexibility, demand side flexibility aggregation becomes more important. An issue is how to make such activities profitable, which may be obtained by selling flexibility in multiple markets. A challenge is to allocate volumes to the different markets in an optimal way, which motivates the need for advanced decision support models. In this paper, we propose a methodology for optimal bidding for a flexibility aggregator participating in three sequential markets. We demonstrate the approach in a generalized market design that includes an options market for flexibility reservation, a spot market for day-ahead or shorter and a flexibility market for near real-time dispatch. Since the bidding decisions are made sequentially and the price information is gradually revealed, we formulate the decision models as multi-stage stochastic programs and generate scenarios for the possible realizations of prices. We illustrate the application of the models in a realistic case study in cooperation with four industrial companies and one aggregator. We quantify and discuss the value of flexibility and find that our proposed models are able to capture most of the potential value, except for some extreme cases. The value of aggregation is quantified to 3%. (C) 2018 Elsevier Ltd. All rights reserved.