Over the last three decades, the electricity sector worldwide underwent massive deregulation. Power market participants have encountered a growing number of challenges due to competition and other pertinent factors. For most markets at present, electricity is a non-storable commodity and its price is extremely sensitive to changes in supply and demand. The evolution of electricity prices exhibits pronounced mean reversion and cyclical patterns, possesses extreme volatility and relatively frequently occurring spikes, and manifests presence of memory property. These observed features necessitate the development of models aimed to simultaneously capture such price characteristics for forecasting, risk management, and valuation of electricity-driven derivatives. Majority of current research studies examining electricity spot prices are based on one-state models. Such models though may not accurately capture the stylised behaviours of price evolution, especially during periods of sudden spikes driven by the abrupt changes of market sentiments. This paper tackles the modelling and estimation problems under a new paradigm that integrates the deterministic calendar seasons and stochastic factors governing electricity prices with a regime-switching feature. The de-seasonalised component of our proposed model has both the jump and mean-reverting properties to account for spikes and periodic cycles alternating between lower price returns and compensating periods of higher price returns. The parameters of the de-seasonalised model components are also modulated by a higher-order hidden Markov chain (HOHMC) in discrete time. This provides a mechanism to extract latent information from historical data. The HOHMC's state is interpreted as the "state of the world" resulting from the interaction of various forces impacting the electricity market. Filters are developed to generate optimal estimates of HOHMC-relevant quantities using the observation process, and these provide online estimates of model parameters. Empirical demonstrations using the daily electricity spot prices, compiled by the Alberta Electric System Operator (AESO), show that our HOHMM approach has considerable merits in terms of price data fitting and forecasting metrics. Our proposed framework driven by a hidden Markov chain with a state memory-capturing mechanism provides a powerful alternative to support the pricing of contracts in the electricity markets.