Artists usually carefully select different colors in artistic work so as to convey special visual and emotional feelings. Color theme extraction techniques can help users to acquire the color styles in an image. However, current color theme extraction methods ignore the emotional factors, and they can only provide a single theme result for an image as well, neither of which meet people's favor on different colors under different mood states. This article introduces the conception of emotional color theme, introduces the color emotion theory into color theme extraction, and proposes a novel emotion color theme extraction framework. To achieve these goals, we perform the theme extraction with emotion value of each pixel instead of color value. The emotional discrepancy is proposed between the colors in the theme to evaluate a color theme quality and prove it with color theme dataset. Then a data driven approach is adopted to optimize the color theme. Different from previous optimizing strategies, we built the emotional relation between the target theme and the candidate theme. Our method can enhance the color editing results of previous methods, e.g., for color transfer our results can demonstrate hierarchical emotions from a single reference image. (C) 2015 Wiley Periodicals, Inc. Col Res Appl, 41, 513-522, 2016; Published Online 24 September 2015 in Wiley Online Library (wileyonlinelibrary.com).