Color sensations are tied not only to other sensations, but also emotions. There have been many studies on this. One study regarding architectural color showed that colors were associated with mental status; for example, red relates to arousal, excitation, and stimulus. The purpose of the present study is to investigate how colors are evoked by emotions. The emotions were described both by emotional words and by schematic faces. Since facial expressions are accompanied by facial color, facial expressions should relate more closely to facial color than emotional words. Therefore, we used numerous color samples for our experiments to show discrimination sensitivity to stimuli in subtle differences of color. Some associations between colors and emotions were found, and the tendencies of associations were different among emotions. Anger, joy, surprise, sadness, and no emotion were connected to particular colors. The distribution of color responses in sadness was spread among bluish colors. The emotional tendencies, among anger, joy, surprise, and sadness, were similar in the two conditions of our experiment. However, in the schematic face condition, the color responses for all emotions were increased in the skin-colored samples. Thus, the context of the face elicited the color responses.