To clarify the higher-order mechanism of human color perception, we measured the color appearances of 78 colored lights by an elemental color-scaling method and by a categorical color naming method. The colors covered nearly the entire CIE 1931 xy-chromaticity diagram with three different surrounds. The results showed that firm basic color Zones derived by categorical color naming can be mapped with no overlap in an opponent-color response space. We propose a network model with a threshold selector, maximum selectors, and multiplication units with gain factors to generate the categorical color responses quantitatively from the elemental color responses. The model can predict the categorical color naming results in different surround conditions with no change of parameters. This suggests that a nonlinear color vision mechanism for color categorization exists between the primary visual cortex (VI) and the inferior temporal cortex (IT) in the human brain.