With the development of perceptual consumption, consumers sometimes cannot explicitly describe the purchase demands or only based on impression, like the color perception and aesthetic experience. Based on the consumer's expression, it is difficult to design a new fabric by repeated proofing to meet the consumer's demands. To retrieve the existing patterns incorporating human intuition and emotion, this study proposed a novel pattern retrieval method of yarn-dyed plaid fabric using modified interactive genetic algorithm. Each pattern was encoded based on the design elements and visual features were extracted to bridge the semantic gap between the designer and the consumer. Survival of the fittest and two special mutation operators, addition and deletion, were designed to increase the diversity of the generations. During the evolution, the generated patterns were replaced by the most similar patterns in the database based on visual features. Experimental results showed that the proposed scheme is feasible and effective to extract the consumer's preferences and retrieve satisfactory patterns, helping the factory obtain the process sheet to guide production and save labor and material resources.