In plants, multiple lineages have evolved sex chromosomes independently, providing a powerful comparative framework, but few specific determinants controlling the expression of a specific sex have been identified. We investigated sex determinants in the Caucasian persimmon, Diospyros lotus, a dioecious plant with heterogametic males (XY). Male-specific short nucleotide sequences were used to define a male-determining region. A combination of transcriptomics and evolutionary approaches detected a Y-specific sex-determinant candidate, OGI, that displays male-specific conservation among Diospyros species. OGI encodes a small RNA targeting the autosomal MeGI gene, a homeodomain transcription factor regulating anther fertility in a dosage-dependent fashion. This identification of a feminizing gene suppressed by a Y-chromosome-encoded small RNA contributes to our understanding of the evolution of sex chromosome systems in higher plants.