Accurate chromosome segregation requires the proper assembly of kinetochore proteins. A key step in this process is the recognition of the histone H3 variant CENP-A in the centromeric nucleosome by the kinetochore protein CENP-N. We report cryo-electron microscopy (cryo-EM), biophysical, biochemical, and cell biological studies of the interaction between the CENP-A nucleosome and CENP-N. We show that human CENP-N confers binding specificity through interactions with the L1 loop of CENP-A, stabilized by electrostatic interactions with the nucleosomal DNA. Mutational analyses demonstrate analogous interactions in Xenopus, which are further supported by residue-swapping experiments involving the L1 loop of CENP-A. Our results are consistent with the coevolution of CENP-N and CENP-A and establish the structural basis for recognition of the CENP-A nucleosome to enable kinetochore assembly and centromeric chromatin organization.