The lunar surface, a key proxy for the early Earth, contains relics of asteroids and comets that have pummeled terrestrial planetary surfaces. Surviving fragments of projectiles in the lunar regolith provide a direct measure of the types and thus the sources of exogenous material delivered to the Earth-Moon system. In ancient [>3.4 billion years ago (Ga)] regolith breccias from the Apollo 16 landing site, we located mineral and lithologic relics of magnesian chondrules from chondritic impactors. These ancient impactor fragments are not nearly as diverse as those found in younger (3.4 Ga to today) regolith breccias and soils from the Moon or that presently fall as meteorites to Earth. This suggests that primitive chondritic asteroids, originating from a similar source region, were common Earth-Moon-crossing impactors during the latter stages of the basin-forming epoch.