The correlation between chirality of the B2 and B4 phases was examined by measuring the circular dichroism (CD) in the classic bent-core molecule, P-12-O-PIMB, which shows a direct B2 to B4 phase transition on cooling. Among the various structures formed in the tilted B2 phase, the homochiral SmCSPF and SmCAPA layer structures show a distinct CD peak at around 400 nm, while no CD effect is observed in the racemic SmCSPA layer structure. In contrast, the lower temperature smectic B4 phase is essentially chiral and it appears to be independent of the type of layer structures of the initial B2 phase. It always exhibits a CD peak at a similar wavelength to that of the homochiral B2 phase, but with opposite sign. By incorporating the optical microscopic observation with the CD measurements, we found that once the chiral domains are formed in the B4 phase, they are conserved thereafter on successive transformation between the B2 and B4 phases. The resulting chiral structure in the B2 phase is homochiral SmCAPA. For this conservation of chirality, there must be an inherent chiral source for both the B2 and B4 phases. We propose that conformational chirality fills this role.