We report a method for in situ atomic-scale observation of electrochemical delithiation in a working all-solid-state battery using a state-of-the-art chip based in situ transmission electron microscopy (TEM) holder and focused ion beam milling to prepare an all-solid-state lithium-ion battery sample. A battery consisting of LiCoO2 cathode, LLZO solid state electrolyte and gold anode was constructed, delithiated and observed in an aberration corrected scanning transmission electron microscope at atomic scale. We found that the pristine single crystal LiCoO2 became nanosized polycrystal connected by coherent twin boundaries and antiphase domain boundaries after high voltage delithiation. This is different from liquid electrolyte batteries, where a series of phase transitions take place at LiCoO2 cathode during delithiation. Both grain boundaries become more energy favorable along with extraction of lithium ions through theoretical calculation. We also proposed a lithium migration pathway before and after polycrystallization. This new methodology could stimulate atomic scale in situ scanning/TEM studies of battery materials and provide important mechanistic insight for designing better all-solid-state battery.