Using two-step plasmid integration in the presence of 5-fluorouracil (5-FU), we developed a stable and markerless Lactobacillus casei strain for vaccine antigen expression. The upp of L. casei, which encodes uracil phosphoribosyltransferase (UPRTase), was used as a counterselection marker. We employed the Delta upp isogenic mutant, which is resistant to 5-FU, as host and a temperature-sensitive suicide plasmid bearing upp expression cassette as counterselectable integration vector. Extrachromosomal expression of UPRTase complemented the mutated chromosomal upp allele and restored sensitivity to 5-FU. The resultant genotype can either be wild type or recombinant. The efficacy of the system was demonstrated by insertion and expression of porcine rotavirus (PRV) VP4. To improve VP4 expression, we analyzed L. casei transcriptional profiles and selected the constitutive highly expressed enolase gene (eno). The VP4 inserted after the eno termination codon were screened in the presence of 5-FU. Using genomic PCR amplification, we confirmed that VP4 was successfully integrated and stably inherited for at least 50 generations. Western blot demonstrated that VP4 was steadily expressed in medium with different carbohydrates. RT-qPCR and ELISA analysis showed that VP4 expression from the chromosomal location was similar to that achieved by a plasmid expression system. Applying the recombinant strain to immunize BALB/c mice via oral administration revealed that the VP4-expressing L. casei could induce both specific local and systemic humoral immune responses in mice. Overall, the improved gene replacement system represents an efficient method for chromosome recombination in L. casei and provides a safe tool for vaccine production.