Biomass-derived xylose is an economically interesting substrate for the sustainable microbial production of value-added compounds. Escherichia coli could barely use xylose to directly produce gamma-aminobutyric acid. In this study, E. coli strains that could directly produce gamma-aminobutyric acid were developed through the deletion of eight genes sucA, puuE, gabT, gabP, xylA, xylB, waaC, and waaF, and the overexpression of two E. coli genes gadB and gdhA, as well as five Caulobacter crescent genes CcxylA, CcxylB, CcxylC, CcxylD, and CcxylX. Both E. coli strains W3110 and JM109 could directly produce gamma-aminobutyric acid from xylose after either overexpression of the seven genes or deletion of the eight genes. Overexpression of the seven genes of in the multiple deletion mutants further increased gamma-aminobutyric acid production. Among the 28 recombinant E. coli strains constructed in this study, the highest gamma-aminobutyric acid was produced by JWZ08/pWZt7-g3/pWZt7-xyl. JWZ08/pWZt7-g3/pWZt7-xyl could produce 3.95 g/L gamma-aminobutyric acid in flask cultivation, using xylose as the sole carbon source.