This work presents an observed retention (R(obs)) model for xylo-oligosaccharides (XOs) syrup in nanofiltration (NF) process. A commercial-purchased HDS-12-2540 NF membrane was operated in batch recycling model. The volume flux (J(v)), trans-membrane pressure (Delta P), solute concentrations in feed solution (C(b)) and permeate solution (C(p)) were collected and the experimental R(obs) was obtained. On the basis of the extended Nernst-Planck equation and film theory, a theoretical R(obs) model was established. The model parameters including solute permeability (P(s)) and reflection coefficient (sigma) for arabinose, xylose and xylobiose were estimated by curving fitting using genetic algorithm (GA) method. It was found that the reflection coefficient (sigma) for arabinose (0.895) was higher than that for xylose (0.776), which indicated that the membrane retention to arabinose was stronger. Membrane structural parameters, such as ratio of solute radius to pore radius (lambda) and ratio of effective membrane porosity to membrane thickness (A(k)/Delta x), were also estimated using steric-hindrance pore (SHP) model. (C) 2010 Elsevier Ltd. All rights reserved.