Fluid flow and solute transport in a 3D rough-walled fracture-matrix system were simulated by directly solving the Navier-Stokes equations for fracture flow and solving the transport equation for the whole domain of fracture and matrix with considering matrix diffusion. The rough-walled fracture-matrix model was built from laser-scanned surface tomography of a real rock sample, by considering realistic features of surfaces roughness and asperity contacts. The numerical modeling results were compared with both analytical solutions based on simplified fracture surface geometry and numerical results by particle tracking based on the Reynolds equation. The aim is to investigate impacts of surface roughness on solute transport in natural fracture-matrix systems and to quantify the uncertainties in application of simplified models. The results show that fracture surface roughness significantly increases heterogeneity of velocity field in the rough-walled fractures, which consequently cause complex transport behavior, especially the dispersive distributions of solute concentration in the fracture and complex concentration profiles in the matrix. Such complex transport behaviors caused by surface roughness are important sources of uncertainty that needs to be considered for modeling of solute transport processes in fractured rocks. The presented direct numerical simulations of fluid flow and solute transport serve as efficient numerical experiments that provide reliable results for the analysis of effective transmissivity as well as effective dispersion coefficient in rough-walled fracture-matrix systems. Such analysis is helpful in model verifications, uncertainty quantifications and design of laboratorial experiments.