The effect of a cationic surfactant, dodecyltrimethylammonium bromide (DTAB), on low-rank coal flotation using an oily collector (dodecane) was investigated by a combination of experimental tests and molecular dynamics simulations. Flotation results showed that the addition of DTAB during pulp conditioning increased the clean coal yield, while a high concentration of DTAB exerted a negative influence. To explain the flotation behavior and analyze the interaction between low-rank coal, DTAB and dodecane, a series of experimental tests were conducted and the results indicated that a relatively lower concentration of DTAB is beneficial for the adsorption of dodecane on low-rank coal surface. In addition, by measuring the induction time, it was found that low-rank coal particles were easily adhered to bubbles in the presence of suitable concentration of DTAB. However, a high concentration of DTAB brings about an adverse effect. Later, molecular dynamics (MD) simulations were carried out to illustrate the positive effect of DTAB on the adsorption reaction between low-rank coal and dodecane. The results showed that the abundant number of oxygen-containing groups in low-rank coal are responsible for the limited adsorption of nonpolar dodecane molecules on its surface, while the pre-adsorption of DTAB on the surface of low-rank coal enhances dodecane adsorption. This can be ascribed to the exposed hydrophobic structure of the coal-DTAB complex. As a result, dodecane molecules were more easily adsorbed on the surface of low-rank coal and the attraction of low-rank coal to dodecane in the presence of DTAB resulted in a lower mobility of dodecane molecules, and a larger interaction energy between dodecane and low-rank coal. The simulation results are in good agreement with experimental results.