Cholesterol plays an important role in maintaining the correct fluidity and rigidity of the plasma membrane of all animal cells, and hence, it is present in concentrations ranging from 20 to 50 mol %. Whereas the effect of cholesterol on such mechanical properties has been studied exhaustively over the last decades, the structural basis for cholesterol effects on membrane permeability is still unclear. Here we apply systematic molecular dynamics simulations to study the partitioning of solutes between water and membranes. We derive potentials of mean force for six different solutes permeating across 20 different lipid membranes containing one out of four types of phospholipids plus a cholesterol content varying from 0 to 50 mol %. Surprisingly, cholesterol decreases solute partitioning into the lipid tail region of the membranes much more strongly than expected from experiments on macroscopic membranes, suggesting that a laterally inhomogeneous cholesterol concentration and permeability may be required to explain experimental findings. The simulations indicate that the cost of breaking van der Waals interactions between the lipid tails of cholesterol-containing membranes account for the reduced partitioning rather than the surface area per phospholipid, which has been frequently suggested as a determinant for solute partitioning. The simulations further show that the partitioning is more sensitive to cholesterol (i) for larger solutes, (ii) in membranes with saturated as compared to membranes with unsaturated lipid tails, and (iii) in membranes with smaller lipid head groups.