Understanding the (de)mixing behavior of multicomponent lipid bilayers is an important step toward unraveling the nature of spatial composition heterogeneities in cellular membranes and their role in biological function. We use coarse-grained molecular dynamics simulations to study the composition phase diagram of a quaternary mixture of phospholipids and cholesterol. This mixture is known to exhibit both uniform and coexisting phases. We compare and combine different statistical measures of membrane structure to identify the onset of phase coexistence in composition space. An important element in our approach is the dependence of composition heterogeneities on the size of the system. While homogeneous phases can be structured and display long correlation lengths, the hallmark behavior of phase coexistence is the scaling of the apparent correlation length with system size. Because the latter cannot be easily varied in simulations, our method instead uses information obtained from observation windows of different sizes to accurately distinguish phase coexistence from structured homogeneous phases. This approach is built on very general physical principles, and will be beneficial to future studies of the phase behavior of multicomponent lipid bilayers.