Understanding the ferroelectrocity in magnetic ferroelectric oxides is of both fundamental and technological importance. Here, we identify the nature of the ferroelectric phase transition in the hexagonal manganite, YMnO3, using a combination of single-crystal X-ray diffraction, thorough structure analysis and first-principles density-functional calculations. The ferroelectric phase is characterized by a buckling of the layered MnO5 polyhedra, accompanied by displacements of the Y ions, which lead to a net electric polarization. Our calculations show that the mechanism is driven entirely by electrostatic and size effects, rather than the usual changes in chemical bonding associated with ferroelectric phase transitions in perovskite oxides. As a result, the usual indicators of structural instability, such as anomalies in Born effective charges on the active ions, do not hold. In contrast to the chemically stabilized ferroelectrics, this mechanism for ferroelectricity permits the coexistence of magnetism and ferroelectricity, and so suggests an avenue for designing novel magnetic ferroelectrics.