Understanding the prevalence and impact of personal attacks in online discussions is challenging. A method that combines crowdsourcing and machine learning provides a way forward, but caveats must be considered.