Abstract
Perceptions of hate can vary greatly across cultural contexts. Hate speech(HS) datasets, however, have traditionally been developed by language. Thishides potential cultural biases, as one language may be spoken in differentcountries home to different cultures. In this work, we evaluate cultural biasin HS datasets by leveraging two interrelated cultural proxies: language andgeography. We conduct a systematic survey of HS datasets in eight languages andconfirm past findings on their English-language bias, but also show that thisbias has been steadily decreasing in the past few years. For threegeographically-widespread languages -- English, Arabic and Spanish -- we thenleverage geographical metadata from tweets to approximate geo-cultural contextsby pairing language and country information. We find that HS datasets for theselanguages exhibit a strong geo-cultural bias, largely overrepresenting ahandful of countries (e.g., US and UK for English) relative to their prominencein both the broader social media population and the general population speakingthese languages. Based on these findings, we formulate recommendations for thecreation of future HS datasets.