Twitter Corpus of the #BlackLivesMatter Movement And Counter Protests: 2013 to 2020

• These data are useful because they showcase the entire course of a large, ongoing social movement (Black Lives Matter) and its counter protests (All Lives Matter and Blue Lives Matter). To our knowledge, no other Twitter data sets exist that cover the entire span of the Black Lives Matter movement to date.

• All researchers interested in systemic racism, social movements, grassroots campaigns, racial inequality, police brutality and counter protests, especially those working in the fields of computational social science, communications, and political science, can benefit from this data.

• The data set contains 41.8 million posts from 10 million users and can be used to identify linguistic patterns associated with the social movements and their counter protests, social networks (through friend/follower user data), temporal and spatial patterns (through the use of timestamps and latitude/longitude coordinates), inter- and intra- movement dialog and the spread of news and misinformation (through retweets and tweets linking news articles).

• Since 2013, the BLM movement has grown exponentially, resulting in global protests and several counter protests. This historical data, starting in 2013 and ending in 2020, permits researchers to track this grassroots movement from its social media beginnings.

Tweets containing the keywords BlackLivesMatter, AllLivesMatter and BlueLivesMatter were collected from the Twitter API from January 2013 to June 2020. Table 1 contains counts of total number of tweets and users for the entire data set and each keyword. It also includes counts for the following: retweets (original tweets which are shared by other users on the platform), replies (tweets which directly respond to another tweet), geotagged (latitude/longitude coordinates associated with the tweet) and top languages (automatically detected language of the tweet). Retweets may or may not contain additional content created by the user doing the retweeting.

Tweets also contain a large number of other pieces of metadata, such as user profile data and place information. User profiles contain information such as user handles, free text descriptions and profile images. Places are named locations users decide to associate with a tweet. While Places describe physical locations, they do not imply the tweet originated from this location. Twitter users may manually tag a location when their tweet is about that Place. Due to the large number of additional fields available for each tweet, we do not provide counts for any additional content.

The monthly volume of each keyword is plotted in Figure 1. Here we plot the seven day running average of the total count (logged) of all tweets containing one of our keywords. All labels marked with a single name indicate the date of high profile police brutality-related killings.

On July 14, 2016, we set up a data puller using the Python package TwitterMySQL² to collect tweets matching at least one of our keywords: BlackLivesMatter, AllLivesMatter and BlueLivesMatter. This package uses the official Twitter Application Programming Interface (API) to stream tweets in real time. The data puller continuously collected tweets from the Twitter stream until the time of writing (July 2020). In total we collected 50,574,955 tweets. While the Twitter API was queried using the keywords BlackLivesMatter, AllLivesMatter and BlueLivesMatter, the API delivers a more robust set of matching tweets. For example, a tweet might contain the phrase “black lives matter” or “blm”, among other variations, instead of the keyword BlackLivesMatter.

We note that the Twitter API limits such streams to 1% of the total Twitter volume at any given moment. To see if our keyword data set was limited at any point, we compared the monthly keyword volume to a full 1% monthly pull (not limited to any single keyword, location, etc.). Over the 4-year time span, our keyword data set pulled in a monthly average of 1,176,161 tweets (4,629,878 SD) as compared to a monthly average of 94,893,476 tweets (27,394,826 SD) from the full 1% pull. Thus, we do not believe our data set was limited by the Twitter API.

Due to server maintenance, there were periods when we were unable to collect data. These include: October 17, 2016 through November 23, 2016; January 1, 2017 through January 21, 2017; March 11, 2017 through March 16, 2017; May 2, 2018 through December 18, 2018; and March 16, 2019 through March 20, 2019. Additionally, the Black Lives Matter movement began in 2013, roughly three years before the beginning of our data collection. In order to fill these gaps, we used the Python package GetOldTweets3 [9], which pulls historical tweets containing a given keyword. These tweets were pulled in June 2020. Using this method, we collected 4,276,423 historical tweets.

Having two separate methods of pulling tweet data (prospective using the streaming API and retrospective using GetOldTweets3³) caused inconsistencies when reconstructing timelines of keyword use. While Twitter data is publicly available, at any point a user may delete a tweet, delete their account, or set their account to private. Thus, when pulling prospective data, we collected tweets which may have been deleted or made private at some point after the initial pull. On the other hand, one cannot pull deleted or private tweets with a retrospective collection. In order to ensure the data set only contained presently available tweets, we executed a one-time historical pull in June 2020. As a result, any tweet deleted after our initial pull will not be made available. Our final data set consisted of 41,801,153 tweets.

Due to Twitter’s Terms of Service, only numeric tweet IDs can be publicly shared. The numeric IDs can be used to pull the full tweet set using the Twitter API. There are a number of open source software packages which allow researchers to easily interface with the API. The authors used the Python package TwitterMySQL${}^1$, which saves tweet information in a MySQL database. Other packages exist which do not rely on relational databases, such as the Python package twarc⁴, which saves tweets to text files in JSON format. Finally, Hydrator⁵, which relies on an easy to use GUI, saves tweets to both JSON and CSV formats.

The data used in this article is publicly available and distributed within Twitter’s Terms of Services. Additionally, no human subjects were used in the data collection.

This research was supported in part by the Intramural Research Program of the NIH, National Institute on Drug Abuse (NIDA).

The authors declare that they have no known competing financial interests or personal relationships which have, or could be perceived to have, influenced the work reported in this article.