BotSlayer

botslayer logo

BotSlayer is an application that helps track and detect potential manipulation of information spreading on Twitter. The tool is developed by the Observatory on Social Media at Indiana University --- the same lab that brought to you Botometer and Hoaxy.

BotSlayer can be used, for example, by journalists, corporations, and political candidates to discover in real-time new coordinated campaigns in their domains of interest, without any prior knowledge of these campaigns. The system is easily installed and configured in the cloud to monitor bot activity around a standing user-defined query. All you need is a Twitter developer app key to fetch data from the Twitter streaming API.

BotSlayer uses an anomaly detection algorithm to flag hashtags, links, accounts, and media that are trending and amplified in a coordinated fashion by likely bots. A Web dashboard lets users explore the tweets and accounts associated with suspicious campaigns via Twitter, visualize their spread via Hoaxy, and search related images and content on Google.

BotSlayer is designed as a crowdsourcing platform: in exchange for the free service, the system provides anonymous data back to our lab for research, in a way that is compliant with Twitter’s terms and guidelines. The data will aid in the study and early detection of social media manipulation phenomena.

We gratefully acknowledge generous support for this project from Craig Newmark Philanthropies.


Getting the Software (Currently in Beta-Testing)

BotSlayer is released under this End User License Agreement (EULA). The tools bundled with BotSlayer use these open source licenses. In addition, the following operating systems are aggregated and provided in conjunction with BotSlayer: Debian (license) and Ubuntu (license).

botslayer dashboard

If you would like to install and run BotSlayer, submit this form. You must be logged into a Google account to verify your identity, and agree to the EULA. You will then receive details needed to follow the installation instructions below.

You have two installation options:

  1. The easiest is to use Amazon Web Services (go to instructions)
  2. To install the software on your own server you can use Docker (go to instructions)

Once you install BotSlayer and access the Web dashboard, click "Config" in the menu, enter a password of your choosing, and then provide your Twitter developer app keys and a standing query. See the Help page for further instructions and pointers about Twitter keys and query format.

BotSlayer-CE

BotSlayer-CE (Community Edition) is the open source version of BotSlayer. Please be aware that the "CE" version uses simple heuristics to calculate bot scores, which may not be suitable for research. We strongly recommend the "Pro" version of BotSlayer introduced here, which uses proprietary BotometerLite software and has many other improvements.

Credit

If you use BotSlayer for research, please cite the following publication: Hui et al., (2019). BotSlayer: real-time detection of bot amplification on Twitter. Journal of Open Source Software, 4(42), 1706, DOI: 10.21105/joss.01706

Warning

The dashboard is accessible via the Web using the IP address of the server. DO NOT SHARE THE IP OR URL with anyone outside your organization or anyone you do not trust. They could make changes to the system or access data in violation of Twitter terms of service. To prevent potential security issue and terms violation, BotSlayer blocks search engine indexing by default.

Installation Instructions for AWS Users (Easiest)

BotSlayer uses Amazon Web Services (AWS) via an Amazon Machine Image (AMI) to streamline the setup process for non-technical users.

  • Step 1. Go to AWS. If you do not have an account, you will have to create one. This is free but requires a credit card. You can select the Free Tier. Sign in to the Console. In the AWS Management Console, Click on "All services", then "EC2".
    step 1
  • Step 2. Click the button to launch an EC2 instance.
    step 2
  • Step 3. After you requested the software and agreed to the EULA, you should have received instructions that include a long secret string like 0e...f2. Type this in the search box at the top. Click "Community AMIs" tab on the left. Select the one result under Community AMIs. If the search returns nothing, please set your AWS service region to Ohio in the top right corner of the page and search again.
    step 3
  • Step 4. Select the correct image.
    step 4
  • Step 5. You can select the instance type marked "free tier eligible" for free. Alternatively you can select the "t2.xlarge", which is recommended but not free; it may cost around $0.20/hour. Click "Configure Instance Details" at bottom-right to proceed.
    step 5
  • Step 6. Use the default settings on the "Configure Instance Details" page, and directly click "Add Storage" at bottom-right to proceed.
    step 6
  • Step 7. Choose your hard drive size. You can select up to 30GB for the free tier; we recommend 100GB or more for keeping data beyond several days, depending on how much information you track. Click "Add Tags" at bottom-right to proceed.
    step 7
  • Step 8. Use the default settings on the "Add Tags" page, and directly click "Configure Security Group" to proceed.
    step 8
  • Step 9. Add two rules to open the ports required by BotSlayer, as shown in the screenshot. Click "Review and Launch" at bottom-right to proceed.
    step 9
  • Step 10. Carefully review the configuration of the machine. If you spot any error, you can go back to fix the error. Otherwise, click "Launch" at bottom-right to proceed.
    step 10
  • Step 11. Create a new key pair, and give it a meaningful name, such as BotSlayer. This key pair is required to access the EC2 machine. Download the key pair and keep it at a secured environment. Then launch the instance.
    step
                11
  • Step 12. You have finished setting up BotSlayer, so click the instance link to go to Step 13.
    step
                12
  • Step 13. Copy either the domain name or the IP address and paste it into your browser to access the web interface of BotSlayer. Please wait 5 minutes so that the machine has enough time to start BotSlayer. Bookmark the IP address, as this is how you will access the BotSlayer dashboard. Note: if you restart the EC2 instance, the IP address will change. To assign a static IP address you can use an Elastic IP Address (not free).
    step 13

Installation Instructions for Docker Users

BotSlayer can be installed on any server via a prebuilt Docker image. To install Docker on your server machine, please follow the instruction on the Docker website.

After you requested the software and agreed to the EULA, you should have received a URL linking to a Docker image, and a password. Download the image file. You will need to enter a username (botslayer) and the given password. For example, you can use the following command to download the Docker image (replace url2image with the URL you received):

wget --user=botslayer --ask-password url2image

Then run the following commands to load the downloaded image and run the Docker container. Replace filename with the name of the downloaded file.

gunzip filename.gz
docker load < filename 
docker volume create pgdata
docker volume create rpdata
docker run -dit -p 5000:5000 -p 9001:9001 -v rpdata:/root/bev -v pgdata:/var/lib/postgresql/data bev

The last command maps ports that provide different functionalities. The web interfaces are available at localhost or the IP address of your server. The BotSlayer dashboard is at port 5000. The frontail logging interface is exposed at port 9001, streaming log files inside the container.

For easier access to the dashboard on the default HTTP port (80), you could either (1) setup a reverse proxy from port 5000 to port 80, or (2) use sudo to force map port 5000 to port 80 when running your docker container.

If your server restarts for any reason, you will have to restart the BotSlayer Docker container. An alternative solution is to setup some process manager, such as supervisord.

If you would like to see an example of how we setup our EC2 environment, which includes installing Docker, setting up reverse proxy on nginx, and configuring supervisord, please refer to this gist for our setup steps.