Twitter API Question posted in
The official Twitter API documentation can be found here.

Twitter-api – R – Use Twitter API to get every tweet from an account

My goal is to get EVERY tweet ever for any twitter account. I picked the NYTimes for this example.

The code below works, but it only pulls the last 100 tweets. max_results does not allow you to put a value over 100.

The code below almost fully copy-paste-able, you would have to have your own bearer token.

How can I expand this to give me every tweet from an account?

One idea is that I can loop it for every day since the account was created, but that seems tedious if there is a faster way.

# NYT Example --------------------------------------------------------------------

library(httr)
library(jsonlite)
library(tidyverse)

bearer_token <- "insert your bearer token here"
headers <- c(`Authorization` = sprintf('Bearer %s', bearer_token))

params <- list(`user.fields` = 'description')
handle <- 'nytimes'
url_handle <- sprintf('https://api.twitter.com/2/users/by?usernames=%s', handle)

response <- httr::GET(url = url_handle,
                      httr::add_headers(.headers = headers),
                      query = params)
json_data <- fromJSON(httr::content(response, as = "text"), flatten = TRUE)
json_data %>% 
  as_tibble()

NYT_ID <- json_data$data$id


url_handle <- paste0("https://api.twitter.com/2/users/", NYT_ID, "/tweets")
params <- list(`tweet.fields` = 'id,text,author_id,created_at,attachments,public_metrics',
               `max_results` = '100')

response <- httr::GET(url = url_handle,
                      httr::add_headers(.headers = headers),
                      query = params)
json_data <- fromJSON(httr::content(response, as = "text"), flatten = TRUE)

NYT_tweets <- json_data$data %>% 
  as_tibble() %>% 
  select(-id, -author_id, -9)
NYT_tweets
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    For anyone that finds this later on, I found a solution that works for me.

    Using the parameters of start_time and end_time you can clarify dates for the tweets to be between. I was able to pull all tweets from November for example and then rbind those to the ones from December, etc. Sometimes I had to do two tweet pulls (half of March, second half of March) to get all of them, but it worked for this.

    params <- list(`tweet.fields` = 'id,text,author_id,created_at,attachments,public_metrics',
               `max_results` = '100',
               `start_time` = '2021-11-01T00:00:01.000Z',
               `end_time` = '2021-11-30T23:58:21.000Z')

  2. 0

    when I run this, it says the client is forbidden and that I must use keys and tokens from a Twitter dev app that is attached to a project. I have already made a project but where do I input my keys/tokens into R.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search