Twitter-api – R – Use Twitter API to get every tweet from an account

My goal is to get EVERY tweet ever for any twitter account. I picked the NYTimes for this example.

The code below works, but it only pulls the last 100 tweets. max_results does not allow you to put a value over 100.

The code below almost fully copy-paste-able, you would have to have your own bearer token.

How can I expand this to give me every tweet from an account?

One idea is that I can loop it for every day since the account was created, but that seems tedious if there is a faster way.

# NYT Example --------------------------------------------------------------------

library(httr)
library(jsonlite)
library(tidyverse)

bearer_token <- "insert your bearer token here"
headers <- c(`Authorization` = sprintf('Bearer %s', bearer_token))

params <- list(`user.fields` = 'description')
handle <- 'nytimes'
url_handle <- sprintf('https://api.twitter.com/2/users/by?usernames=%s', handle)

response <- httr::GET(url = url_handle,
                      httr::add_headers(.headers = headers),
                      query = params)
json_data <- fromJSON(httr::content(response, as = "text"), flatten = TRUE)
json_data %>% 
  as_tibble()

NYT_ID <- json_data$data$id


url_handle <- paste0("https://api.twitter.com/2/users/", NYT_ID, "/tweets")
params <- list(`tweet.fields` = 'id,text,author_id,created_at,attachments,public_metrics',
               `max_results` = '100')

response <- httr::GET(url = url_handle,
                      httr::add_headers(.headers = headers),
                      query = params)
json_data <- fromJSON(httr::content(response, as = "text"), flatten = TRUE)

NYT_tweets <- json_data$data %>% 
  as_tibble() %>% 
  select(-id, -author_id, -9)
NYT_tweets