Twitter API Question posted in
The official Twitter API documentation can be found here.

Tweets scraping – how to measure tweeting intensity? – Twitter API

I am looking for a method to get information of a "trend" regarding some hashtag/key word on Twitter. Let`s say I want to measure how often the hashtag/key word "Python" is tweeted in time. For instance, today, "Python" is tweeted on average every 1 minute but yesterday it was tweeted on average every 2 minutes.

I have tried various options but I am always bouncing off the twitter API limitations, i.e. if I try to download all tweets for a hashtag during the last (for example) day, only a certain franction of the tweets is downloaded (via tweepy.cursor).

Do you have any ideas / script examples of achieving similar results? Libraries or guides to recommend? I did not find any help searching on the internet. Thank you.

Tags:

2

Answers


  1. 0

    Try a library called:
    GetOldTweets or GetOldTweets3

    Twitter Search, and by extension its API, are not meant to be an exhaustive source of tweets. The Twitter Streaming API places a limit of just one week on how far back tweets can be extracted from that match the input parameters. So in order to extract all historical tweets relevant to a set of search parameters for analysis, the Twitter Official API needs to be bypassed and custom libraries that mimic the Twitter Search Engine need to be used.

    Login or Signup to reply.
  2. 0

    You should check twint repository.

    • Can fetch almost all Tweets (Twitter API limits to last 3200 Tweets only);
    • Fast initial setup;
    • Can be used anonymously and without Twitter sign up;

    here is a sample code:

    import twint


def scrapeData(search):
    c = twint.Config()

    c.Search = search

    c.Since = '2021-03-05 00:00:00'
    c.Until = '2021-03-06 00:00:00'
    c.Pandas = True
    c.Store_csv = True
    c.Hide_output = True
    c.Output = f'{search}.csv'
    c.Limit = 10  # number of tweets want to fetch

    print(f"n#### Scraping from {c.Since} to {c.Until}")
    twint.run.Search(c)

    print("n#### Preview: ")
    print(twint.storage.panda.Tweets_df.head())


if __name__ == "__main__":
    scrapeData(search="python")
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search