Twitter API Question posted in
The official Twitter API documentation can be found here.

Twitter streaming stop collecting data – Twitter API

I’ve this following code that retrieves Twitter Streaming data and crete a JSON file. What I’d like to get is to stop the data collecting after fo eg.1000 tweets. How can I set the code?

#Import the necessary methods from tweepy library
from tweepy.streaming import StreamListener
from tweepy import OAuthHandler
from tweepy import Stream

# Other libs
import json

#Variables that contains the user credentials to access Twitter API
access_token = "XXX"
access_token_secret = "XXX"
consumer_key = "XXX"
consumer_secret = "XXX"

#This is a basic listener that just prints received tweets to stdout.
class StdOutListener(StreamListener):

    def on_data(self, data):

        try:
            tweet = json.loads(data)
            with open('your_data.json', 'a') as my_file:
                json.dump(tweet, my_file)


        except BaseException:
            print('Error')
            pass

    def on_error(self, status):
        print ("Error " + str(status))
        if status == 420:
            print("Rate Limited")
            return False


if __name__ == '__main__':

    #This handles Twitter authetification and the connection to Twitter Streaming API
    l = StdOutListener()
    auth = OAuthHandler(consumer_key, consumer_secret)
    auth.set_access_token(access_token, access_token_secret)
    stream = Stream(auth, l)


    stream.filter(track=['Euro2016', 'FRA', 'POR'], languages=['en'])
Tags:

2

Answers


  1. 0

    Here is a possible solution:

    class StdOutListener(StreamListener):

    tweet_number=0   # class variable

    def __init__(self,max_tweets):
        self.max_tweets=max_tweets # max number of tweets

    def on_data(self, data):
        self.tweet_number+=1   
        try:
            tweet = json.loads(data)
            with open('your_data.json', 'a') as my_file:
                json.dump(tweet, my_file)
        except BaseException:
            print('Error')
            pass
        if self.tweet_number>=self.max_tweets:
            sys.exit('Limit of '+str(self.max_tweets)+' tweets reached.')

    def on_error(self, status):
        print ("Error " + str(status))
        if status == 420:
            print("Rate Limited")
            return False

l = StdOutListener(1000) # Here you can set your maximum number of tweets (1000 in this example)

    After having defined the class variable tweet_number, I used the init() method to initialize a new StdOutListener object with the maximum number of tweets you want to collect. tweet_number is increased by 1 each time the on_data(data) method is called, causing the program to terminate when tweet_number>=max_tweets

    P.S. You need to import sys for the code to work.

    Login or Signup to reply.
  2. 0

    This is the 2.7 code I would use — sorry, I do not know 3.0 as well… I think you want what is is on my second line. .items(1000) part…?

    stackoverflow messed up my indentations in my code. I am also using tweepy.

    CODE:

            results = []
    for tweet in tweepy.Cursor(api.search, q='%INSERT_SEARCH_VARIABLE HERE').items(1000): #THE 1000 IS WHERE YOU SAY SEARCH FOR 1000 TWEETS. 
        results.append(tweet)
    
    print type(results)
    print len(results)
def toDataFrame(tweets):

    DataSet = pd.DataFrame()

    DataSet['tweetID'] = [tweet.id for tweet in tweets]
    DataSet['tweetText'] = [tweet.text for tweet in tweets]
    DataSet['tweetRetweetCt'] = [tweet.retweet_count for tweet 
    in tweets]
    DataSet['tweetFavoriteCt'] = [tweet.favorite_count for tweet 
    in tweets]
    DataSet['tweetSource'] = [tweet.source for tweet in tweets]
    DataSet['tweetCreated'] = [tweet.created_at for tweet in tweets]


    DataSet['userID'] = [tweet.user.id for tweet in tweets]
    DataSet['userScreen'] = [tweet.user.screen_name for tweet 
    in tweets]
    DataSet['userName'] = [tweet.user.name for tweet in tweets]
    DataSet['userCreateDt'] = [tweet.user.created_at for tweet 
    in tweets]
    DataSet['userDesc'] = [tweet.user.description for tweet in tweets]
    DataSet['userFollowerCt'] = [tweet.user.followers_count for tweet 
    in tweets]
    DataSet['userFriendsCt'] = [tweet.user.friends_count for tweet 
    in tweets]
    DataSet['userLocation'] = [tweet.user.location for tweet in tweets]
    DataSet['userTimezone'] = [tweet.user.time_zone for tweet 
    in tweets]
    
        return DataSet
    
    #Pass the tweets list to the above function to create a DataFrame
    tweet_frame = toDataFrame(results)
tweet_frame[0:999]
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search