Question 9291 in Twitter API Question posted in
The official Twitter API documentation can be found here.

Streaming multiple users in Twitter API v2

I am migrating from Twitter APIv1.1 to v2, and trying to stream tweets of users filtered by their ids. I use twarc.client2.Twarc2 in python and build a list of rules in this form:


{'value': 'from:user1 or from:user2 or ...'}

I have more than thousand users and since each rule is restricted with 512 characters according to twitter API (because of my account), I splitted user ids to different rules. And while streaming I change rules in time interval of 37 seconds, since API restricts to add only 25 rules in 15 minutes. What I have tried so far is rougly like this:

def manage_streaming():
    event = threading.Event()
    thread1 = threading.Thread(target=streaming, args=(event,))
    thread1.start()
    index = 0
    while True:
        sleep(37)
        update_rule(index)
        index = 0 if index == max_index else index+1
        if event.is_set():
            break

where streaming function is like:


def streaming(event):
    results = client.stream(event=event)
    for result in results:
        process_tweet(result)

and update_rule function is like:

def update_rule(i):
    clear_rules()
    add_rule(rules_2_add[i])

It is working correctly, but it takes about 43 minutes for all users to finish streaming and start again.
So my question:

Is there a better way to stream multiple users in API v2? Or is it possible to open multiple streaming ports with different rules in each of them?

Thank you in advance. All suggestions are appreciated.

EDIT!!

I mixed API restrictions, my account is essential so I am allowed to add only 5 rules; I can add 5 rules concurrently or asynchronously after that If I wanna add more rules I need to delete the old ones and wait for 15 mins, there is no way to add more than 5 rules. Still this is not efficient with more users.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    After searching for methods and ways I saw that there are two possible ways (not programmatically) to overcome this problem:

    1. Upgrade your essential access level account to elevated access level. You can retrieve up to 2 million tweets per month for free and create up to 3 apps in the same project. Moreover, each of these apps accepts up to 25 rules, enabling you to add 12800 character long rules in total. I created nearly 60 rules and added each app at most 25 rule, by this I was able to solve the problem. Also, while creating rules, I compared length of twitter user name and id, then I used smaller one. This is a valid rule:
       {'value': 'from:user_1_name or from:user_2_id or ...'}
    1. If your account is not eligible for elevated access, try to create multiple essential accounts and use all of them to stream.

    For more about account levels: Official Page


  2. 0

    Here is a simple example, for more details you can visit the following URLs

    class TweetListener(tweepy.StreamingClient):

    def on_tweet(self, tweet):
        print(tweet.text)


stream = TweetListener(bearer_token=tw.bearer_token)

stream.add_rules(tweepy.StreamRule("from:username01 OR from:username02"))
stream.filter()

    https://twittercommunity.com/t/deprecation-announcement-removing-compliance-messages-from-statuses-filter-and-retiring-statuses-sample-from-the-twitter-api-v1-1/170500

    example api v1 and v2 in tweepy

    more stream ruls docs

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search