Twitter API Question posted in
The official Twitter API documentation can be found here.

Tweepy OpenSSL.SSL.WantReadError – Twitter API

Python 3.6. I use the streamer of tweepy to get tweets. It works well. But sometimes, if I let it open for more than 24h, I have this error

Traceback (most recent call last):
  File "C:ProgramDataAnaconda3libsite-packagesrequestspackagesurllib3contribpyopenssl.py", line 277, in recv_into
return self.connection.recv_into(*args, **kwargs)
  File "C:ProgramDataAnaconda3libsite-packagesOpenSSLSSL.py", line 1547, in recv_into
self._raise_ssl_error(self._ssl, result)
    File "C:ProgramDataAnaconda3libsite-packagesOpenSSLSSL.py", line 1353, in _raise_ssl_error
raise WantReadError()
  OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred:


Traceback (most recent call last):
 File "C:ProgramDataAnaconda3libsite-packagesrequestspackagesurllib3contribpyopenssl.py", line 277, in recv_into
return self.connection.recv_into(*args, **kwargs)
    File "C:ProgramDataAnaconda3libsite-packagesOpenSSLSSL.py", line 1547, in recv_into
self._raise_ssl_error(self._ssl, result)
  File "C:ProgramDataAnaconda3libsite-packagesOpenSSLSSL.py", line 1370, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (10054, 'WSAECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:ProgramDataAnaconda3libsite-packagesrequestspackagesurllib3response.py", line 302, in _error_catcher
yield
  File "C:ProgramDataAnaconda3libsite-packagesrequestspackagesurllib3response.py", line 384, in read
data = self._fp.read(amt)
  File "C:ProgramDataAnaconda3libhttpclient.py", line 449, in read
n = self.readinto(b)
  File "C:ProgramDataAnaconda3libhttpclient.py", line 483, in readinto
return self._readinto_chunked(b)
  File "C:ProgramDataAnaconda3libhttpclient.py", line 578, in _readinto_chunked
chunk_left = self._get_chunk_left()
  File "C:ProgramDataAnaconda3libhttpclient.py", line 546, in _get_chunk_left
chunk_left = self._read_next_chunk_size()
  File "C:ProgramDataAnaconda3libhttpclient.py", line 506, in _read_next_chunk_size
line = self.fp.readline(_MAXLINE + 1)
  File "C:ProgramDataAnaconda3libsocket.py", line 586, in readinto
return self._sock.recv_into(b)
  File "C:ProgramDataAnaconda3libsite-packagesrequestspackagesurllib3contribpyopenssl.py", line 293, in recv_into
return self.recv_into(*args, **kwargs)
  File "C:ProgramDataAnaconda3libsite-packagesrequestspackagesurllib3contribpyopenssl.py", line 282, in recv_into
raise SocketError(str(e))
OSError: (10054, 'WSAECONNRESET')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:ProgramDataAnaconda3libthreading.py", line 916, in _bootstrap_inner
self.run()
  File "C:ProgramDataAnaconda3libthreading.py", line 864, in run
self._target(*self._args, **self._kwargs)
 File "twitter_aspi_v0.8.py", line 179, in _init_stream
tweepy.Stream(auth, listener).userstream()
  File "C:ProgramDataAnaconda3libsite-packagestweepystreaming.py", line 396, in userstream
self._start(async)
  File "C:ProgramDataAnaconda3libsite-packagestweepystreaming.py", line 363, in _start
self._run()
  File "C:ProgramDataAnaconda3libsite-packagestweepystreaming.py", line 296, in _run
raise exception
  File "C:ProgramDataAnaconda3libsite-packagestweepystreaming.py", line 265, in _run
self._read_loop(resp)
  File "C:ProgramDataAnaconda3libsite-packagestweepystreaming.py", line 315, in _read_loop
line = buf.read_line().strip()
  File "C:ProgramDataAnaconda3libsite-packagestweepystreaming.py", line 180, in read_line
self._buffer += self._stream.read(self._chunk_size)
  File "C:ProgramDataAnaconda3libsite-packagesrequestspackagesurllib3response.py", line 401, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "C:ProgramDataAnaconda3libcontextlib.py", line 100, in __exit__
self.gen.throw(type, value, traceback)
  File "C:ProgramDataAnaconda3libsite-packagesrequestspackagesurllib3response.py", line 320, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection broken: OSError("(10054, 'WSAECONNRESET')",)', OSError("(10054, 'WSAECONNRESET')",))

My code is pretty long and regarding the error, it seems it comes from urllib3, OpenSSL and tweepy way of accessing the Twitter API. So I could handle this with a try before launching the streamer but I would like to know if there is maybe a better fix I could do to understand and avoid this? Thanks!

Tags:

2

Answers


  1. 0

    According to Twitter Developer documentation : rate limiting this is expected to connection reset/failure when you cross your usage limit.

    Requests / 15-min window (user auth) = 900

    Requests / 15-min window (app auth) = 1500

    Also it clearly states as below.

    If the initial reconnect attempt is unsuccessful, your client should continue attempting to reconnect using an exponential back-off pattern until it successfully reconnects.

    (Update)

    Regardless of how your client gets disconnected, you should configure
    your app to reconnect immediately. If your first reconnection attempt
    is unsuccessful, we recommend that your app implement an exponential
    back-off pattern in subsequent reconnection attempts (e.g. wait 1
    second, then 2 seconds, then 4, 8, 16, etc), with some reasonable
    upper limit. If this upper limit is reached, you should configure your
    client to notify your team so that you can investigate further.

    The standard (free) Twitter APIs i.e Tweepy API consist of a REST API and a Streaming API. The Streaming API provides low-latency access to Tweets. Ads API has other limits when white listed.

    REST API Limit

    Clients may access a theoretical maximum of 3,200 statuses via the
    page and count parameters for the user_timeline REST API methods.
    Other timeline methods have a theoretical maximum of 800 statuses.
    Requests for more than the limit will result in a reply with a status
    code of 200 and an empty result in the format requested. Twitter still
    maintains a database of all the Tweets sent by a user. However, to
    ensure performance, this limit is in place on the API calls.

    Could be simple reason that users should not spam that this thing is enforced.

    Solution :

    You may catch exception and re-establish connection to Twitter and continue reading tweets.

    There is no alternative than getting better usage allowance from Twitter as of now unfortunately.

    Login or Signup to reply.
  2. 0

    This looks more like a temporary connection timeout which is not handled by Tweepy, so you should just write a wrapper around the same and catch the exception and restart it. I don’t think the exception can be avoided as such because you connect to an external site and sometimes it could timeout

    You should look at this http://docs.tweepy.org/en/v3.5.0/streaming_how_to.html#handling-errors for the error handling part and see if on_error gets called in your case when connection timeout happens

    class MyStreamListener(tweepy.StreamListener):

    def on_error(self, status_code):
        if status_code == 420:
            #returning False in on_data disconnects the stream
            return False

    If this doesn’t help then use the wrapper approach

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search