Get Twitter Followers Using Tweepy And Multiple Api Keys

I have multiple twitter dev keys that I am using to get followers from a list of handles. There are two ways I can do this but have a problem with both. The first: try: ....

Solution 1:

I actually had to end up abandoning the cursored method in favor of manually setting the next cursor. The nice thing about this is that the "non-cursored" method returns the previous and next cursor as part of it's function.

Here's how I achieved what you are going for (note: adding a try/catch is probably in order):

users = ['user_one', 'user_two', 'user_three']

current_profile = 9# I HAVE TEN IN AN ARRAY

tweepy_api = get_api(auth_profiles[current_profile]) #A FUNCTION I CREATED TO REINITIALIZE API'Sfor user in users:

    next_cursor = -1# START EVERY NEW USER RETRIEVAL WITH -1print'CURRENT USER:', user, 'STARTING CURSOR:', next_cursor

    while next_cursor: # THAT IS, WHILE CURSOR IS NOT ZEROprint'AUTH PROFILE', current_profile, 'CURRENT CURSOR:', next_cursor

        # RETURNS A TUPLE WITH ELEMENT[0] A LIST OF IDS, ELEMENT [1][0] PREVIOUS CURSOR, AND ELEMENT[1][1] NEXT CURSOR
        ids, cursors = tweepy_api.followers_ids(screen_name=user, count=5000, cursor=next_cursor)

        next_cursor = cursors[1] # STORE NEXT CURSOR# FUNCTION I CREATED TO GET STATUS FROM API.rate_limit_status()
        status = get_rate_limit_status(tweepy_api, '/followers/ids')

        print'ID\'S RETRIEVED:', len(ids), 'NEXT CURSOR:', cursors[1], 'REMAINING:', status['remaining']

        if not status['remaining']: # IF STATUS IS REMAINING IS ZEROprint''print'RATE LIMIT REACHED'if current_profile < len(auth_profiles) - 1: # IF THE CURRENT PROFILE IS LESS THAN NINE (IN MY CASE)print'INCREMENTING CURRENT PROFILE:', current_profile, '<', len(auth_profiles) - 1

                current_profile += 1# INCREMENT THE PROFILEprint'CURRENT PROFILE:', current_profile

            else: # ELSE, IT MUST EQUAL NINE (COULD BE NEG I SUPPOSE BUT...)print'RESETTING CURRENT PROFILE TO ZERO:', current_profile, '=', len(auth_profiles) - 1

                current_profile = 0# RESET CURRENT PROFILE TO THE BEGINNINGprint'CURRENT PROFILE:', current_profile

            tweepy_api = get_api(auth_profiles[current_profile]) # GET NEW TWEEPY API WITH NEW AUTHprint''

The output should be something like this (I've removed some of the print statements for simplicity):

CURRENTUSER: user_one STARTING CURSOR: -1
AUTH PROFILE 9CURRENTCURSOR: -1

ID'S RETRIEVED: 5000 NEXT CURSOR: 1594511885763407081 REMAINING: 14
…
ID'S RETRIEVED: 5000 NEXT CURSOR: 1582249691352919104 REMAINING: 0

RATE LIMIT REACHED
RESETTING CURRENT PROFILE TO ZERO: 9=9CURRENT PROFILE: 0

ID'S RETRIEVED: 5000 NEXT CURSOR: 1580277475971792716 REMAINING: 14
…
ID'S RETRIEVED: 4903 NEXT CURSOR: 0 REMAINING: 7CURRENTUSER: user_two STARTING CURSOR: -1
AUTH PROFILE 0CURRENTCURSOR: -1

ID'S RETRIEVED: 5000 NEXT CURSOR: 1592820762836029887 REMAINING: 6
…
ID'S RETRIEVED: 5000 NEXT CURSOR: 1592737463603654258 REMAINING: 0

RATE LIMIT REACHED
INCREMENTING CURRENT PROFILE: 0<9CURRENT PROFILE: 1

As a side note, if you are going to use a cursored version, at least in Tweepy 3.5.0 the prev_cursor and next_cursor are stored in cursor.iterator.next_cursor, cursor.iterator.prev_cursor. I think this is also the case for 3.6.0 (see Cursor and CursorIterator in cursor.py)

For me, cursor.page_iterator.next_cursor returns:

AttributeError: 'Cursor'object has no attribute 'page_iterator'

Solution 2:

You can get the cursor that was used when the rate limit occured through the next_cursor variable on the iterator being used. When you create a new Cursor using the new API instance, you can pass the previous cursor as a parameter:

current_cursor = cursor.iterator.next_cursor
# re-create the cursor using the new api instancecursor = tweepy.Cursor(api.followers, screen_name=s, cursor=current_cursor)