How To Exit From A Generator At Some Specific Time?

I'm reading tweets from Twitter Streaming API. After connecting to the API, I'm getting a generator. I'm looping through each tweet received but I want to exit from the iterator,

Solution 1:

Create a separate thread for the producer and use a Queue to communicate. I also had to use a threading.Event for stopping the producer.

import itertools, queue, threading, time

END_TIME = time.time() + 5# run for ~5 secondsdeftime_left():
    return END_TIME - time.time()

defConnectAndGetStream():             # stub for the real thingfor i in itertools.count():
        time.sleep(1)
        yield"tweet {}".format(i)

defproducer(tweets_queue, the_end):   # producer
    it = ConnectAndGetStream()
    whilenot the_end.is_set():
        tweets_queue.put(next(it))

defgetStream(tweets_queue, the_end):  # consumertry:
        whileTrue:
            tweet = tweets_queue.get(timeout=time_left())
            print('Got', tweet)
    except queue.Empty:
        print('THE END')
        the_end.set()

tweets_queue = queue.Queue()  # you might wanna use the maxsize parameter
the_end = threading.Event()
producer_thread = threading.Thread(target=producer,
                                   args=(tweets_queue, the_end))
producer_thread.start()
getStream(tweets_queue, the_end)
producer_thread.join()

Solution 2:

Your problem could be resolved by splitting the functionality of your design into two separated processes:

1. A twitter process that acts as wrapper to Twitter API and
2. A monitor process that is able to terminate the twitter process when the exit time is reached.

The following piece of code prototypes the functionality described above using Python's multiprocessing module:

import multiprocessing as mp
import time

EXIT_TIME = '12:21'#'18:00'deftwitter():

    whileTrue:
        print'Twittttttttttt.....'
        time.sleep(5)

defget_time():

    return time.ctime().split()[3][:5]

if __name__ == '__main__':

    # Execute the function as a process
    p = mp.Process( target=twitter, args=() )
    p.start()

    # Monitoring the process pwhileTrue:
        print'Checking the hour...'if get_time() == EXIT_TIME:
            p.terminate()
            print'Current time:', time.ctime()
            print'twitter process has benn terminated...'break
        time.sleep(5)

Of course you can use p.join(TIMEOUT) instead of using the while True loop presented in my example as pointed here.

Solution 3:

Here is an example with threading and python scheduler:

import threading
import time
import os
import schedule

def theKillingJob():
    print("Kenny and Cartman die!")
    os._exit(1)

schedule.every().day.at("18:00").do(theKillingJob,'It is 18:00')

def getStream(tweet_iter):
    for tweet in tweet_iter:
        #do stuff

def kenny():
    while True:
        print("Kenny alive..")
        schedule.run_pending()
        time.sleep(1)

def cartman():
    while True:
        print("Cartman alive..")

        tweet_iter = ConnectAndGetStream()
        getStream(tweet_iter)

        # You can change whenever you want to check for tweets by changing sleep time here
        time.sleep(1)

if __name__ == '__main__':
    daemon_kenny = threading.Thread(name='kenny', target=kenny)
    daemon_cartman = threading.Thread(name='cartman', target=cartman)
    daemon_kenny.setDaemon(True)
    daemon_cartman.setDaemon(True)

    daemon_kenny.start()
    daemon_cartman.start()
    daemon_kenny.join()
    daemon_cartman.join()