Python 2.6 Send Connection Object Over Queue / Pipe / Etc

Given this bug (Python Issue 4892) that gives rise to the following error: >>> import multiprocessing >>> multiprocessing.allow_connection_pickling() >>>

Solution 1:

(What I believe is) A better method, after some playing around (I was having the same problem. Wanted to pass a pipe through a pipe.) before discovering this post:

>>>from multiprocessing import Pipe, reduction>>>i, o = Pipe()>>>reduced = reduction.reduce_connection(i)>>>newi = reduced[0](*reduced[1])>>>newi.send("hi")>>>o.recv()
'hi'

I'm not entirely sure why this is built this way (someone would need insight into what the heck the reduction part of multiprocessing is about for that) but it does definitely work, and requires no pickle import. Other than that, it's pretty close to the above in what it does, but simpler. I also threw this into the python bug report so others know of the workaround.

Solution 2:

Here's roughly what I did:

# Producerfrom multiprocessing.reduction import reduce_connection
from multiprocessing import Pipe

   # Producer and Consumer share the Queue we call queuedefhandle(queue):
   reader, writer = Pipe()
   pickled_writer = pickle.dumps(reduce_connection(writer))
   queue.put(pickled_writer)

and

# Consumerfrom multiprocessing.reduction import rebuild_connection

def wait_for_request():
    pickled_write = queue.get(block=True) # block=True isn't necessary, of course
    upw = pickle.loads(pickled_writer) # unpickled writer
    writer = upw[0](upw[1][0],upw[1][1],upw[1][2])

The last line is cryptic, coming from the following:

>>> upw
(<functionrebuild_connectionat 0x1005df140>,
(('/var/folders/.../pymp-VhT3wX/listener-FKMB0W',
17, False), True, True))

Hope that helps someone else. It works fine for me.