How To Resolve Spinlock Issues With Multithreaded Python?

I'm writing a multithreaded application in Python and have came across an issue where kernel time skyrockets. Using perf I see that it is indeed a spinlock: 54.89% python

Solution 1:

I really do not know which spinlock this is but how many threads do you start?

How about not starting threads but processes?

have a look at multiprocessing or in Python 3.

from concurrent.futures import ThreadPoolExecutor # should do exactly what your code does nowfrom concurrent.futures import ProcessPoolExecutor # I recommend that

If the ProcessPoolExecutor does not resolve the problem then the switches between the Python threads should not be the problem.