In Python3 it is actually very easy to create a fix number of worker threads and fetch a result as soon as it's available. The thread target in this case is a function which takes one argument.
According to the multiprocessing documentation, the developers recommend to use the concurrent.futures module because of it's compatibility. I also prefer it since multiple function arguments can be simply passed as comma separated list, e.g. executor.submit(sleep, first_arg, second_arg)
Those two example implementations process the result for every thread as soon as it's available.
concurrent.futures implementation
from concurrent.futures import ThreadPoolExecutor, as_completed
import time
tasks = [55, 34, 21, 13, 8, 5]
num_threads = 3
def sleep(a):
time.sleep(a)
return a
with ThreadPoolExecutor(max_workers=num_threads) as executor:
futures = []
for task in tasks:
futures.append(executor.submit(sleep, task))
for future in as_completed(futures):
r = future.result()
print(f'Sleeped for {r} seconds')multiprocessing.pool implementation
from multiprocessing.pool import ThreadPool
import time
tasks = [55, 34, 21, 13, 8, 5]
num_threads = 3
def sleep(a):
time.sleep(a)
return a
with ThreadPool(num_threads) as pool:
for r in pool.imap_unordered(sleep, tasks):
print(f'Sleeped for {r} seconds')