Saturday, May 03, 2014

A multiprocessing map function for python

Often one wishes for a simple way to speed up CPU intensive tasks in python. This version of python's map( .. ) function takes of process based parallelization and circumvents the restrictions of the global interpreter lock (GIL). One problem with using the multiprocessing Queue in python is that the submitted jobs are not processed in the submitted order. Therefore, before anything is returned, it is sorted by the input. Now just use mp_map instead of map and enjoy unrestricted speed :)

No comments: