API Reference

The loky module manages a pool of worker that can be re-used across time. It provides a robust and dynamic implementation os the ProcessPoolExecutor and a function get_reusable_executor() which hide the pool management under the hood.

loky.get_reusable_executor(max_workers=None, context=None, timeout=10, kill_workers=False, reuse='auto', job_reducers=None, result_reducers=None, initializer=None, initargs=(), env=None)

Return the current ReusableExectutor instance.

Start a new instance if it has not been started already or if the previous instance was left in a broken state.

If the previous instance does not have the requested number of workers, the executor is dynamically resized to adjust the number of workers prior to returning.

Reusing a singleton instance spares the overhead of starting new worker processes and importing common python packages each time.

max_workers controls the maximum number of tasks that can be running in parallel in worker processes. By default this is set to the number of CPUs on the host.

Setting timeout (in seconds) makes idle workers automatically shutdown so as to release system resources. New workers are respawn upon submission of new tasks so that max_workers are available to accept the newly submitted tasks. Setting timeout to around 100 times the time required to spawn new processes and import packages in them (on the order of 100ms) ensures that the overhead of spawning workers is negligible.

Setting kill_workers=True makes it possible to forcibly interrupt previously spawned jobs to get a new instance of the reusable executor with new constructor argument values.

The job_reducers and result_reducers are used to customize the pickling of tasks and results send to the executor.

When provided, the initializer is run first in newly spawned processes with argument initargs.

The environment variable in the child process are a copy of the values in the main process. One can provide a dict {ENV: VAL} where ENV and VAL are string literals to overwrite the environment variable ENV in the child processes to value VAL. The environment variables are set in the children before any module is loaded. This only works with with the loky context.

class loky.ProcessPoolExecutor(max_workers=None, job_reducers=None, result_reducers=None, timeout=None, context=None, initializer=None, initargs=(), env=None)

Initializes a new ProcessPoolExecutor instance.

Args:

max_workers: int, optional (default: cpu_count())

The maximum number of processes that can be used to execute the given calls. If None or not given then as many worker processes will be created as the number of CPUs the current process can use.

job_reducers, result_reducers: dict(type: reducer_func)

Custom reducer for pickling the jobs and the results from the Executor. If only job_reducers is provided, result_reducer will use the same reducers

timeout: int, optional (default: None)

Idle workers exit after timeout seconds. If a new job is submitted after the timeout, the executor will start enough new Python processes to make sure the pool of workers is full.

context: A multiprocessing context to launch the workers. This

object should provide SimpleQueue, Queue and Process.

initializer: An callable used to initialize worker processes. initargs: A tuple of arguments to pass to the initializer. env: A dict of environment variable to overwrite in the child

process. The environment variables are set before any module is loaded. Note that this only works with the loky context.

map(fn, *iterables, **kwargs)

Returns an iterator equivalent to map(fn, iter).

Args:

fn: A callable that will take as many arguments as there are

passed iterables.

timeout: The maximum number of seconds to wait. If None, then there

is no limit on the wait time.

chunksize: If greater than one, the iterables will be chopped into

chunks of size chunksize and submitted to the process pool. If set to one, the items in the list will be sent one at a time.

Returns:

An iterator equivalent to: map(func, *iterables) but the calls may be evaluated out-of-order.

Raises:

TimeoutError: If the entire result iterator could not be generated

before the given timeout.

Exception: If fn(*args) raises for any values.

shutdown(wait=True, kill_workers=False)

Clean-up the resources associated with the Executor.

It is safe to call this method several times. Otherwise, no other methods can be called after this one.

Args:

wait: If True then shutdown will not return until all running

futures have finished executing and the resources used by the executor have been reclaimed.

submit(fn, *args, **kwargs)

Submits a callable to be executed with the given arguments.

Schedules the callable to be executed as fn(*args, **kwargs) and returns a Future instance representing the execution of the callable.

Returns:

A Future representing the given call.

Task & results serialization

To share function definition across multiple python processes, it is necessary to rely on a serialization protocol. The standard protocol in python is pickle but its default implementation in the standard library has several limitations. For instance, it cannot serialize functions which are defined interactively or in the __main__ module.

To avoid this limitation, loky relies on cloudpickle when it is present. cloudpickle is a fork of the pickle protocol which allows the serialization of a greater number of objects and it can be installed using pip install cloudpickle. As this library is slower than the pickle module in the standard library, by default, loky uses it only to serialize objects which are detected to be in the __main__ module.

There are three ways to temper with the serialization in loky:

• Using the arguments job_reducers and result_reducers, it is possible to register custom reducers for the serialization process.

• Setting the variable LOKY_PICKLER to an available and valid serialization module. This module must present a valid Pickler object. Setting the environment variable LOKY_PICKLER=cloudpickle will force loky to serialize everything with cloudpickle instead of just serializing the object detected to be in the __main__ module.

• Finally, it is possible to wrap an unpicklable object using the loky.wrap_non_picklable_objects decorator. In this case, all other objects are serialized as in the default behavior and the wrapped object is pickled through cloudpickle.

The benefits and drawbacks of each method are highlighted in this example.

loky.wrap_non_picklable_objects(obj, keep_wrapper=True)

Wrapper for non-picklable object to use cloudpickle to serialize them.

Note that this wrapper tends to slow down the serialization process as it is done with cloudpickle which is typically slower compared to pickle. The proper way to solve serialization issues is to avoid defining functions and objects in the main scripts and to implement __reduce__ functions for complex classes.

Processes start methods in loky

The API in loky provides a set_start_method() function to set the default start_method, which controls the way Process are started. The available methods are {'loky', 'loky_int_main', 'spawn'}. On unix, the start methods {'fork', 'forkserver'} are also available. Note that loky is not compatible with multiprocessing.set_start_method() function. The default start method needs to be set with the provided function to ensure a proper behavior.

Protection against memory leaks

The memory size of long running worker processes can increase indefinitely if a memory leak is created. This can result in processes being shut down by the OS if those leaks are not resolved. To prevent it, loky provides leak detection, memory cleanups, and workers shutdown.

If psutil is installed, each worker periodically [1] checks its memory usage after it completes its task. If the usage is found to be unusual [2], an additional gc.collect() event is triggered to remove objects with potential cyclic references. If even after that, the memory usage of a process worker remains too high, it will shut down safely, and a fresh process will be automatically spawned by the executor.

If psutil is not installed, there is no easy way to monitor worker processes memory usage. gc.collect() events will still be called periodially [1] inside each workers, but there is no guarantee that a leak is not happening.

Footnotes

[1] (1,2)

every 1 second. This constant is defined in loky.process_executor._MEMORY_LEAK_CHECK_DELAY

[2]

an increase of 300MB compared to a reference, which is defined as the residual memory usage of the worker after it completed its first task. This constant is defined in loky.process_executor._MAX_MEMORY_LEAK_SIZE