Published on

Parallel Processing on AWS Lambda with Python using Multiprocessing


If you are trying to use multiprocessing.Queue or multiprocessing.Pool on AWS Lambda, you are probably getting the exception:

[Errno 38] Function not implemented: OSError

    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented

The reason for that is due to the Lambda execution environment not having support on shared memory for processes, therefore you can’t use multiprocessing.Queue or multiprocessing.Pool.

As a workaround, Lambda does support the usage of multiprocessing.Pipe instead of Queue.

Parallel Processing on Lambda Example

Below is a very basic example on how you would achieve the task of executing parallel processing on AWS Lambda for Python:

import time
import multiprocessing

region_maps = {
        "eu-west-1": {
        "us-east-1": {
        "us-east-2": {
            "dynamodb": ""

def multiprocessing_func(region):
    endpoint = region_maps[region]['dynamodb']
    print('endpoint for {} is {}'.format(region, endpoint))

def lambda_handler(event, context):
    starttime = time.time()
    processes = []
    regions = ['us-east-1', 'us-east-2', 'eu-west-1']
    for region in regions:
        p = multiprocessing.Process(target=multiprocessing_func, args=(region,))

    for process in processes:

    output = 'That took {} seconds'.format(time.time() - starttime)
    return output

The output when the function gets invoked:

pid: 30913 - endpoint for us-east-1 is
pid: 30914 - endpoint for us-east-2 is
pid: 30915 - endpoint for eu-west-1 is
That took 1.014902114868164 seconds

Thank You

Thanks for reading, feel free to check out my website, and subscrube to my newsletter or follow me at @ruanbekker on Twitter.