AWS Lambda function has a timeout of 900 seconds which sometimes is not enough to finish tasks that take a longer time. In such cases, recursion is a technique that can be applied to solve the problem, provided the single task can be broken into sub-tasks. I am not talking about the recursion provided by the language, but recursion by re-invocation of the same lambda function with additional state for continuity of the task.

This can be accomplished by utilizing the context object’s method get_remaining_time_in_millis. This method returns the milliseconds left before the Lambda function execution will time out.

To achieve recursion, we keep checking the time remaining in the loop while processing the items and make a recursive call when the time remaining is less than 10 seconds. Below Python code snippet illustrates the same:

for item in items:
if context.get_remaining_time_in_millis() < 10000:
_make_recursive_call(event, context, processed_id)
return
else:
_process_item(item)

After making the recursive call, it is important to end the currently executing function or else we will end up making multiple recursive calls which could provide unintended results.

When making the recursive lambda call, we can pass the state of the currently executing function so that the subsequent function can continue from that point onwards. Below code snippet illustrates the same:

def _make_recursive_call(event, context, processed_id):
event['CURRENT_STATE'] = processed_id
lambda_client = boto3.client('lambda')
lambda_client.invoke(FunctionName=context.function_name,
InvocationType='Event',
Payload=json.dumps(event))
return

The state does not have to be a simple id. It can be a complex JSON object as long as it is within the AWS Lambda limits which is 6 MB for the request payload.

It is important to handle the state correctly at the entry point of the lambda function as shown below to be able to continue the processing from the point where the previous invocation ended.

def lambda_handler(event, context):
processed_id = 0
if event.get('CURRENT_STATE') is not None:
processed_id = event['CURRENT_STATE']

There are a couple of things to keep in mind when using recursive Lambda functions

  1. Make sure that the end condition for recursion is properly tested. Otherwise, it could lead to an unintended volume of function invocations and escalated costs. If this accidentally happens, set the function concurrent execution limit to 0 to throttle all invocations to the function, before working on the fix.
  2. The role used for Lambda execution should have lambda:InvokeFunction permission to be able to make the recursive call successfully.
  3. AWS Lambda reuses the same container for subsequent runs. So, initialize SDK clients, database connections outside the function handler. Each container has 512 MB of storage available at mount point /tmp. This space can be used to store static assets that won’t change across executions.

This code sample shows a lambda function written in Python to process all objects in an S3 bucket recursively