theCodingInterface

Keeping Python AWS Serverless Apps DRY with Lambda Layers

By Adam McQuistan in Serverless 02/11/2021 Comment

Introduction

The beauty of building serverless applications using Lambda is you don't have to manage compute infrastructure or the associated headaches of keeping servers secure, up-to-date, or scaling resources to meet fluctuating demands. However, with that architecture comes trade offs in your ability to design modular, well organized, reusable code familiar to traditional server based development. In this article I demonstrate how utilize Lambda Layers to share reusable code among multiple Python AWS Lambda functions within AWS Serverless Application Model (SAM) applications.

To facilitate demonstrating this functionality I'll be using a silly Serverless application that exposes a REST API with three endpoints:

/hello/ which returns "Hello World!"
/hello/{other}/ returns "Hello other!" where other is a dynamic url parameter
/quotes/ reaches out to a webpage that generates random quotes, parses the author and text and returns them in a response

I also create a Python based library that gets packaged up in a Lambda Layer and deployed with the SAM CLI. This Lambda Layer then is referenced by the Lambda functions in the REST API application providing them access to the custom reusable library.

Creating, Packaging, and Deploying Reusable Python Library with Lambda Layers and SAM

To start I initialize a SAM Project for the shared code Lambda Layer using the Hello World template app with the Python3.8 runtime configured to work with Zip packaging like so.

sam init --name sam-awspyutils \
  --runtime python3.8 \
  --package-type Zip \
  --app-template hello-world

This produces the following project structure.

sam-awspyutils
├── README.md
├── __init__.py
├── events
│   └── event.json
├── hello_world
│   ├── __init__.py
│   ├── app.py
│   └── requirements.txt
├── template.yaml
└── tests
    ├── __init__.py
    ├── integration
    │   ├── __init__.py
    │   └── test_api_gateway.py
    ├── requirements.txt
    └── unit
        ├── __init__.py
        └── test_handler.py

Next I remove some of the boiler plate project files that I will not need.

cd sam-awspyutils
rm -rf hello_world
rm -rf tests
rm -rf events

Then I create a new src directory to hold the Python library (aka package) source code and within the src directory I create a Python package structure named awspyutils which contains \_\_init\_\_.py, greeter.py, and quotes.py files. I additionally add a requirements.txt file in the root of the src directory.

mkdir -p src/awspyutils
touch src/awspyutils/__ini__.py src/awspyutils/greeter.py src/awspyutils/quotes.py
touch requirements.txt

This leaves the new src directory looking as follows.

src
└── awspyutils
    ├── __ini__.py
    ├── greeter.py
    └── quotes.py

Then inside the greeter.py file I create a function named make_statement like so.

# greeter.py

def make_statement(value='World'):
    return "Hello %s!" % value

Now over in the requirements.txt file I add dependencies for the Requests and BeautifulSoup Python packages.

requests>=2.25.1,<2.26
beautifulsoup4>=4.9.3,<4.10

Next I add another new function named fetch_quote but this time to the quotes.py file.

# quotes.py

import requests

from bs4 import BeautifulSoup


def fetch_quote():
    response = requests.get('http://quotes.toscrape.com/random')
    soup = BeautifulSoup(response.text)
    quote_element = soup.find(class_='quote')
    author  = quote_element.find(class_='author').get_text()
    text = quote_element.find(class_='text').get_text()\
                        .replace('“', '').replace('”', '')
    return { 'author': author, 'text': text }

Alright at this point the awspyutils package is complete and I can update the SAM/CloudFormation template.yaml file to map this awspyutils code to a Lambda Layer and Serverless LayerVersion resource which is what I'm showing below.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-awspyutils

  Sample SAM Template for sam-awspyutils

Resources:
  AWSAlchemyLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: AWSAlchemy
      Description: Python based reusable code for AWS Severless Apps
      ContentUri: src/
      CompatibleRuntimes:
        - python3.7
        - python3.8
      LicenseInfo: MIT
      RetentionPolicy: Retain

Outputs:
  AWSAlchemyLayer:
    Value: !Ref AWSAlchemyLayer
    Description: ARN For AWS PyUtils Layer

Now back in the project root directory along side the above template.yaml I create a build.sh unix script which will help me to package this Lambda Layer SAM application.

#!/bin/bash -e

if [ -d src/python ]; then
	rm -rf src/python
fi

pip3 install --target src/python -r src/requirements.txt

cp -R src/awspyutils src/python

sam build

In order to understand why the build.sh script is doing what it is doing I should explain that when the sam build command runs in a Python based SAM Lambda Layer project such as this one if it sees a folder named python it knows to package that up, along with any source code and libraries within it, into a zip archive and deploy it with the Layer. So each time the build script runs it removes the existing source within the python directory, uses pip to install the requirements specified in the requirements.txt file, then copies the awspyutils package in there as well. Finally the script calls sam build. For more details checkout the SAM Layers Build specification on GitHub.

Ok at this point I can run the build.sh script (don't forget to make it executable) and follow that with the sam deploy command like so.

./deploy.sh
sam deploy --guided

This will output the ARN for the Lambda Layer version which I'll use in the next section when I deploy the SAM REST API.

Creating the Initial Python Serverless REST API

I initialize another new project with the Hello World app template.

sam init --name sam-rest-api \
  --runtime python3.8 \
  --package-type Zip \
  --app-template hello-world

This produces the following boilerplate application file structure.

sam-rest-api
├── README.md
├── __init__.py
├── events
│   └── event.json
├── hello_world
│   ├── __init__.py
│   ├── app.py
│   └── requirements.txt
├── template.yaml
└── tests
    ├── __init__.py
    ├── integration
    │   ├── __init__.py
    │   └── test_api_gateway.py
    ├── requirements.txt
    └── unit
        ├── __init__.py
        └── test_handler.py

I remove the files I don't need such as the entire tests, events, and hello_world directories.

cd sam-rest-api
rm -rf events
rm -rf tests
rm -rf hello_world

Next I create a src directory with three additional subdirectories in it named hello-world, hello-other, and quotes. Within each of the hello-world, hello-other, and quotes directories I also add api.py and requirements.txt files.

mkdir -p src/hello-world
mkdir src/hello-other
mkdir src/quotes
touch src/hello-world/api.py src/hello-world/requirements.txt
touch src/hello-other/api.py src/hello-other/requirements.txt
touch src/quotes/api.py src/quotes/requirements.txt

This leaves me with a project structure that looks like what is shown below.

sam-rest-api
├── README.md
├── __init__.py
├── src
│   ├── hello-other
│   │   ├── api.py
│   │   └── requirements.txt
│   ├── hello-world
│   │   ├── api.py
│   │   └── requirements.txt
│   └── quotes
│       ├── api.py
│       └── requirements.txt
└── template.yaml

Before implementing the code for each lambda directory I will first define the SAM / CloudFormation template.yaml file specifying that each of these three functions should use the Lambda Layer ARN that was generated in the first project. I source the Lambda Layer Arn as a Parameter to the template named AWSPyUtilsLayerArn.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-rest-api

  Sample SAM Template for sam-rest-api

Globals:
  Function:
    Timeout: 10

Parameters:
  AWSPyUtilsLayerArn:
    Type: String
    Description: The ARN for the AWS Python Utils Layer

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/hello-world/
      Handler: api.lambda_handler
      Runtime: python3.8
      Layers:
        - !Ref AWSPyUtilsLayerArn
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello/
            Method: get

  HelloWorldFunctionLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName:
        Fn::Join:
          - ''
          - - /aws/lambda/
            - Ref: HelloWorldFunction
      RetentionInDays: 3

  HelloOtherFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/hello-other/
      Handler: api.lambda_handler
      Runtime: python3.8
      Layers:
        - !Ref AWSPyUtilsLayerArn
      Events:
        HelloEchoer:
          Type: Api
          Properties:
            Path: /hello/{other}/
            Method: get

  HelloOtherFunctionLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName:
        Fn::Join:
          - ''
          - - /aws/lambda/
            - Ref: HelloOtherFunction
      RetentionInDays: 3

  QuoteFetcherFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/quotes/
      Handler: api.lambda_handler
      Runtime: python3.8
      Layers:
        - !Ref AWSPyUtilsLayerArn
      Events:
        QuotesApi:
          Type: Api
          Properties:
            Path: /quotes/
            Method: get

  QuoteFetcherFunctionLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName:
        Fn::Join:
          - ''
          - - /aws/lambda/
            - Ref: QuoteFetcherFunction
      RetentionInDays: 3

Outputs:
  HelloApi:
    Description: "API Gateway endpoint URL for Prod stage for HelloWorld and HelloOther function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"

  QuotesApi:
    Description: "API Gateway endpoint URL for Prod stage Quotes function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/quotes/"

Now on to the application code.

I start with the hello-world function by placing the following code within the api.py file. Notice that it sources the make_statement function from the awspyutils package that was specified in the first Lambda Layer project.

# hello-world/api.py

from awspyutils.greeter import make_statement


def lambda_handler(event, context):
    stmt = make_statement()
    return {
      'statusCode': 200,
      'body': stmt
    }

Next for the hello-other lambda function implemention which only differs in that it expects to receive a dynamic query path parameter accessible from the request event. The "other" path parameter is feed into the make_statement function imported and shared by the Lambda Layer.

# hello-other/api.py

from awspyutils.greeter import make_statement


def lambda_handler(event, context):
    value = event['pathParameters']['other']
    stmt = make_statement(value=value)
    return {
      'statusCode': 200,
      'body': stmt
    }

Lastly, the quotes function is implemented as shown below. This imports the fetch_quote function from the awspyutils package shared via the Lambda Layer, calls the function to pull a quote from the web and returns the reponse back through the AWS API Gateway.

# quotes/api.py

import json

from awspyutils import quotes

def lambda_handler(event, context):
    quote = quotes.fetch_quote()
    return {
      'statusCode': 200,
      'body': json.dumps(quote)
    }

At this point all that remains is to build, deploy, and test.

During the build step I need to pass in a parameter override with the Lambda Layer ARN produced in the first project as shown below (be sure to replace the LAMBDA-LAYER-ARN with yours). Similarly, I must give the same Lambda Layer Arn while issuing the sam deploy command.

sam build --parameter-overrides AWSPyUtilsLayerArn=LAMBDA-LAYER-ARN
sam deploy --guided

The output from the deploy command will list two urls, one for the HelloWorldFunction as well as the HelloOtherFunction then, a second for the QuotesFunction.

I can test them using an HTTP client such as HTTPie which I demo below with the quotes REST API endpoint generated for my project.

$ http -j https://k4xftb7hxe.execute-api.us-east-1.amazonaws.com/Prod/quotes/
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 121
Content-Type: application/json
Date: Thu, 11 Feb 2021 03:57:22 GMT
Via: 1.1 d643c18c094f3cd17f1bf4efe422c295.cloudfront.net (CloudFront)
X-Amz-Cf-Id: hTPbT8IpTwzLdvGPWFAC9AVuTcrrML2ElB1Twq649SEVm1tU3ITcMQ==
X-Amz-Cf-Pop: SFO20-C1
X-Amzn-Trace-Id: Root=1-6024ab22-42a9a14a634ecb60021bc1c6;Sampled=0
X-Cache: Miss from cloudfront
x-amz-apigw-id: aj-tcGd-oAMFZFQ=
x-amzn-RequestId: e32ada05-7889-47dc-8157-934427e0f049

{
    "author": "André Gide",
    "text": "It is better to be hated for what you are than to be loved for what you are not."
}

Resources to Learn More About Serverless Architecture and Computing

For more thorough coverage of using Python with AWS SAM please checkout my course Ultimate Guide to Deploying Flask to AWS which presents how to deploy Flask using both Traditiona EC2 based Server and AWS SAM Serverless Architectures and Methods
One of the best practical guide to the AWS Serverless Application Model is Gojko Adzic's book Running Serverless
Serverless isn't just for APIs and Microsites, it can be a very power component of modern Data Engineering Pipelines as demonstrated in Serverless Conversions From GZip to Parquet Format with Python AWS Lambda and S3 Uploads
Deployments in a Serverless World can be difficult to wrap your head around, luckily thecodinginterface.com has covered this topic so have a read of Using Lambda Versions, Aliases and Safe Deployments with AWS Serverless Application Model

thecodinginterface.com earns commision from sales of linked products such as the book suggestions above. This enables providing high quality, frequent, and most importantly FREE tutorials and content for readers interested in Software Engineering so, thank you for supporting the authors of these resources as well as thecodinginterface.com

Conclusion

In this article I have demonstrated how to use Lambda Layers to package and share reusable code, which itself has other third party dependencies, for an AWS Serverless Application Model project built with the Python programming language.

As always, thanks for reading and please do not hesitate to critique or comment below.

AWS Serverless Application Model AWS AWS-Lambda BeautifulSoup requests DevOps

Share with friends and colleagues

[[ likes ]] likes