Building Modern Applications with AWS CDK

This article is concise material from “AWS Dev Hour”

Introduction

This write-up is for anyone looking to start learning about AWS Cloud Development Kit (CDK) and how to create components like s3, dynamoDB and lambda, deploy these and use them in an app.

What we’ll be building with CDK

• An app to let users upload a photo through a react JS UI, stored in s3 (upcoming sessions).
• It will then trigger a lambda upon the s3 action and store it in dynamoDB (which stores metadata and labels, done this session)
• We can then query(scan) the db to check labels for the image have been added after going through rekognition service.

Background

AWS host regular sessions on how to use their services— this post is based on a recent one https://www.twitch.tv/videos/892197005. The video lasts just over an hour, today we’ll go through the main notes, as well as fixes and ways to avoid certain pitfalls I went into and has a separate working GitHub you can clone:

DarrenBro/modernApp-AWS-CDK

There were however some issues I came across not covered by the video that I’ve included fixes for or tasks you should complete on your end.

1. Node Version 15+ had issues with deploying lambda assets https://github.com/aws/aws-cdk/issues/12536
   Fix ->Downgrading node and deleting cdk.out/.cache should stop this issue from occurring. https://stackoverflow.com/questions/8191459/how-do-i-update-node-js
2. If you continue without completing steps not shown in the video for getting pillow(PIL) into your directory (steps shown below title Pillow) you’ll get the following CloudWatch log error;
   Fix -> I’ve included the necessary extracted zip file and put it in the project under the folder “reklayer”.
3. The video misses out showing the creation of the lambda layer. This allows the lambda to import ‘PIL’ from your attached zip file you created.
   Fix -> I’ve attached code snippets that need included in cdkMainStacks.ts.
   Otherwise you’ll get the below error in your CloudWatch logs.

‘index’ is referring to the main body of the rekognition lambda

I’ve flooded cdkMainStacks.ts with comments better explaining different components. This file is the main area that most of the code will get added.

For me this has been a great way to get more hands on with CDK with good progression intervals and building it out yourself is the best way to get familiar with any tech, plus finding solutions for these errors has helped commit it more to memory.

Tech Stack

1. aws-cli
   https://docs.aws.amazon.com/cli/latest/userguide/install-macos.html
2. Node.js to run the project (Version 10.3.0–14.15.3)
   https://nodejs.org/en/download/
3. IDE that can run any of these languages;
   Typescript, Javascript, Java, C# or Python
   (I’m using IntelliJ with Typescript)
4. AWS CDK toolkit and bootstrap

All graphics are taken from the AWS run through video.

Session 1 will consist of;

1. Installing prerequisites
2. Explaining what AWS CDK is
3. Creating a new project and getting dependencies installed
4. Building 3 components -> s3 / lambda & dynamo db
5. Copying an image to scan and scanning the table

What is AWS CDK?

• A framework we can use to declare our AWS resources
• Sits on top of Cloud Formation
  (CDK produces the CFT Template after synthesise “cdk synth”)
• Simplifies the process of building out these resource templates

This is what an application looks like in the CDK, starting with the App, and broken into stacks and constructs(which is what will be built later on).

Steps to get started;

1. “mkdir cdk-project”
2. “cdk init” (shows templates available)
3. “cdk init app -l=typescript” (Go with typescript)
4. Install aws-iam “npm i @aws-cdk/aws-iam”

Notes

• app initialised in cdk-projects.ts
• cdk.json tells cdk toolkit how to run the project

Pillow

The AWS Lambda function uses the Pillow library for the generation of thumbnail images. This library needs to be added into our project so that we can allow the CDK to package it and create an AWS Lambda Layer for us. To do this, you can use the following steps.

1. Launch an Amazon EC2 Instance (t2-micro) using the Amazon Linux 2 AMI
2. SSH into your instance and run the following commands:

sudo yum install -y python3-pip python3 python3-setuptools
python3 -m venv my_app/env
source ~/my_app/env/bin/activate
cd my_app/env
pip3 install pillow
cd /home/ec2-user/my_app/env/lib/python3.7/site-packages
mkdir python && cp -R ./PIL ./python && cp -R ./Pillow-8.1.0.dist-info ./python && cp -R ./Pillow.libs ./python && zip -r pillow.zip ./python

1. Copy the resulting archive ‘pillow.zip’ to your development environment (we used an Amazon S3 bucket for this)
2. Extract the archive into the ‘reklayer’ folder in your project directory

Your project structure should look something like this:

project-root/reklayer/python/PIL
project-root/reklayer/python/Pillow-8.1.0.dist-info
project-root/reklayer/python/Pillow.libs

1. Remove the python.zip file to clean up
2. Terminate the Amazon EC2 Instance that you created to build the archive

1st component to build is s3

npm i @aws-cdk/aws-s3

In cdkMainStack.ts we’ll be adding;

import s3 = require(‘@aws-cdk/aws-s3’);
const imageBucket = new s3.Bucket(this, imageBucketName, {
  removalPolicy: cdk.RemovalPolicy.DESTROY
})
new cdk.CfnOutput(this, 'imageBucket', {value: imageBucket.bucketName});

A quick note on the constructs here.

So for s3 code above it’s the same principle.

2nd component to build is dynamoDB for storing the image labels

npm i @aws-cdk/aws-dynamodb
const imageTable = new dynamodb.Table(this, 'ImageLabels', {
  partitionKey: {name: 'image', type: dynamodb.AttributeType.STRING},
  removalPolicy: cdk.RemovalPolicy.DESTROY
});
new cdk.CfnOutput(this, 'cdkTable', {value: imageTable.tableName});

3rd component to build is Lambda layer and Lambda

Layer needed to import the PIL library when rekognition lambda is executed.

const layer = new lambda.LayerVersion(this, 'pil', {
  code: lambda.Code.fromAsset('reklayer'),
  compatibleRuntimes: [lambda.Runtime.PYTHON_3_7],
  license: 'Apache-2.0',
  description: 'A layer to enable the PIL library in our Rekognition Lambda',
});

Lambda’s job is to pull an image from the s3 bucket, take the image and send it to the rekognition service to do the image detection.

npm i @aws-cdk/aws-lambda @aws-cdk/aws-lambda-event-sources
const rekognitionLambdaFunc = new lambda.Function(this, 'rekognitionFunction', {
  code: lambda.Code.fromAsset('rekognitionlambda'),
  runtime: lambda.Runtime.PYTHON_3_7,
  handler: 'index.handler',
  timeout: Duration.seconds(30),
  memorySize: 1024,
  layers: [layer],
  environment: {
    "TABLE": imageTable.tableName,
    "BUCKET": imageBucket.bucketName,
    "RESIZEDBUCKET": resizedBucket.bucketName
  }
});

Last part to add is Permissions and Roles

To trigger Lambda when object(image) is created in S3.

rekognitionLambdaFunc.addEventSource(new event_sources.S3EventSource(imageBucket, {events: [s3.EventType.OBJECT_CREATED]}))

Permission to read from s3.

imageBucket.grantRead(rekognitionLambdaFunc);
resizedBucket.grantPut(rekognitionLambdaFunc);

Permission to allow the result of rekognition service from the sent image to be stored in dynamoDB.

imageTable.grantWriteData(rekognitionLambdaFunc);

Permission policy to allow label detection from rekognition across all resources.

rekognitionLambdaFunc.addToRolePolicy(new iam.PolicyStatement({
  effect: iam.Effect.ALLOW,
  // permission policy to allow label detection from rekognition across all resources
  actions: ['rekognition:DetectLabels'],
  resources: ['*']
}))

And make sure you’ve updated your bucket name to the same as the stack. (Would recommend changing prefix to your name, but stack with add a unique suffix ID value in template after synthesising).

To create your template for your stacks run:

cdk synth
A set of files in a folder called "cdk.out" will be produced with assets and your json template file.
You can re-run cdk synth to populate any code updates.

synth will manufacture a stack (our components) down into fully formed json yaml template (CloudFormation template) which will include all the resources we added above.

When you see the generated template it will already be quite large. Already cdk has saved us a bunch of effort!

The diagram below helps show what we’ve done in a smaller scale.

To deploy the resources that we just created run the below command.

cdk deploy

After seeing the below you just want to hit ‘y’

This should take around a few minutes to complete.

Result of “cdk deploy” -> ✅ cdkMainStack

Output example

cdkMainStack.cdkTable = cdkMainStack-ImageLabelsE524135D-104WIEO86Q2JP
cdkMainStack.imageBucket = cdkmainstack-dbrocdkimagebucketb661dc68-uok6ax6q62sh

You can also view your CF stacks in AWS console straight away, pretty nice!

So that’s your s3, lambda and dynamo db all created for you.

A quick note on the lambda. For the clients below:

As this lambda is being invoked we don’t have to re-instantiate these above clients every time the lambda is run, it can stay ‘hot’ and this helps with performance.

Test the image upload (now that s3 resources have been created using cdk deploy)

You’ll need to use your unique bucket name, e.g.

aws s3 cp testimage.jpg s3://cdkmainstack-dbrocdkimagebucketb661dc68-uok6ax6q62sh

Logs

Check CloudWatch logs for events and for any errors

In log group “/aws/lambda/cdkMainStack……” -> click latest log stream to see the image being processed.

Or you can check your new dynamoDB table in the console.

These labels are from an image of Stargate’s Atlantis’ wormhole after it was sent through AWS Rekognition

Or you can scan the dynamoDB table, using the output name from the cdk synth. e.g.

aws dynamodb scan --table-name cdkMainStack-ImageLabelsE524135D-104WIEO86Q2JP

And you should see a single output of your image in the terminal

To get rid of all resources and stop any future running costs run:

cdk destroy
(which is also why we added to dynamoDB)
removalPolicy: cdk.RemovalPolicy.DESTROY

However, even with this policy added to S3 and using cdk destroy the formation will have a status “DELETE_FAILED”

Without the policy added you will see that the CloudFormation stack is deleted successfully but the S3 bucket remains. Why?

By default, the Construct that comes from the S3 package, has a default prop called removalPolicy: cdk.RemovalPolicy.RETAIN.

(CloudFormation does not destroy buckets that are not empty).

Option 1: Manually clean the bucket contents before destroying the stack

You can do this from the AWS S3 user interface or through the command line, using the AWS CLI:

# Cleanup bucket contents without removing the bucket itself;
aws s3 rm s3://bucket-name --recursive
# Then run;
cdk destroy

Then the cdk destroy will proceed without errors. However, this can quickly become a tedious activity if your stacks contain multiple S3 buckets or you use stacks as a temporary resource so some automation would help (option 2).

Option 2: Automatically clear bucket contents and delete the bucket

A 3rd party package called @mobileposse/auto-delete-bucket provides a custom CDK construct that wraps around the standard S3 construct and internally uses the CloudFormation Custom Resources framework, to trigger an automated bucket contents cleanup when a stack destroy is triggered.

Install the package:

npm i @mobileposse/auto-delete-bucket

Use the new CDK construct instead of the standard one:

import { AutoDeleteBucket } from '@mobileposse/auto-delete-bucket'
const bucket = new AutoDeleteBucket(this, 'my-data-bucket')

Summary

# Add components / build / deploy
cdk synth
cdk deploy
(If you makes any changes, you can run another 'cdk deploy')
# Take note of outputs (example)
cdkMainStack.cdkTable = cdkMainStack-ImageLabels
cdkMainStack.imageBucket = cdkmainstack-dbrocdkimagebucket
# Copy test image
aws s3 cp testimage.jpg s3://{bucketName}
# Check table has been populated, if not check logs
aws dynamodb scan --table-name {tableName}
# Cleanup tasks
aws s3 rm s3://{bucketName} --recursive
cdk destroy

Some helpful reading links;

Getting started with the AWS CDK

Bootstrapping

Amazon Rekognition - get started - AWS

That’s all for this session, take care!