A Serverless EC2 Inventory with the AWS CDK (part 3)

In this article, we will tidy up our code, put it all together and deploy our first AWS CDK stack. This article continues on from part 2 but the complete solution will be made available through GitHub.

Serverless Architecture
Serverless EC2 Inventory Stack

To be able to tie these components together we’ll need to add two key items:

  • Events
  • Lambda functions

Firstly, the events will be what triggers the various parts of our stack and the lambda functions will do the processing of data and inserting to our DynamoDB tables.

Events

We will start by adding a couple of additional imports, these will be required for both creating events as well as using them in conjunction with Lambda for event sources.

from aws_cdk.aws_events import Rule, EventPattern
from aws_cdk import aws_events_targets
from aws_cdk.aws_lambda import Function, Runtime, Code, StartingPosition
from aws_cdk.aws_lambda_event_sources import SqsEventSource, DynamoEventSource

Now that we have imported what we need we can add the CloudWatch event that will trigger our stack.

        # Cloudwatch Event
        event_ec2_change = Rule(
            self, "ec2_state_change",
            description="trigger on ec2 start, stop and terminate instances",
            event_pattern=EventPattern(
                source=["aws.ec2"],
                detail_type=["EC2 Instance State-change Notification"],
                detail={
                    "state": [
                        "running",
                        "stopped",
                        "terminated"]
                    }
                ),
            targets=[aws_events_targets.SqsQueue(state_change_sqs)]
        )

The event source will be the aws.ec2 API and the detail type we are looking for is “EC2 Instance State-change Notification”.

There are quite a lot of different states within the EC2 API so we will limit it to only the following changes:

  • “running”
  • “stopped”
  • “terminated”

The other states are all transitional and since our aim is to keep an inventory of our instances the states above will achieve our outcome.

Lambda Functions

The last piece of the puzzle we need are the lambda functions, there are two key pieces of functionality we need to process.

Event capture

Our event capture lambda function will take the CloudWatch event and record when an EC2 instance has changed to one of the states previously mentioned.

import os
import json
import boto3

try:
    dynamodb = boto3.resource('dynamodb')
    state_table = dynamodb.Table(os.getenv('state_table'))
except Exception as e:
    raise e

def handler(event, context):

    state_list = list()

    for e in event["Records"]:
        state_change = json.loads(e['body'])
        state_list.append(state_change)

    for instance in state_list:
        print(f"Adding state change for {instance['detail']['instance-id']} : {instance['detail']['state']}")
        state_table.put_item(
            Item={
                'instance-id': instance['detail']['instance-id'],
                'time': instance['time'],
                'state': instance['detail']['state']
            }
        )

As you can see this is a very straight forward function, it essentially receives a CloudWatch even and records three key pieces of information into a DynamoDB table which will be used by our event processor lambda.

The data recorded in the ec2_states table will give you a chronologically ordered sets of states for all instances in an account. On the other hand, the ec2_inventory table will store detailed information about an instance but it will only keep the most current details for the instance.

Event processor

In the event processor lambda function we are going to read any updates made to the ec2_states table. Updates will be passed to this function via a DynamoDB stream (event source).

import os
from datetime import datetime
import json
import boto3
from boto3.dynamodb.conditions import Key, Attr

try:
    dynamodb = boto3.resource('dynamodb')
    inventory_table = dynamodb.Table(os.getenv('inventory_table'))
except Exception as e:
    raise e

def handler(event, context):
    # boto3.resource('dynamodb')
    # Used to deserialize the DynamoDB stream
    deserializer = boto3.dynamodb.types.TypeDeserializer()

    for record in event["Records"]:
        instance = {k: deserializer.deserialize(v) for k, v in record['dynamodb']['NewImage'].items()}
        inventory(instance['instance-id'])


def inventory(instance_id: dict):
    ec2 = boto3.client('ec2')
    check_inventory = inventory_table.query(
        KeyConditionExpression=Key('instance-id').eq(instance_id))

    if check_inventory['Count'] != 0:
        print(f"Instance {instance_id} exists in inventory, deleting existing record")
        for i in check_inventory['Items']:
            inventory_table.delete_item(
                    Key={'instance-id': i['instance-id'],
                         'time': i['time'],
                    })

    inventory = ec2.describe_instances(
        InstanceIds=[instance_id])
    
    instance = inventory['Reservations'][0]['Instances'][0]
    # This is the inventory record, we will extract out some important fields and keep the whole
    # inventory data in the 'Inventory' field.
    inventory_table.put_item(
        Item={
                'instance-id': instance_id,
                'time': datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
                'State': instance['State']['Name'],
                'ImageId': instance['ImageId'],
                'InstanceType': instance['ImageId'],
                'KeyName': instance['KeyName'],
                'Tags': instance.get('Tags', 'None'),
                'VpcId': instance['VpcId'],
                'Inventory': str(instance),
            }
        )
    print(f"Record updated for {instance_id}") 

Finalising the CDK stack

The last three items we need to add to our CDK in order to complete our stack are the lambda deployments and the CloudWatch event.

        # event capture lambda
        lambda_event_capture = Function(
            self, "lambda_event_capture",
            handler="event_capture.handler",
            runtime=Runtime.PYTHON_3_7,
            code=Code.asset('event_capture'),
            role=rl_event_capture,
            events=[SqsEventSource(state_change_sqs)],
            environment={"state_table": tb_states.table_name}
        )

        # event processor lambda
        lambda_event_processor = Function(
            self, "lambda_event_processor",
            handler="event_processor.handler",
            runtime=Runtime.PYTHON_3_7,
            code=Code.asset('event_processor'),
            role=rl_event_processor,
            events=[
                DynamoEventSource(
                    tb_states,
                    starting_position=StartingPosition.LATEST)
            ],
            environment={
                "inventory_table": tb_inventory.table_name,
                }
        )

        # Cloudwatch Event
        event_ec2_change = Rule(
            self, "ec2_state_change",
            description="trigger on ec2 start, stop and terminate instances",
            event_pattern=EventPattern(
                source=["aws.ec2"],
                detail_type=["EC2 Instance State-change Notification"],
                detail={
                    "state": [
                        "running",
                        "stopped",
                        "terminated"]
                    }
                ),
            targets=[aws_events_targets.SqsQueue(state_change_sqs)]
        )

        # Outputs
        core.CfnOutput(self, "rl_state_capture_arn", value=rl_event_capture.role_arn)
        core.CfnOutput(self, "rl_state_processor_arn", value=rl_event_processor.role_arn)
        core.CfnOutput(self, "tb_states_arn", value=tb_states.table_arn)
        core.CfnOutput(self, "tb_inventory_arn", value=tb_inventory.table_arn)
        core.CfnOutput(self, "sqs_state_change", value=state_change_sqs.queue_arn)

For both of the lambda functions we pass the relevant DynamoDB table names as environment variables.

Lastly our CloudWatch event rule is sending its payload directly to the SQS queue we created in part 2.

Deploying the stack

We now have everything we need for our inventory stack, to deploy the stack run the following commands within your project directory.

(.env) $ cdk deploy
AWS CDK deployment

The deployment should take 5 – 10 minutes, at which point once it’s deployed you will have two newly created DynamoDB tables which will contain any state changes and inventory of EC2 instances.

The complete project source code is on the github page, https://github.com/letsfigureout/ec2inventory feel free to fork the repository and deploy your own stack.

I hope this has given you a good foundation for how to build an application in AWS using the AWS CDK.

If you have any questions or comments please add a question or submit a pull request via github.