Intelligent Cloud Operations

The need to deliver flawless digital experiences to customers by enterprises is more critical than ever in this pandemic world. To achieve this, companies are leveraging cloud native technologies such as microservices, container technologies such as Docker and orchestration technologies such as Kubernetes. To move faster and operate more efficiently, some customers are using serverless technologies such as AWS Fargate and AWS Lambda.

What is AWS Lambda?

According to AWS, Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers.

Lambda Use-cases

AWS highlights the following use-cases for Lambda:

• File processing: S3 triggers to lambda after an upload for data processing
• Stream processing: Process real-time streaming data for use-cases such as application activity tracking, transaction order processing, clickstream analysis, data cleansing, log filtering, indexing, social media analysis, IoT device data telemetry
• Web application: Leverage lambda and other services to build powerful web applications
• IoT backend: To handle web, mobile, Internet of Things (IoT), and third-party API requests
• Mobile backend: Use lambda and API Gateway to authenticate and process API requests

Challenges with Lambda

Lambda by its nature is event-driven and operate in isolation. The lack of control over the ephemeral infrastructure makes it difficult to debug. It can be very tedious and labor intensive if you are investigating an issue without sophisticated observability solution.

In a busy site, number of lambda executions may reach millions!

I have been working on lambda related use-cases with my team. I this blog I will explain how to setup observability for lambda so that your SRE/DevOps/support teams can manage the distributed microservices running on serverless infrastructure. My goal is to show how you may centralize all logs and metrics from lambda execution and visualize the data in a dashboard with a few simple steps.

Register

Picture 1: CloudAEye Sign up Page

First, sign up if you don't have an account yet. You may register with CloudAEye using your full name, work email address and company name. Once you have successfully registered you will be directed to a new page to set your password. You will enter the temporary password that you have received in email and set a new password. You may log in using your email and password and request for a free trial.

Serverless Application

To demonstrate the end-to-end use-case, I have deployed one of the AWS serverless samples -- shopping-cart. This example shows to how to implement a shopping cart microservice with serverless technologies on AWS. To follow along, you may use this or use your own serverless application.

Picture 2: AWS Sample - Shopping Cart

Setup Logs Service

Setup a logs service by entering a name, service groups and labels (refer to picture 3 below). This will take several minutes. After the logs service is created, add the application(s) you want to setup for logs streaming and follow the instructions. Refer to picture 4 & 5 below.

Picture 3: Create a logs service

Picture 4: Add an applications

Picture 5: View logs streaming instructions

Setup Logs Streaming

After setting up the logs service, follow the instructions to download the lambda agent (picture 6), install boto3 (one time activity, see picture 7) and set up the agent (picture 8). Boto3 is the AWS SDK for Python. It makes it convenient to setup the necessary AWS lambda layers for logs streaming. The agent uses OpenTelemetry collector. At this point, you are all set!

Picture 6: Download lambda agent

Picture 7: Install boto3

Picture 8: Set up agent

Visualize Lambda Data

You may visualize the logs and metrics from lambda by going to the dashboards. Click on the dashboard button in logs services. You will see that all the logs from lambda is centralized and searchable (see picture 9). You may also see the important KPIs and execution metrics from lambda functions (picture 10).

Picture 9: Lambda logs

Picture 10: Lambda dashboard

Do It Yourself (DIY)

If you want to understand how this works or do this yourself, refer to this blog that describes how AWS lambda extensions work. and the associated APIs. You will also need to setup backend logs services that can parse and store your logs (see this).

CloudAEye References

• The documentation for the CLI is available at docs.cloudaeye.com/cli-reference
• Product documentation for CloudAEye SaaS is available at docs.cloudaeye.com/index.html