• Netenrich
  • /
  • ...
  • /
  • Serverless Framework for Log Analysis: A Step by Step Guide
Subscribe To Our Newsletter!

Stay up to date on the top trending threats as well as the top stories in Security, Networks, Cloud, IT Ops & AIOps.

Serverless Framework for Log Analysis: A Step by Step Guide

Log analysis is used to collect, index, and store massive amounts of data from any source deployed in the cloud.

Tanuj Mitra
Post by Tanuj Mitra Apr 14, 2019


In the modern world, business applications continue to evolve; the log data generated becomes enormous. Log analytics tools help in extracting the data as desired. The analysis will also help in deriving the metrics about an application and its performance over a period.

Log Analysis is used to collect, index, and store massive amounts of data from any source deployed in the cloud. Since each log file includes audit information, and we can use the dashboards to analyze the collected log data and compare results specific to business needs.

Further, log analytics tools can help in identifying the root cause of an issue and consequently give the admins a chance to prevent such issues from occurring in the future. When a problem occurs, the critical concerns are:

  1. Identifying the log file which contains the issue
  2. Locating the server
  3. Searching for the data (e.g., timestamp, version, etc.)


  • Amazon Simple Server Storage (S3) is a storage service to store and retrieve any amount of data.
  • Amazon Athena is a query service to analyze data directly from files stored in S3 using standard SQL statements.
  • Amazon QuickSight helps build interactive visualizations, perform ad-hoc analysis, and get useful business insights from various data sources hosted on the AWS infrastructure.


The following are the steps for building the solution for log analytics on AWS.

Building the solution for log analytics on aws


The logs generated are uploaded to S3 for further processing. Create an S3 bucket in your AWS account.


Athena is used to analyze the data by querying the source datasets.

Open the AWS Management Console and type ‘Athena’ in the AWS Services search box. Once you find Athena, click on ‘Get Started.’

Amazon Athena get started

Using the Query Editor, run the command CREATE DATABASE to create a new database. You can save the command by clicking the ‘Save as’ option for future use.

Amazon Athena create database

Once the query is executed, the new database will appear in the drop-down menu on the left side of your screen. Now select the database that you created.

Amazon Athena query editor

Create a new Table for the files in S3 as below:

Amazon Athena files in s3

Once you create the table, verify it by browsing for the table on the left-side panel.


To load all partitions of the table, run the command – MSCK REPAIR TABLE. After creating the table, you can run various queries to investigate your logs.
For, E.g., select * from

Athena QuickSight

After receiving confirmation on data access via Athena, the next step is to visualize the data using QuickSight.


Select ‘Quicksight’ from the AWS search bar.

Sign up for QuickSight

Select the relevant QuickSight edition based on your requirement. Here in this example, we will go with the standard version for the demo.

Create your QuickSight account

After selecting the QuickSight edition, click on ‘Continue.’ You will then be directed to the web page shown below. Fill the necessary details and click ‘Finish.’

QuickSight testing

After creating the QuickSight account from the QuickSight home page, click ‘Manage data.’

QuickSight manage data

Select the ‘New data set’ option as below.

QuickSight new data set

Now select the ‘Athena’ option from Data Sets.

Athena from data sets

For the data source name, enter the same name as the ‘Athena’ database and click ‘Validate’ to connect QuickSight to Athena. After validation, click ‘Create data source.’

New Athena data source

Select the database and table from the following window. Click on ‘Edit/preview data.’

Athena edit preview

Here you can change the following variables as below:

  1. Data type‘ of the data field
  2. Rename‘ the data field
  3. Exclude‘ a data field if you don’t need it

Athena data type

After completing the changes, click ‘Save and Visualize.’ You can now view the QuickSight dashboard, as depicted in the below diagram. Here, you can create your dashboard by adding visuals.

Save and visualize QuickSight

Choose Add on the application bar, and then choose Add visual. Select the fields to use from the Fields list, pane at left. Then create a visual by selecting a visual type.

You can also customize the visuals per your requirements.

  1. Creating
  2. Renaming
  3. Changing fields
  4. Changing visual layout

VPC flow logs

This is how we can leverage the AWS services to process, analyze, and visualize the logs generated from different sources of log data. If you want to know more about effective cloud management, visit our page today.

Tanuj Mitra

About the Author

Tanuj Mitra

Tanuj is a storyteller whose ideas and snackable insights are in-sync with dynamic IT operations and modern networks. He likes to develop content that's smartly worded, clutter-breaking, and easy to digest.

Subscribe To Our Newsletter!

The best source of information for Security, Networks, Cloud, and ITOps best practices. Join us.

Thank you for subscribing!

Related Post

Aug 12 2020

7 Ways AIOps Is Redefining Hybrid Cloud Management

AI and automation to alleviate modern cloud concer...

Read More
Dec 06 2018

5 Reasons to Run Open Source Software on the Microsoft Azure

Don’t let history scare you away from taking advan...

Read More