Create a continuous integration pipeline using Jenkins, GitLab/Github and deploy on AWS EC2 with Email notification after successful/failed deployment

Executive Summary :
As part of the routine practises, the CloudFormation, Jenkins on AWS EC2, Github and AWS Code Deploy is routine combination when it comes to Code pipeline and Continuous Delivery.
But this article is with slightly different blend of local Jenkins/Jenkins on any VM, GitLab/Github and AWS Code Deploy along with email distribution after successful or failure of application build.
Just to give you heads up, AWS Code Deploy is a developer tool that AWS offer which automates the code deployment on Amazon EC2 instances.

The first part of this workshop is here

To actively practise this  workshop below are the pre requisites.

Pre-requisites :
1. Jenkins must be installed on your local machine or remote machine (VM)
2. You must have basic knowledge about how to use Jenkins and its different plugins
3. A GitLab/Github account
4. A gmail account which will be utilised for post build notification confirmation to the team about build status
5. Most important is to define the IAM roles for AWS services

We need to have two IAM roles: a Service Role and an Instance Profile Role.
Service Role: Service Role access will be granted to AWS CodeDeploy to read through your EC2 instance tags. For auto-scaled instances, it looks for your autoscaling name tags. Service Role Policy is as following:

{
"Version": "2012-10-17",
"Statement": [
   {
     "Action": [
       "autoscaling:PutLifecycleHook",
       "autoscaling:DeleteLifecycleHook",
       "autoscaling:RecordLifecycleActionHeartbeat",
       "autoscaling:CompleteLifecycleAction",
       "autoscaling:DescribeAutoscalingGroups",
       "autoscaling:PutInstanceInStandby",
       "autoscaling:PutInstanceInService",
       "ec2:Describe*"
     ],
     "Effect": "Allow",
     "Resource": "*"
   }
]
}

Name this role as CodeAccessRole

Once the IAM role is created, you need to establish a trusted relationship for this role with AWSCodeDeploy. Modify trust relationship policy for this particular role and set it to below mentioned policy :

{
"Version": "2012-10-17",
"Statement": [
    {
    "Sid": "",
    "Effect": "Allow",
    "Principal": {
    "Service": [
        "codedeploy.us-east-1.amazonaws.com",
        "codedeploy.us-west-2.amazonaws.com"
    ]
    },
    "Action": "sts:AssumeRole"
    }
]
}

Instance Profile Role: EC2 instances needs to be launched with proper permissions to access files from S3 buckets.

Name this role as S3CodeDepRole

{
    "Version": "2012-10-17",
    "Statement": [
    {
        "Action": [
            "s3:Get*",
            "s3:List*"
        ],
        "Effect": "Allow",
        "Resource": "*"
    }
    ]
}

Condition :

For now AWS supports only 2 regions for AWS CodeDeploy service which are 1. N.Virginia, 2.Oregon

Just to give you an insight, we are going to utilise the AWS' CodeDeploy Dev Tool to deploy our application and to enable that workflow below are the necessary steps to be executed.


Step 1 : Sign up for free tier AWS account

Open https://aws.amazon.com/aispl/registration-confirmation/ in your browser, find the "Create an AWS Account" button on the top right and sign up for the free tier AWS account.
There might be a bit lenghty process to get enrolled for AWS free tier account but once you are through to it, you will be able to access all the resources of AWS. Simply just follow the instructions and enter asked details.

If you already have the AWS account next step would be get logged into the AWS console. Once you are logged in you will see kind of below interface.

Step 2 : Set up AWS - EC2 instance :

When it comes to deployment, in earlier days, dedicated server was required and for that many machines used to be allocated for single application along with necessary infrastructure (OS, dependency softwares, etc).
But nowadays CLOUD absorbs that complexity and we just need to create one machine on cloud which will hold the application and underlying its services.
Here we are going to consume AWS services and AWS as Cloud provider.

So further elaborating on that, we will create AWS EC2 instance (Amazon Linux virtual machine) on that we will feature different AWS service to support the application.
To create the EC2 instance, follow below steps :

1. Select Expand All Services from AWS Management Console
2. Select EC2
3. Select Launch Instance
4. Step 1 : Choose an Amazon Machine Image (AMI) Select the free tiered Amazon Linux AMI 2015.03.0 x86_64 HVM GP2 (ami-1ecae776)
5. Step 2 : Accept the defaults & Choose an Instance Type and choose Next
6. Step 3 : Select the IAM Role with S3 rights here as per above instructions you should select S3CodeDepRole role here which created earlier

7. Step 4 : Once completed select Add Storage and accept the defaults
8. Step 5 : Add Tag for an example key as "Name" and value as "codedeploy" (this tag will help you to define this machine while creating Deployment Group which state that you want to deploy your application on this particular instance or group of instance in case of Auto Scaling)
9. Step 6 : Configure Security Group if no existing groups are there or else select any of the existing. Please make sure here you give access to SSH and HTTP protocols here to your instance
10.Click on Add Rule and make sure it is Custom TCP (Also add ip adress 0.0.0.0/0 - it means any ip can access your instance you can specify the ip for accessibility limitations)
12.Fill in the Port Range Field with 3000 since our NodeJS application will run on port 3000
13.Finally press Review and Launch
14.Press Review Instance  & Launch

After you select Launch you will be prompted to Select an existing key pair or to create new one, you can create fresh key pair and select the same. I have added fresh named 'nodepair', you can choose any name you want.

Please make sure that you will need to download the .pem file for the new key-pair you have just created, store the same .pem file at safe location this will be require it later.

Select the acknowledgement e.g. I acknowledge that I have access to the selected private key file (nodepair.pem), and that without this file, I won't be able to log into my instance. and press Launch Instances.

For AWS you are all set and this will launch EC2 instances.

Below is the video tutorials to launch the EC2 instance :

Step 3 : Install Code deploy agent to your instance  :
The instance you have created must have code deploy agent installed in that. To install the same SSH connect to the public DNS. To find the same EC2 Dashboard => Instance you have created => in bottom window the instance details must be described => you may find Public DNS there as per below snap

You can copy the DNS url and SSH to it through terminal. To do the same open the terminal, redirect the same to the path of the key you downloaded, in our case that was nodepair.pem. You can ssh using below command

ssh -i "nodepair.pem" ec2-user@ec2-18-233-159-142.compute-1.amazonaws.com

once you get logged into that, execute below command

Sudo su

Then you can execute below commands one by one

sudo yum update
sudo yum install aws-cli
cd /home/ec2-user
aws s3 cp s3://aws-codedeploy-us-east-1/latest/install . --region us-east-1
chmod +x ./install
sudo ./install auto
service codedeploy-agent status

The last command will show you the status of the code deploy agent with its process id.

Once you are all set with this step, next and final step to create Application and Deployment group from AWS Code Deploy.


Step 4 : Create S3 bucket for the application code :
To create S3 bucket where the app code will fall in, from the Services => Storage pane select S3. Enter the name of S3 bucket and give it public access by unchecking all the block access checks from the Set Permissions stage, by default those might be selected.

Please note the s3 bucket name should be unique and you will require the same to be entered in the Jenkins code deploy plugin moving further.




Step 5 : Configure AWS CodeDeploy Service :


Dependencies to create AWS CodeDeploy are as below.

1. Create an application
2. Create Deployment Group

Next step will be to create an Application on AWS which will hold the demo application and then Deployment Group. Below is the video which will demonstrate you how to create an application on AWS along with the Deployment Group.

Please make sure you have noted down the application and deployment group names as you will need the same to be put in the Jenkins code deploy plugin in the post build actions.

Now you are all set to start thinking about how you can automate your first deployment using Jenkins Pipeline.

Step 4 : Configure Jenkins with AWS CodeDeploy Plugin  :

Below are the steps where you can install the AWS CodeDeploy plugin which will help Jenkins to communicate with AWS in real time and deploy the built application.

1. GoTo Jenkins dashboard

2. Select Manage Jenkins

3. Select Manage Plugins

4. Select Available Tab

5. Search for AWS CodeDeploy Plugin for Jenkins

6. Select install 

7. Select Restart Jenkins after installation

You can consume the same plugin as Post Build phase and that is what our next step would be to Create a freestyle project in Jenkins and configure it with Git Code base and Post Build actions. You can review the first part of this tutorial for detailed guidance about how to configure Jenkins Job with GitLab/Github as part of Continuous Integration Pipeline.

Step 5 : Develop Jenkins Pipeline & Configure Jenkins for AWS Code Deploy :

First and foremost, you may require (if the job is not already configured) to create a Jenkins freestyle project/job to configure with application code base integration (Git) and post build action configuration.

Below are the steps :

1. GoTo Jenkins dashboard

2. Select New Item

3. Select Freestyle Project

4. Name the Project

This will create job successfully and it will open the  job configuration page consisting of different stages e.g. General, SCM - Source Code Management, Build Triggers, Build and Post Build Actions. Below video will guide you how to execute this all steps.

Summary of the video tutorial :

As you can initially video walk you through above steps. Apart from that, there are couple of other different important aspects which are partially covered in this video due to security reasons which I can walk you through thoroughly.

SCM : 

First step here is SCM - Source Code Management, as described in video you can add the source code GitLab SSH URL into the SCM => Git, now question arises isn't there any authentication hook required here. Of course there might be SSH authentication required where you will need to authenticate your Jenkins user to the repository by adding the SSH key in the profile settings. This step is greatly demonstrated in the first part of this blog which is here.

Build Triggers :

Continuing to next step which is Jenkins Build Triggers. Basically, build trigger is the main part as far as the pipeline is concern why, because it listens to different events and based on that Jenkins triggers the Build phase. Although, as an admin you will have to configure the Build triggers for your build and there are various options to configure in that. In our case, we will use Webhook to trigger build from Git Source commit. Below is the high level steps how you can configure Webhook in your Git Repo.

1. GoTo your GitLab/Github Repo Settings

2. Select Webhook in case of Github

3. Select Integrations in case of GitLab

4. Enter URL http://IP_WHERE_JENKINS_SETUP:PORT/project/FREESTYLE_PROJECT_NAME/ 

5. Save the configuration

For detailed understanding regarding Webhooks for both GitLab and Github, you can visit first part of this blog which is here

Furthermore, GoTo jenkins Build Triggers and select below options.

1. Build when a change is pushed to GitLab. GitLab webhook URL: http://localhost:8080/project/DemoNodeAppPipeline

2. GitHub hook trigger for GITScm polling 

Build :

Now as part of the next step, select Build phase and select Execute shell script. As we are building and deploying a NodeJs application we will add a phase where the project dependencies should get installed and unit test cases should also automatically should get executed. As a result, we will be adding 2 commands in the added shell script section as mentioned below.

1. npm install

2. ./script/test

Post Build Actions :

As you can review from the video tutorial, add AWS CodeDeploy phase from the dropdown menu and if you are not able to find, the plugin you have installed in Step 4 is not installed properly. Once you will add that phase in to post build actions that will consist of a few fields which will help Jenkins to link with AWS EC2 instances. Below are the information you can add in that.

1. AWS CodeDeploy Application Name (which is your application name which you have created on AWS EC2)

2. AWS ClodeDeploy Deployment Group (which is the Deployment Group within that application)

3. AWS CodeDeploy Deployment Config (which you can find in the Deployment Group detail page on AWS)

4. Region

5. S3 Bucket name (which you can create as per video and paste that bucket name here)

Now the main part is authenticate this hand shaking as Jenkins will try to connect with AWS EC2 instances. Using AWS Access/Secret keys, to see how follow below steps

• Using AWS Access/Secret Keys : 

• For security reasons I can demonstrate this part through video but to acquire the AWS Access and Secret keys, expand the account name menu => Select My Secret Credentials option.
• This will pop up a modal having option Continue with Security Credentials
• This will open an expandable list - find Access Keys option from the list and expanding that pane, you will be able to create Access/Secret Key
• Select Create Access Key => Pop up modal => Select show access keys
• Capture those keys in some sticky notes and paste the same in the Jenkins fields

Step 6 : Configure for email notification in Jenkins :

As a first step to configure for post build notification, Jenkins must be having configurable settings pane for Email Notification. To verify the same, GoTo Jenkins Dashboard => Manage Jenkins => Configure System => Check whether this screen consist of either E-mail Notification. If it does thats where you will require to set up your host account there from which you want to send the post build notifications to recipients.

P.S. : Jenkins app will require some email sending rights from your account through which you want to distribute the post build notifications. Although its all safe and nothing to worry about it if you are using Gmail Account as Google separately give feature to set App Based passwords, so you can define password to that particular app only which is gonna access your email account, in this case it is Jenkins.

Below is the video tutorial how you can configure the post build email notifications from Jenkins. In addition, as you see in the video, select job configuration and from its post build actions you can add one more phase which is Editable Email Notification where you can configure the parameters as below (also check below picture) :

• Project From : Project Name
• Project Recipient List : Recipients to whom you want to send notifications
• Project Reply-To List : Reply email address
• Content Type : Content type from dropdown
• Default Subject : Email Subject
• Default Content : Email content

You might be wondering its as simple as that! But... how to allow Jenkins to send emails on behalf of you that is bit tricky! For security reasons this can not be demonstrated through video.

How to allow Jenkins to send email on behalf of you :

• sing-in to gmail account >> navigate to settings >> privacy and security settings
• setup two step verification settings (because without two step verification we cannot generate application specific password)
• after setting up two step verification setting in gmail account navigate back to security and privacy settings
• click on application specific password >> give the name of the application in the drop down as Jenkins (google by default does not have any specific application password setting for Jenkins) >> this will generate password note down the password generated
• Now as you can see in the video, there is a field called Password, paste the above application specific password to that field 
• Apply & Save and you are done!

Please make sure these steps may vary with email server you are using, I had gmail as a email provider for which I have mentioned the process. 

Conclusion :

As a conclusion, the take away form this tutorial is how to build a well versed Jenkins pipeline with AWS deployment with NodeJS application. You can build any pipeline using this tutorial with slight configuration changes in Jenkins and you will be good to go.

Although this 6 step walkthrough/workshop is fair enough lengthy but as soon as you will start exploring this, I am sure you will find it interesting to execute end to end. Finally, the result will be sweet :)