Saturday, February 24, 2018

Enforce Tagging. Using AWS Lambda Part-2

Hello Guys,

As we have already completed with prerequisites in the part-1. so Lets get going with lambda function.

Go to the https://aws.amazon.com  and login to console and click on lambda.
Then click on the Create Function --> Author from Scratch

When you click on author from scratch scroll down and you will find a form so fill it as shown in the snapshot

I have use the Run-time environment as Python 2.7 as I have written the code in Python 2.7
In the Role section Select the Role as Choose from existing role and in the next combo/drop-down box select the role Our role name "Lambda_basic_execution".
and click on Create function.

A nice screen will pop up

and on the left side of the function mean who will trigger it add cloud watch event which we have created in the Part1


Its configuration will look something like this at the bottom of the function.


Now click on the Function and Copy Paste the code.

import boto3
import logging

#setup simple logging for INFO
logger = logging.getLogger()
logger.setLevel(logging.INFO)

TopicArn = 'Write the Topic ARN for notification here'

#define the connection
ec2 = boto3.resource('ec2')

def lambda_handler(event, context):
    # Use the filter() method of the instances collection to retrieve
    # all running EC2 instances.
    filters = [
        {
            'Name': 'instance-state-name',
            'Values': ['running']
        }
    ]
   
    #filter the instances
    instances = ec2.instances.filter(Filters=filters)

    #locate Untagged untagged instances
    untaggedInstances = [instance.id for instance in instances if 'Name' not in [t['Key'] for t in instance.tags]]
#    The below line was added for debugging
    print untaggedInstances
    ptower_untaggedInstances = [instance.id for instance in instances if 'Product Tower' not in [t['Key'] for t in instance.tags]]
    app_untaggedInstances = [instance.id for instance in instances if 'Application' not in [t['Key'] for t in instance.tags]]
    scon_untaggedInstances = [instance.id for instance in instances if 'Support Contact' not in [t['Key'] for t in instance.tags]]
    appown_untaggedInstances = [instance.id for instance in instances if 'Application Owner' not in [t['Key'] for t in instance.tags]]
    dom_untaggedInstances = [instance.id for instance in instances if 'Domain' not in [t['Key'] for t in instance.tags]]
    untaggedInstances = untaggedInstances + ptower_untaggedInstances + app_untaggedInstances  + scon_untaggedInstances + appown_untaggedInstances + dom_untaggedInstances
    untaggedInstances = list(set(untaggedInstances))
    print untaggedInstances
    #print the instances for logging purposes
    #print untaggedInstances
   
    #make sure there are actually instances to shut down.
    if len(untaggedInstances) > 0:
        #perform the shutdown
         #print "Right now doing testing"
         shuttingDown = ec2.instances.filter(InstanceIds=untaggedInstances).stop()
        #publish_to_sns(shuttingDown)
         print shuttingDown
    #     print untaggedInstances
    else:
        print "Nothing to see here"



def publish_to_sns(message):
    sns = boto3.client('sns')
    sns_message = "We have shutdown the instances and Instace IDS are..."+str(message)
    response = sns.publish(TopicArn=topic_arn, Message = sns_message)
--------------------------------------------------------------------------------------------------------------------------
Code is Ends Here dont copy the ------
and After pasting it in the Function code section it will be like.


We are mostly Done.

Now If you Don't Create an Instance with The Tag :

  • Name
  • Product Tower
  • Application
  • Support Contact
  • Application Owner
  • Domain
Then the Instance will be ShutDown also Before enabling the script also make sure your exesting Instances also Have These tags otherwise it will also Stop them....

let me know If you face any issue while Implementing this....Am Happy to help you....
and also Belive me guys You wont find a better way to do this If you have multiple accounts and if they have different tagging requirement this will be the portable and simple solution....Enjoy...


Enforce Tagging. Using AWS Lambda Part-1

Hello Guys,

I have done a very interesting assignment in recent days in my office as the requirement was to shutdown all the instances of aws account which are not tagged with tag 'Owner' in it so I have started working on it for couple of days and created a simple lambda function

So lets start with the implementation
1. Created a IAM role using which the lambda service is going to execute the lambda function.
SO I have created a role name lambda_basic_execution  and attached two one is inline policy which looks something like this and other is EC2fulladmin which is available in aws.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        }
    ]

}

So my IAM role is look something like below snapshot.


Cloudwatch Rule

Then I have Created a cloud-watch Rule which will we use to trigger the function
Sc  this is my rule which will trigger the even when ever the instance change its state.
{ "source": [ "aws.ec2" ], "detail-type": [ "EC2 Instance State-change Notification" ] }


We are almost done with our preparations  We will continue in the next part about the lamda function...

Tuesday, January 30, 2018

Install Sound-wave and AWS inventory

Hello Guys,
Welcome back in the recent event in my office i need to work on a searchable inventory solution in AWS. so that i can get the list of AWS instances in CSV format. So i have google out and try to get the solution suitable for my requirement so i choose the soundwave. So lets not waste the time and getting started with the installation and some sample queries.

So lets install a Ubuntu server on virtual box if you don't want to waste money on AWS server. So i have install on my local server.

I have installed  Ubuntu server and install git on it also installed terraform on it.

#sudo apt-get update
#sudo apt-get upgrade -y
#sudo apt-get install git

Install oracle jdk1.8 on ubuntu srever as a prerequisite for that you need to install ppa repo by using following command

#sudo apt-add-repository ppa:webupd8team/java
#sudo apt-get update
#sudo apt-get install oracle-java8-installer

Now lets install terraform

#wget https://releases.hashicorp.com/terraform/0.9.8/terraform_0.9.8_linux_amd64.zip
#unzip terraform_0.9.8_linux_amd64.zip
#sudo mv terraform /usr/local/bin/

Now  lets conform terraform binary is accessible
#terraform --version

Now lets install docker as well on the server you can take reference from Here . and install docker on your server.

also Keep the Access and secret key of your AWS account handy you can get the access and secret  key from the AWS console.

After this you can follow the instruction form here. After all of done you can check its install and working.check the public ip or IP you use to ssh on server.
and put it in browser.

Tuesday, July 18, 2017

Sonar Analysis with Pull request Raise part 2

Hello Guys,
Part - 2
Follow the link https://www.digitalocean.com/community/tutorials/how-to-install-jenkins-on-ubuntu-16-04
This link will help you in the installation of Jenkins on the server.
Part -3
Also follow the below link This will help you in the installation of SonarQube.
http://rathinavneet.blogspot.in/2017/05/install-sonarqube-on-ubuntu-part-1.html

Install github plugin in the sonarqube and in the jenkins install Sonarqube scanner plugin.


Open jenkins go to manage Jenkins in configure sonar Qube server as shown in figure.



Once the sonar qube is configured now start configuring the github-server in jenkins as shown in figure.

Now lets configure github pull request builder in jenkins.


and Thats it we are done with the integration of github with jenkins now we can configure the job.
which will automatically triggered when you raise a PR(pull request).
Optionally you can also configured Email alert.
That I will cover in the next article.


Wednesday, July 12, 2017

Sonar Analysis with Pull request Raise part 1.

Hello Guys,

In recent days one of my friend had ask me is it possible that when ever some one raise a pull request in the github repo.
ex. Assume i have  github repo called newrepo. Now whenever some one push his code into my repo I need to validate that code against some bench mark of quality and i also need to make sure that the code is of good quality and If it has any defect then I should get those in github comment on the pull request.
Its has to be done in  steps

  1. Create  a Github webhooks
  2. Generate personal access token with appropriate permissions .
  3.  Create a jenkins Job
  4. setup of Sonarqube
  5. Step up Quality gate for the Project (Optional)  
1. Create Github webhook.

In My case go to URL https://github.com/navneet-rathi/newrepo

Open your repository in that click on settings and in settings click on Webhooks or  goto URL.

https://github.com/navneet-rathi/newrepo/settings/hooks.

Replace newrepo with the name of your project repository.


click on add a Webhook and fill the details of your Jenkins server as shown in the picture.




 This web-hook will send an information about the Pull request  and  to trigger a jenkins job we need to create one more web hook as shown in below screenshot



and Thats It We are done with part 1
.
In next part  I will setup jenkins and remaining configuration and sonar changes  

Sunday, June 4, 2017

Getting started with Sonar part 3

Hello Guys,

This is the 3rd and last part of sonar series in which I will explain about how you can detect the pull request which is raise on github it is safe to merge with your main or master branch or not and it will also comment what are the reasons why is it not safe and it will comment on github so lets getting started. with it.
 so first thing first.

1. You have git repository setup
2. Standard sonar setup.

Now How we can active it.

So for testing we have a testing repository
https://github.com/navneet-rathi/sample-code-java.git

and we have a pull request raise for the repository for me the pull request raise we need to know the pull request no. which we can find out through the
   
from This I come to know that the pull request no is 2

you need to provide access the sonar so that it can write on your github. which is possible using personal access token which you will get using github you need to provide proper


I have given the below access it will give me a access code please note it down.

now its start to work.

I will use maven to check go to the project repository and fire the below command

mvn -Dsonar.analysis.mode=preview \
              -Dsonar.github.pullRequest=2 \
              -Dsonar.github.repository=navneet-rathi/sample-code-java \
              -Dsonar.github.oauth=<This token which you have previously noted down> \
              -Dsonar.host.url=http://localhost:9000 \
              -Dsonar.login=admin \
              -Dsonar.password=<your sonar password>

and thats it after this you will where it is feasible to merge the request or not if we merge what will be the consequence.

In my case its failed and the below is the reason.
  
Ohh yes guys below is the genral command 

mvn -Dsonar.analysis.mode=preview \
              -Dsonar.github.pullRequest=$PULL_REQUEST_ID \
              -Dsonar.github.repository=myOrganisation/myProject \
              -Dsonar.github.oauth=$GITHUB_ACCESS_TOKEN \
              -Dsonar.host.url=https://server/sonarqube \
              -Dsonar.login=$SONARQUBE_ACCESS_TOKEN or sonar username \
              -Dsonar.passord=$ Sonar Password
 
So thats it guys let me know if you want to know any thing more on this.

Sunday, May 14, 2017

Getting started with Basic sonar Use Part -2

Hello,

lets start with the sonar analysis we will do the analysis of sonarqube code itself.

first we will download or clone the code on local using below command.I have fork from the sonar repo so I will work on this you can do the same with sonarqube repo as well.

$ git clone https://github.com/navneet-rathi/sonarqube.git

After downloading the code from the github you will see its a maven project we have to install the maven first which is pretty straight forward.

after installing maven we need to change the directory and open then sonarqube directory by command

$ cd sonarqube

Lets start with clean and do analysis of sonarqube

$mvn clean sonar:sonar -Dsonar.host.url=http://localhost:9000 -Dsonar.login=admin -Dsonar.password=<your sonar password>

you can even use -X option with maven for debugging info.

and done then just wait and watch  and rest of work is done by maven and sonar.
you will see some thing after its start its  sonar who is doing analysis.



you can also see some thing like this if you have build breaker plugin install in your sonar.This plugin is useful when you are working with sonar and jenkins together.


Once the analysis is complete on the command line you can go to browser and login in the sonarqube and check the sonarqube code quality check for the project.


  

You can see some thing like this after the analysis is completed.So this is about the basic use of sonar with maven.In the net article i will show you how you can analysis the pull request in github using sonar if it is ok to merge or not.