Saturday, October 12, 2019

MongoDB Getting Started and Setup Cluster

In the past couple of days. I have change my job started working on server less model.so all of there data they don't store in the traditional way in RDBMS in fact thay store all of there application logs in the DB.

So We have started using MongoDB As a document database few of the advantages of mongodb.
  • Document Oriented Storage − Data is stored in the form of JSON style documents.
  • Index on any attribute
  • Replication and high availability
  • Auto-sharding
  • Rich queries
  • Fast in-place update
  • Professional support by MongoDB.

    Now Lets start Building the Mongo cluster with 1 master 1 stave 1 Arbiter

    Pre-requisites.
  • 3 Centos machines (in cloud or on premises)
mongo0 192.168.56.131
mongo1 192.168.56.132
mongo2 192.168.56.133
  1. 1 Windows Host for testing using studio 3T
Login on all 3 vms,
  1. Disable the firewalld service
# systemctl disable firewalld
  1. Stop the firewalld service
# systemctl stop firewalld
  1. Disable SeLinux
To permanently disable SELinux, use your favorite text editor to open the file /etc/sysconfig/selinux 
Then change the directive SELinux=enforcing to SELinux=disabled 
  1. Update the all packages and Install EPEL release repo
# yum update -y && yum insall epel-release
  1. Edit hostname file and change the hostname mongo0,mongo1,mongo2… /etc/hostanme


  1. Add the below entry in the /etc/hosts
mongo0 192.168.56.131
mongo1 192.168.56.132
mongo2 192.168.56.133


  1. reboot the system
# reboot now


Configure the package management system (yum).
Create a /etc/yum.repos.d/mongodb-org-4.2.repo file so that you can install MongoDB directly using yum:
[mongodb-org-4.2]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.2/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.2.asc
Install the MongoDB packages.
To install the latest stable version of MongoDB, issue the following command:
sudo yum install -y mongodb-org
Alternatively, to install a specific release of MongoDB, specify each component package individually and append the version number to the package name, as in the following example:
sudo yum install -y mongodb-org-4.2.0 mongodb-org-server-4.2.0 mongodb-org-shell-4.2.0 mongodb-org-mongos-4.2.0 mongodb-org-tools-4.2.0
You can specify any available version of MongoDB. However yum upgrades the packages when a newer version becomes available. To prevent unintended upgrades, pin the package. To pin a package, add the following exclude directive to your /etc/yum.conf file:
exclude=mongodb-org,mongodb-org-server,mongodb-org-shell,mongodb-org-mongos,mongodb-org-tools

Start one member (master) of the replica set.

This mongod should not enable auth.

Create administrative users.

The following operations will create two users: a user administrator that will be able to create and modify users (siteUserAdmin), and a root user (siteRootAdmin) that you will use to complete the remainder of the tutorial:
use admin
db.createuser( {
user: "siteUserAdmin",
pwd: "",
roles: [ "userAdminAnyDatabase" ]
});
db.createuser( {
user: "siteRootAdmin",
pwd: "",
roles: [ "userAdminAnyDatabase",
"readWriteAnyDatabase",
"dbAdminAnyDatabase",
"clusterAdmin" ]
});

Stop the mongod instance.

Create the key file to be used by each member of the replica set.

Create the key file your deployment will use to authenticate servers to each other.
To generate pseudo-random data to use for a keyfile, issue the following openssl command:
mkdir -p /etc/mongod
openssl rand -base64 741 > /etc/mongod/mongodb-keyfile
chown -R mongod:mongod /etc/mongod
chmod 600 /etc/mongod/mongodb-keyfile
You may generate a key file using any method you choose. Always ensure that the password stored in the key file is both long and contains a high amount of entropy. Using openssl in this manner helps generate such a key.

Copy the key file to each member of the replica set.

Copy the mongodb-keyfile to all hosts where components of a MongoDB deployment run. Set the permissions of these files to 600 so that only the owner of the file can read or write this file to prevent other users on the system from accessing the shared secret.
mkdir -p /etc/mongod
chown -R mongod:mongod /etc/mongod
chmod 600 /etc/mongod/mongodb-keyfile
on master
Execute the scp command for all the slaves one by one.
scp /etc/mongod/mongodb-keyfile user@slave: /etc/mongod/mongodb-keyfile

Sample mongod.conf file is looks something like this



# mongod.conf
# for documentation of all options, see:
# http://docs.mongodb.org/manual/reference/configuration-options/
# where to write logging data.
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log


# Where and how to store data.
storage:
dbPath: /var/lib/mongo
journal:
enabled: true
# engine:
# wiredTiger:
# how the process runs
processManagement:
fork: true # fork and run in background
pidFilePath: /var/run/mongodb/mongod.pid # location of pidfile
timeZoneInfo: /usr/share/zoneinfo
# network interfaces
net:
port: 27017
bindIp: 0.0.0.0 # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.
replication:
replSetName: "rs0"
security:
keyFile: /etc/mongod/mongodb-keyfile
#operationProfiling:
#replication:
#sharding:
## Enterprise-Only Options
#auditLog:
#snmp:

Make sure the changes highlighted in yellow are available in your conf file

Connect to the member of the replica set where you created the administrative users.

Connect to the replica set member you started and authenticate as the siteRootAdmin user. From the mongo shell, use the following operation to authenticate:
use admin
db.auth("siteRootAdmin", "");

Initiate the replica set.

Use rs.initiate() on the replica set member:
rs.initiate()
MongoDB initiates a set that consists of the current member and that uses the default replica set configuration.

Verify the initial replica set configuration.

rs.conf()
The replica set configuration object resembles the following:
{
"_id" : "rs0",
"version" : 1,
"members" : [
{
"_id" : 1,
"host" : "mongo0:27017"
}
]
}

Add the remaining members to the replica set.

Add the remaining members with the rs.add() method.
The following example adds two members:
rs.add("mongo1")
rs.addArb("mongo2")
When complete, you have a fully functional replica set. The new replica set will elect a primary.

Check the status of the replica set.

Use the rs.status() operation:
rs.status()
On secondary (slave) execute
rs.slaveOk()

Tuesday, July 31, 2018

Getting Started with Docker..Management

Docker concepts

Docker is a platform for developers and sysadmins to develop, deploy, and run applications with containers. The use of Linux containers to deploy applications is called containerization. Containers are not new, but their use for easily deploying applications is.
Containerization is increasingly popular because containers are:
  • Flexible: Even the most complex applications can be containerized.
  • Lightweight: Containers leverage and share the host kernel.
  • Interchangeable: You can deploy updates and upgrades on-the-fly.
  • Portable: You can build locally, deploy to the cloud, and run anywhere.
  • Scalable: You can increase and automatically distribute container replicas.
  • Stackable: You can stack services vertically and on-the-fly.
As its a trending technology I also try to learn it but I could not get much so I was looking for some thing which can help me with docker management and I can focus of writing containers and can manage them from GUI so after a search on google i came across a tool call portainer which it self is deployed on  docker but can be a great help to manage docker containers. so i started to evaluate it. Its really easy to install just install docker on host using command


# sudo apt-get install docker.io

# docker volume create portainer_data




# docker run -d -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer




and your portainer get up and running in no time and you can access the same on the browser
http://:9000





You can do everything from building containers to deploying it.

Friday, July 27, 2018

AWS Lambda To start and Stop Instance as Per the Schedule Tag

Hello Guys,

I have recently join a new company and development team is used to work in the India working hours and rest of the time the Servers on the AWS are remaining ideal/utilized rest of the day. so they want to reduce the cost so they come to me and ask for the strategy to reduce cost so I suggested to go have shut down the machine/servers in non-working hours which they agree but  they want a governing mechanism to enforce this so I have  come up with a lambda function which allow them to customized startup and stop time as well as they don't need to manually start and stop the same. As i need to implement the same functionality on the 25 account so I have written a lambda function and then put it in a cloud formation and provided them and ask them to execute this in the account and then just add tag name "schedule" and value of start and stop time in UTC as the lambda is run on the UTC lambda function will take care of the rest.
To further reduce costs of our EC2 instances, we determined some instances that are running 24/7 unnecessarily. An automated process to schedule stop and start instances would greatly help cutting costs. Of course, the solution itself should not add an extra instance to our infrastructure. The solution must be an example of serverless computing.
Automatically stop running EC2 instances 24/7 unnecessarily
We created a lambda function that scans all instances for a specific tag. The tag we use is named ‘Schedule’ and contains the desired ‘runtime’ for the specific instance. Instances without a Schedule tag will not be affected. We support the following content format in the Schedule tag:
08:00-19:15   start the instance at 08:00, stop it at 19:15
21:00-07:45   start the instance at 21:00, stop it at 07:45 the next day
-17:45              stop this instance at 17:45 today
-16:30T           Terminate instance at 16:30 today
# whatever     Anything starting with a # is totally ignored
NOTE: All times are UTC times!
Invalid tag-content is automatically prefixed with an # so that it is ignored in future invocations of the lambda function. Also instances with only an end-time will have their tag rewritten to avoid restarts at 00:00.
Our lambda function is written in python using boto3 for AWS integration. The function requires a role to be able to interact with EC2. The lambda function has to be able to read and change tags, stop and start instances and even terminate them!
To further automate the process, we need to automatically execute the lambda function, which can be achieved by creating a CloudWatch Event Rule. It is a bit of a funny place for a schedule, but that’s where you configure it. The event will be used as a trigger to start our lambda function.
The scheduler we create in the stack runs the lambda function every 10 minutes, giving your EC2 scheduler a granularity of 6 runs per hour.
As shown in the below screenshot One need to just add the tag “Schedule” and required timing against it and it will start and stop the instance at the given time. 



Below is the attached cloud formation template .
{
  "AWSTemplateFormatVersion": "2010-09-09",
  
  "Description" : "[INFRA] Lambda code to start/stop EC2 instances based on tag Schedule. Examples of valid content for Schedule tag are: 08:00-18:00 EC2 instance will run from 08:00 till 18:00 every day. -17:00 wil stop the instance today at 17:00 and rewrite the tag to #-17:00. -17:00T will TERMINATE the instance at 17:00. Any invalid content will be rewritten to start with a #. Any content starting with # will be ignored",
  "Resources" : {
  
    "EC2SchedulerLambda": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Handler": "index.EC2Scheduler",
        "FunctionName" : "EC2Scheduler",
"Description" : "[INFRA] Lambda code to start/stop EC2 instances based on tag Schedule. NOTE times are UTC. See stack description for further information",
        "Role": { "Fn::GetAtt" : ["EC2SchedulerRole", "Arn"] },
        "Code": {
          "ZipFile":  { "Fn::Join": ["\n", [
"from datetime import datetime, time",
"import boto3",
"import re",
"",
" theTag='Schedule'",
"",
"doLog = True",
"",
"def EC2Scheduler(event, context):",
"",
"  now = datetime.now().replace(second=0, microsecond=0)",
"",
"  if doLog:",
"    print 'Reference time is ', now.time()",
"",
"  ec2 = boto3.resource('ec2')",
"",
"  allInstances = ec2.instances.filter(Filters=[{'Name': 'instance-state-name', 'Values': ['running', 'stopped']}])",
"",
"  for instance in allInstances:",
"    if instance.tags :",
"      for tag in instance.tags:",
"        if tag['Key'] == theTag :",
"          Range=tag['Value']",
"",
"          if doLog :",
"            print instance.instance_id, ",
"",
"          shouldrun=shouldRunNow(instance, now, Range)",
"",
"          if doLog :",
"            print 'should %s' %( 'run' if shouldrun else 'NOT run'),",
"",
"          alignInstance(instance, shouldrun)",
"",
"def shouldRunNow(instance, now, tRange):",
"",
"  currentState = instance.state['Code'] == 16",
"",
"  # is the tRange commented out",
"",
"  if tRange[0:1] == '#':",
"    if doLog:",
"      print 'Range starts with # -- no changes',",
"    return currentState",
"",
"  # does the time indicate an end-time only [eg -13:15]",
"",
"  if tRange[0:1] == '-':",
"    tEnd=time(int(tRange[1:3]),int(tRange[4:6]))",
"",
"    if doLog:",
"      print 'End time:', tEnd,",
"",
"    if now.time() > tEnd:",
"",
"      msg='stop'",
"      if tRange[6:7] == 'T':",
"",
"        terminateAllowed=instance.describe_attribute(Attribute='disableApiTermination')['DisableApiTermination']['Value'] == False",
"",
"        if terminateAllowed:",
"          instance.modify_attribute( Attribute='instanceInitiatedShutdownBehavior', Value='terminate' )",
"          msg='terminate'",
"        else:",
"          msg='stop [terminate not allowed]'",
"",
"      # re-tag",
"",
"      instance.delete_tags(Tags=[{'Key': theTag, 'Value': tRange}])",
"      instance.create_tags(Tags=[{'Key': theTag, 'Value': '#'+msg+'@'+tRange}])",
"",
"      if doLog:",
"        print 'time to ', msg, ",
"",
"      return False",
"",
"    else:",
"      # leave instance in current state",
"      if doLog:",
"        print 'NO time to stop (yet) ',",
"      return currentState",
"",
"  # some simple checks for tRange",
"",
"  if not re.match('\\d{2}:\\d{2}-\\d{2}:\\d{2}',tRange):",
"    if doLog:",
"      print 'error in format of tag: >', tRange, '< -- no changes required ',",
"",
"    instance.delete_tags(Tags=[{'Key': theTag}, {'Value': tRange}])",
"    instance.create_tags(Tags=[{'Key': theTag, 'Value': '# Err: '+tRange}])",
"",
"    return currentState",
"",
"  tStart=time(int(tRange[0:2]),int(tRange[3:5]))",
"  tEnd=time(int(tRange[6:8]),int(tRange[9:11]))",
"",
"  inInterval = False",
"    ",
"  # first case, start < end --> same day",
"",
"  if tStart < tEnd:",
"    if tStart <= now.time() <= tEnd:        ",
"      inInterval = True ",
"",
"  # second case, end < start --> carry over to next day",
"",
"  else:",
"    if tStart <= now.time() <= time(23,59) or time(0,0) <= now.time() <= tEnd:",
"      inInterval = True",
"",
"  if doLog :",
"    print 'Ref time is %s interval' %( 'in' if inInterval else 'NOT in'), tRange, ",
"",
"  return inInterval",
"",
"def alignInstance(inst, requiredOn):",
"",
"  actualOn = inst.state['Code'] == 16 ",
"  msg='is compliant'",
"",
"  if actualOn != requiredOn:",
"    if requiredOn == True:",
"      msg='starting'",
"      inst.start()",
"    else:",
"      termReq=inst.describe_attribute(Attribute='instanceInitiatedShutdownBehavior')['InstanceInitiatedShutdownBehavior']['Value'] == 'terminate'",
"",
"      if termReq:",
"        msg='terminating'",
"        inst.terminate()",
"      else:",
"        msg='stopping'",
"        inst.stop()",
"",
"  if doLog:",
"    print '-->', msg",
""
  ]]}
        },
        "Runtime": "python2.7"
      }
    },
    
    "EC2SchedulerTick": {
      "Type": "AWS::Events::Rule",
      "Properties": {
"Description": "ScheduledRule for EC2Scheduler",
"ScheduleExpression": "rate(10 minutes)",
        "Name" : "EC2SchedulerTick",
"State": "ENABLED",
"Targets": [{
  "Arn": { "Fn::GetAtt": ["EC2SchedulerLambda", "Arn"] },
  "Id": "TargetFunctionV1"
}]
      }
    },
    "PermissionForEventsToInvokeLambda": {
      "Type": "AWS::Lambda::Permission",
      "Properties": {
"FunctionName": { "Ref": "EC2SchedulerLambda" },
"Action": "lambda:InvokeFunction",
"Principal": "events.amazonaws.com",
"SourceArn": { "Fn::GetAtt": ["EC2SchedulerTick", "Arn"] }
      }
    },
    "EC2SchedulerRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [{ "Effect": "Allow", "Principal": {"Service": ["lambda.amazonaws.com"]}, "Action": ["sts:AssumeRole"] }]
        },
        "Path": "/",
        "RoleName" : "EC2SchedulerRole",
        "Policies": [{
          "PolicyName": "EC2SchedulerPolicy",
          "PolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
    {
"Effect": "Allow",
"Action": ["logs:*"], 
"Resource": "arn:aws:logs:*:*:*" 
    },
    {
"Effect": "Allow",
"Action": ["ec2:*"], 
"Resource": "*"
    }]
          }
        }]
      }
    }
  }

}

this is it, Enjoy...