Syncing code on multiple servers in real time 1

Few days back my boss come up with a requirement of syncing on data across the servers .
it may happen that servers IP address can change as they ware hosted on Amazon.

As all of us know by default they don't provide us white(public )IP Address.so i start searching and was very much worried as syncing should be real time and i dont know on which server the data is going to sync.

i have googled a lot and then i read about the torrent sync. Its a very good open source software having great functionality.I really love it.

So lets start with installation.

Install

As its not directly available we need to add PPA first and then install it Per usual, setting up the Windows and Android clients is a fairly brainless process; setting up a Linux client is a little more exciting. Here’s what I did on my Ubuntu 12.04 server:

Set Up: We don’t set things up manually in Ubuntu, son!

Use the PPA:

$ sudo add-apt-repository ppa:tuxpoldo/btsync
$ sudo apt-get update
$ sudo apt-get install btsync

If you’re on a version of Ubuntu that didn’t come with add-apt-repository, you can grab it from the software-properties-common package.

When you do this you need to go through the following installation cycle and when you done you are all set repeat the same steps and data syncing start automatically

For me i have choose 12345 as a sync port you can peek any but its recommended to use port above 1024 as bellow 1024 all are well known ports

Use windows tools (IE) on Ubuntu

you need to download curl
sudo apt-get install curl

wget -q "http://deb.playonlinux.com/public.gpg" -O- | sudo apt-key add -
sudo wget http://deb.playonlinux.com/playonlinux_precise.list -O 
/etc/apt/sources.list.d/playonlinux.list

sudo apt-get update
sudo apt-get install playonlinux


Now Wine will get automatically updated,

Open PlayonLinux and install IE6/IE7/IE8

Then click click click and done   Enjoy....

Building A Continuous Integration Server java Project part 1

Building A Continuous Integration Server java Project

For building a continuous Integration Sever we will use Jenkins and supported java libraries.

 Setup Jenkins on ubuntu 12.04 server 

sudo apt-get install default-jdk ant

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins

and to upgrade the existing setup

sudo apt-get update
sudo apt-get install jenkins

new after the installation jenkins is online on the port no 8080 you can access it using 

in web browser ipaddress of machine:8080 ex.172.16.16.10:8080 

you can also convert this thing into a URL for that you need to install apache on the 

machine 

sudo apt-get install apache2

and create a file call jenkins.conf as

nano /etc/apache2/sites-available/jenkins.conf

#and add the following line to it 

<VirtualHost *:80>
 ServerAdmin yourname@company.com
 ServerName ci.company.com
 ServerAlias ci
 ProxyRequests Off
 <Proxy *>
  Order deny,allow
  Allow from all
 </Proxy>
 ProxyPreserveHost on
 ProxyPass / http://localhost:8080/
</VirtualHost>

make the DNS settings or make a host entry on the syatem where you want to access 

the server and we are all set with the basic setup of jenkins

Fixing MySQL Replication-with minimum downtime

Its a very critical task for a system Admin/DBA to fix the broken MySQL replication when the data in the system is crucial and does not afford any type of data loses.


Before starting the following tutorial Please fire stop slave on both the server if you have master master replication on slave server if have master slave.to avoid any kind of data loses.                 

mysql>stop slave;

If you are using INNODB as a storage/database engine they its also very critical to fixx the replication bacause though you drop the database from server innodb will never release the hard disk space acquired by the database






Note* before doing any thing Take the backup of all the databases available in the mysql.

ibdata stores all of the UNDO LOGS thus GROWS due to the deletes and space is never reclaimed.

To regain the harddisk space  you need to delete the ibdata file and log file of mysql which are in the /var/lib/mysql

Note* For our convenience we can keep the backup of ibdata and iblogs file 

cp -ar /var/lib/mysql/ib* /tmp/
rm /var/lib/mysql/ib*
service mysqld restart

This process with again regenerate the mysql back with the shrink ibdata file and log file.and mysql will again come up then you can restore all the other databases which are not under the replication.

then take the Mysql Consistent snapshot from the other server(which is having the latest data).
open two terminal and from one login into the mysql
select the database which is under the  rpelication

mysql> use <database name>
mysql> flush tables with read lock;
mysql> show master status;

 Note down the master log file and log position of master or take a snapshot of the terminal.

on the other terminal start taking the backup using mysqldump

#mysqldump -u root -p <databasename> > <databasename.sql>

A SQL dump file get generated and we can copy the file to other server(affected server)

#scp <databasename.sql> <user>@<serverip>:/tmp

after the dump completes fire on first terminal
mysql >unlock tables;
and close the terminal on master.

On the affected server open the mysql and fire 
 mysql >SET sql_log_bin =0;

This will ignore the bin log while restore and You are safe during the restore that data not get replicated from one to another server
then use the restore command  which can be done in two ways.
using the shell and other from mysql

#mysql -u root -p <databasename> < <databasename>.sql
or 
#mysql -u root -p 
mysql> use <databasename>;
mysql> source  <path/to/databasename.sql> 

Note  if my file is in  tmp and database file name as newdb.sql then
mysql> source /tmp/newdb.sql 

ofter the restore please note down the servers master log possition useg command

mysql> show master status;
mysql >SET sql_log_bin =1; # enable the bin log on the affected server

then use change master to command and again re-point the servers

change master to
master_host='ipaddress of master',
master_user='user_having_replication slave_privilege',
master_password='user_password' ,
MASTER_LOG_FILE ='mysql log name of master which we have noted first',
 MASTER_LOG_POS = position no  in log filewhich we have noted first ;

And we have done with the fixing of the server with the minimal down time.

Installing many openerp on same server with different codes

Hello friends

I have created a scrip u just need to u just need to copy paste it and change the permission to be executable and let the script run answer the question asked by script and
you can have a openerp on your system with different names like
 openerp-server,nick-server,navneet-server etc .

#!/bin/bash
#make sure server has all the latest versions
#& patches by doing an update
#
sudo apt-get update
sudo apt-get dist-upgrade
#
#Install SSH
#
sudo apt-get install openssh-server denyhosts
#
#Create User
#
echo "Enter The Openerp Username"
read user
sudo adduser --system --home=/opt/$user --group $user
#
#Make Sure that Postgresql is installed on the system
#
sudo apt-get -y install postgresql
#
#Creating the user same as for postgres
#
createuser --createdb --username postgres --no-createrole --no-superuser $user
#
#Install all the dependencies
#
sudo apt-get -y install python-dateutil python-docutils python-feedparser python-gdata \
python-jinja2 python-ldap python-libxslt1 python-lxml python-mako python-mock python-openid \
python-psycopg2 python-psutil python-pybabel python-pychart python-pydot python-pyparsing \
python-reportlab python-simplejson python-tz python-unittest2 python-vatnumber python-vobject \
python-webdav python-werkzeug python-xlwt python-yaml python-zsi
#
#Downloading the Source
#
wget http://nightly.openerp.com/7.0/nightly/src/openerp-7.0-latest.tar.gz
cd /opt/$user
sudo tar xvf ~/openerp-7.0-latest.tar.gz
#changing the ownwrship of code so that it can be accessiable to the $user
#
sudo chown -R  $user: *
echo "owner ship is given"
sudo cp -a openerp-7.0* /opt/$user/server
rm -rf openerp*
echo "openerp is removed"
sudo cp /opt/$user/server/install/openerp-server.conf /etc/$user-server.conf
sudo chown $user: /etc/$user-server.conf
sudo chmod 640 /etc/$user-server.conf
echo "Enter the Port Number on which Openerp will start"
read port
echo "xmlrpc_port=$port" >>/etc/$user-server.conf
echo "init.d started"
cat<< 'EOT' > /etc/init.d/$user
#!/bin/sh
### BEGIN INIT INFO
# Provides:             openerp-server
# Required-Start:       $remote_fs $syslog
# Required-Stop:        $remote_fs $syslog
# Should-Start:         $network
# Should-Stop:          $network
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    Enterprise Resource Management software
# Description:          Open ERP is a complete ERP and CRM software.
### END INIT INFO
PATH=/bin:/sbin:/usr/bin
DAEMON=/opt/openerp/server/openerp-server
NAME=openerp-server
DESC=openerp-server
# Specify the user name (Default: openerp).
USER=openerp
# Specify an alternate config file (Default: /etc/openerp-server.conf).
CONFIGFILE="/etc/openerp-server.conf"
# pidfile
PIDFILE=/var/run/$NAME.pid
# Additional options that are passed to the Daemon.
DAEMON_OPTS="-c $CONFIGFILE"
[ -x $DAEMON ] || exit 0
[ -f $CONFIGFILE ] || exit 0
checkpid() {
    [ -f $PIDFILE ] || return 1
    pid=`cat $PIDFILE`
    [ -d /proc/$pid ] && return 0
    return 1
}
case "${1}" in
        start)
                echo -n "Starting ${DESC}: "
                start-stop-daemon --start --quiet --pidfile ${PIDFILE} \
                        --chuid ${USER} --background --make-pidfile \
                        --exec ${DAEMON} -- ${DAEMON_OPTS}
                echo "${NAME}."
                ;;
        stop)
                echo -n "Stopping ${DESC}: "
                start-stop-daemon --stop --quiet --pidfile ${PIDFILE} \
                        --oknodo
                echo "${NAME}."
                ;;
        restart|force-reload)
                echo -n "Restarting ${DESC}: "
                start-stop-daemon --stop --quiet --pidfile ${PIDFILE} \
                        --oknodo
     
                sleep 1
                start-stop-daemon --start --quiet --pidfile ${PIDFILE} \
                        --chuid ${USER} --background --make-pidfile \
                        --exec ${DAEMON} -- ${DAEMON_OPTS}
                echo "${NAME}."
                ;;
        *)
                N=/etc/init.d/${NAME}
                echo "Usage: ${NAME} {start|stop|restart|force-reload}" >&2
                exit 1
                ;;
esac
exit 0
EOT
echo "conf file is genrated"
sudo chmod 755 /etc/init.d/$user
sudo chown root: /etc/init.d/$user
echo "permission are applied"
sed -i "s/openerp/$user/" /etc/init.d/$user
sed -i "s/peer/trust/" /etc/postgresql/9.1/main/pg_hba.conf
sed -i "s/md5/trust/" /etc/postgresql/9.1/main/pg_hba.conf
service postgresql restart
/etc/init.d/$user start
su postgres
createuser --createdb --username postgres --no-createrole --no-superuser --pwprompt $user
echo "restart postgresql"
echo "open erp is started"


After the script ran completely add a user in postgresql with same name as you have mention in the script and you are ready to go.

Thanks..

Pfsense and other availabe packages with that

Hi In the last post i have tell you about the creating the a home based Powerful router having lot of functionality that is Pfsense.
It is having a nice GUI and and having a lot of packages you can install in that

1. It is having a small DNS server
2. Proxy server
3.Network Monitor
4.Realtime Network Graphs of net uses
5. Download and Upload in the last month last week and last year and day

 You can install  all other coll stuff in that you can see that from the
System menu in that packages in that packages option you will find all the packages which you can install in the system in the available packages tab

Installing Pfsense

Install Pfsense on any old Pc having two network cards (1 for LAN and other for WAN)

1. Download the Pfsense CD from the pfsense website and burn it on the CD/DVD

2. Put the CD in CDROM and boot the computer using CD and you will get something like this.

let it boot in the default mode and it will ask you for the interfaces provide the interfaces for WAN and LAN and any other optional interface if you have or required.

Then connect the LAN interface to your Laptop or your Desktop which you are using and enter the ipaddress of LAN interface which youhave enter by default its 192.168.1.1

Enter the Ip address in the browser and a setup wizard came and complete the wizard

Enter the Login Details as
User :admin
Password :Pfsense

Your Firewall is all set for home use
In the Next part i will tell you about the bandwidth spiting and other cool stuff about the Pfsense.

Install OpenERP 7.0 on Redhat/Cent OS 6.4

Hi Friends I have Created a Script to install OpenERP on RedHat/CentOS

This is in the following Steps

Step 1 .

Make your system up to date

#yum -y update
#yum -y install man wget vim libevent unzip

# Download additional prerequisites
cd /tmp

wget http://pypi.python.org/packages/source/p/pyparsing/pyparsing-1.5.7.tar.gz && tar -xvzf pyparsing-1.5.7.tar.gz && cd pyparsing-1.5.7 && python setup.py install

wget http://gdata-python-client.googlecode.com/files/gdata-2.0.17.zip && unzip gdata-2.0.17.zip && cd gdata* && python setup.py install

rpm -Uvh http://pkgs.repoforge.org/tmux/tmux-1.6-1.el6.rf.x86_64.rpm

rpm -Uvh http://prdownloads.sourceforge.net/webadmin/webmin-1.620-1.noarch.rpm

rpm -ivh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm

rpm -ivh http://yum.pgrpms.org/9.2/redhat/rhel-6-i386/pgdg-centos92-9.2-6.noarch.rpm

wget https://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg && sh
setuptools-0.6c11-py2.6.egg


# Install core prerequisistes for OpenERP
yum -y install python-psycopg2 python-lxml PyXML python-setuptools libxslt-python pytz python-matplotlib python-babel python-mako python-dateutil python-psycopg2 pychart pydot python-reportlab python-devel python-imaging python-vobject hippo-canvas-python mx python-gdata python-ldap python-openid python-werkzeug python-vatnumber pygtk2 glade3 pydot python-dateutil python-matplotlib pygtk2 glade3 pydot python-dateutil python-matplotlib python python-devel python-psutil python-docutils make automake gcc gcc-c++ kernel-devel byacc flashplugin-nonfree poppler-utils pywebdav postgresql92-libs postgresql92-server postgresql92




#make sure that Postgresql is installed on the system
yum install postgresql

And ofter installing the Postgresql use
service postgresql-9.2 initdb
chkconfig postgresql-9.2 on
service postgresql-9.2 start
This will make you PostgreSQL ready to accept the connection

# Add users for managment and OpenERP. Requires manual input of the password (set to "openerp")
#useradd <name user>
#useradd -e yyyy-mm-dd <mane user>
su - postgres -c "createuser --createdb --username postgres --no-createrole --no-superuser --pwprompt openerp"
adduser openerp
DIR="/var/run/openerp /var/log/openerp"
for NAME in $DIR
do
if [ ! -d $NAME ]; then
   mkdir $NAME
   chown openerp.openerp $NAME
fi
done
rm -rf openerp*

# Fetch and install OpenERP
cd /tmp
wget http://nightly.openerp.com/7.0/nightly/src/openerp-7.0-20130408-232357.tar.gz && tar -xvzf openerp-7.0-20130408-232357.tar.gz --transform 's!^[^/]\+\($\|/\)!openerp\1!' && cd openerp && python setup.py install
rm -rf /usr/local/bin/openerp-server
cp openerp-server /usr/local/bin
cp install/openerp-server.init /etc/init.d/openerp
cp install/openerp-server.conf /etc
chown openerp:openerp /etc/openerp-server.conf
chmod u+x /etc/init.d/openerp

Starting Openerp as a service

chkconfig openerp on
service  openerp start

# Clear firewall, enabling access
iptables -F
iptables -F INPUT
iptables -A INPUT -p tcp --dport ssh -j ACCEPT
iptables -A INPUT -p tcp --dport 8069 -j ACCEPT
iptables -A INPUT -p tcp --dport 10000 -j ACCEPT
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
iptables -A INPUT -j LOG -m limit

Guys now the openerp is production ready you can access it using
in browser http://<ipaddress-of-machine>:8069

Postgresql Replication part2

Further Testing

Quering recovery process

If you have a live database that processes a lot of data it may take some time for the replication server to “catch up”   There are a few ways to see where it is in the process.

From the slave database server log in as root and run the following command

>    ps -AF | grep post

Here are two more ways to query the database to see what WAL file it is working on .

From the Master database you can run the following command

>    psql -c "SELECT pg_current_xlog_location()"

Which returns something like this

So how do you read this?

If you look in the pg_xlog folder on the main server you would see files named like this

0000000100000E8C00000095

0000000100000E8C00000096

0000000100000E8C00000097

0000000100000E8C00000098

0000000100000E8C00000099

0000000100000E8C0000009A

Now lets deconstruct the E8C/987975C0

E8C/987975C0

0000000100000E8C00000098

That is how the numbers line up and the rest is a HEX offset within the file (Where within the file it is reading)

Now from the Slave database you can run the following command.

>    psql -c "SELECT pg_last_xlog_replay_location()"

Which returns something like this

Primary stop/start

As some further testing I stopped the primary database and started it back up after a few minutes.   (I also stopped and restarted my python program)

>    sudo /etc/init.d/postgresql stop         >    sleep 120         >    sudo /etc/init.d/postgresql start         >    ./insertDB.py

The replication database had no problem with this

Replication stop/start

From the replication server I stopped and restarted the postgres server.

>    sudo /etc/init.d/postgresql stop         >    sleep 240         >    sudo /etc/init.d/postgresql start

No problem with this either

Test Delete from Replication

The Replication server only has read access, let’s try and delete.

From the database on the replication server try and run this

>    \c nand         >    delete from data;

Change the Replication server to a primary server

Warning do not do this in a live system!!! I just did this to make sure it works…  

The recover.conf file defined /home/postgres/failover as the trigger_file.   Create a file here and see if it stops being a replication server.

Run the following commands

>    sudo mkdir -p /home/postgres         >    touch /home/postgres/failover

Now when I log into the database from the replication server and run the following commands.

>    \c nand         >    select count(*) from data;

I see that it is no longer getting fed from the primary database server.

And I can run the following command

>    delete from data;

And it can now write to the database.

Also when this occurs recovery.conf was renamed to recovery.done.

Now that the Replication server has become a Primary server you can’t easily switch it back to a Replication server.  Which is it should be!