15 12 2015

In my devops life, one of my customer ask to have the possibility to update one by one his java workers but without interuption. They are java workers behind an apache. Apache load balance request between the workers with mod proxy ajp.

The goal was to :

1. Remove one of both tomcat server from the apache balancer proxy.
2. Do whatever on this tomcat (upgrade code, change parameter, …)
3. Add again the tomcat

How can we remove/add a worker in the apache proxy mod ? The first ideas was :

• Stop worker (service tomcat stop) ?
• Add iptables rules to  ?
• Edit vhost configuration, remove membre, reload ?

But it is difficult to never interupt a request. So I want share a tip about BalancerMember from apache mod_proxy (and proxy_balancer).

I wrote a little python script to do remove a balancer member from the apache proxy without requetes interruption.

You can find my script here https://github.com/talset/apache-tools

How to

To run that script you need to setup standard python-argparse arg parse package :

apt-get install python-argparse

And then, if you need a specific servername (host) or ip to manage your apache mod proxy, you should edit following lines

#vhost name / header
headers = {"Host": '127.0.0.1' }
#ip to reach apache
url="http://127.0.0.1/balancer-manager"

balancer-manage.py 
usage: balancer-manage.py [-h] [-l] [-a ACTION] [-w WORKER]

optional arguments:
  -h, --help            show this help message and exit
  -l, --list            List Worker member and status
  -a ACTION, --action ACTION
                        "enable" or "disable" the specified Worker
  -w WORKER, --worker WORKER
                        Worker name : example ajp://127.0.0.1:8001

Usecase

Context : We need update an application on each webserver without interuption

We currently have Apache proxy with 2 members. Each members are running the old version of our customer application.

 ./balancer-manage.py -l
    Worker                                   | Status     | Elected   
    ajp://10.152.45.1:8001                   | Ok         | 1        
    ajp://10.152.45.2:8001                   | Ok         | 2

We removed the first web 10.152.45.1 to perform an update of the application

./balancer-manage.py -w ajp://10.152.45.1:8001 -a disable
    Worker                                   | Status     | Elected   
    ajp://10.152.45.1:8001                   | Dis        | 0         
    ajp://10.152.45.2:8001                   | Ok         | 0 

Enable the first web 10.152.45.1. At this state we have removed web1 from the balancer and our customer have upgraded his application. Next step : Re enable web1 in the load balancer.

./balancer-manage.py -w ajp://10.152.45.1:8001 -a enable
    Worker                                   | Status     | Elected   
    ajp://10.152.45.1:8001                   | Ok         | 0         
    ajp://10.152.45.2:8001                   | Ok         | 0

We now balance between an old version of the application and new at the same time. We have proceed an upgrade on one tomcat without interrupt the service. Next step do the same thing on web2. Remove the second web 10.152.45.2 and upgrade the application

./balancer-manage.py -w ajp://10.152.45.2:8001 -a disable
    Worker                                   | Status     | Elected   
    ajp://10.152.45.1:8001                   | Ok         | 0         
    ajp://10.152.45.2:8001                   | Dis        | 0

The two webs are upgraded, But web2 still remove from the loadbalancer. So finally add the last member into the apache proxy.

./balancer-manage.py -w ajp://10.152.45.2:8001 -a enable
    Worker                                   | Status     | Elected   
    ajp://10.152.45.1:8001                   | Ok         | 0         
    ajp://10.152.45.2:8001                   | Ok         | 0

We just upgraded 2 webs without request interuption. Smooth upgrade 😉

talset

apache, cli, devops, linux, python, tools

Openshift 3 – where is my permanant token ? Decode EXIF from a raw photo