Run an OpenCms Cluster with a single command

Part II (1)

The all-in-one installation in the first part is pretty good to use as a base installation for development or small installations, but in general for an enterprise grade installation the database server and the application servers are hosted on different machines and for an HA’ed installation you’ll have two different machines running tomcat and apache on each server, where each of which will connect to an OpenCms Database (Cluster) and use Loadbalaning to provide high availability and (auto-) scalability.

Now we will create an OpenCms Stack consisting of separate docker images, one for our mariadb / mysql database, one for apache and tomcat installed in the same container image and an additional HA Proxy image which will act as a template to create a Loadbalancer instance to balance our 2 OpenCms app server instances (apache2 + tomcat7).

At the End of this part you can use a single script to run your OpenCms Stack Cluster with a single command.

Our docker images looks like this:

The first image is our ubuntu base image, the second image is our mysql image, the third image has apache and tomcat installed and was used to create the opencms-tomcat-opencms image, which in turn was used to install opencms and connect it to the mysql database and create the opencms-stack image for instantiating other containers for our clustered OpenCms Stack connecting to our database container.

Some tips about your MySQL / MariaDB image for this installation, after installing MariaDB, you must allow your OpenCms Stack instances to connect to your mysql instance from their local remote IPs.

To do that, you shall run the following commands inside your DB container:

root@539df99fdb4d:/# mysql -uroot -p
Password: secret_pwd
MariaDB [(none)]> grant all privileges on opencms.* to 'root'@'%' identified by 'secret_pwd';

Note: the ‘%’ mark means from any ip, but if you want only to allow specific opencms instances to connect to your opencms DB, its highly recommended to run:

MariaDB [(none)]> grant all privileges on opencms.* to 'root'@'<specific-ip>' identified by 'secret_pwd';

To find the IP of your OpenCms containers, you shall run:

root@4848a41b40e8:/# ifconfig
eth0      Link encap:Ethernet  HWaddr e2:1d:e2:ba:cf:2a  
         inet addr:172.17.0.3  Bcast:0.0.0.0  Mask:255.255.0.0

If you have 2 apache-tomcat container instances with the IPs 172.17.0.3 and 172.17.0.4, then you must run:

MariaDB [(none)]> grant all privileges on opencms.* to 'root'@'172.17.0.3' identified by 'secret_pwd';
MariaDB [(none)]> grant all privileges on opencms.* to 'root'@'172.17.0.4' identified by 'secret_pwd';

Note: during the OpenCms base installation in apache-tomcat7-opencms container, you shall provide the IP address of your MySQL container in the OpenCms Installation Wizard, in my case it was 172.17.0.2.

root@539df99fdb4d:/# ifconfig       
eth0      Link encap:Ethernet  HWaddr d6:84:32:92:25:7c  
         inet addr:172.17.0.2  Bcast:0.0.0.0  Mask:255.255.0.0

The 172.17.0.2 is allways assigned to the first container as its private IP address, the second and third IPs in our case are 172.17.0.3 and 172.17.0.4 which belong to our apache-tomcat containers.

Now run 2 containers on port 81 and 82 from your VM host:

root@dokku:~# docker run -i -t -p 10.0.0.12:81:8080 opencms-stack /bin/bash
root@dokku:~# docker run -i -t -p 10.0.0.12:82:8080 opencms-stack /bin/bash

We will use these ports in our HAProxy config file in the next section to balance them via our HAProxy Container instance.

Installing HA Proxy

In this section we will install HA Proxy in a new container and configure it to connect to 2 OpenCms Stack containers.

First create a new ubuntu:precise container:

root@dokku:~# docker run -i -t ubuntu:precise /bin/bash

Run update and install HA Proxy:

root@355fbca7543e:/# apt-get update
root@355fbca7543e:/# apt-get install haproxy
root@355fbca7543e:/# nano /etc/default/haproxy

Set the enabled option to 1:

ENABLED=1

Configuring HAProxy (for more info please visit this page).

root@355fbca7543e:/# mv /etc/haproxy/haproxy.cfg{,.original}
root@355fbca7543e:/# nano /etc/haproxy/haproxy.cfg
root@355fbca7543e:/# nano /etc/haproxy/haproxy.cfg
global
 log 127.0.0.1 local0 notice
 maxconn 2000
 user haproxy
 group haproxy
defaults
 log global
 mode http
 option httplog
 option dontlognull
 retries 3
 option redispatch
 timeout connect 5000
 timeout client 10000
 timeout server 10000
listen OpenCms-Cluster 0.0.0.0:80
 mode http
 stats enable
 stats uri /haproxy?stats
 stats realm Strictly\ Private
 stats auth Admin:admin
 stats auth Admin2:admin2
 balance roundrobin
 option httpclose
 option forwardfor
 server opencms-stack1 10.0.0.12:81 check
 server opencms-stack2 10.0.0.12:82 check

Now you can start HAProxy and verify if all things are working as desired:

root@355fbca7543e:/# service haproxy start

You can access your HAProxy statistic page at:

http://10.0.0.12/haproxy?stats

Username and password are Admin / admin as defined in the haproxy.cfg config file.

You’re almost done, if all things done properly you might be able to access your balanced OpenCms Cluster through:

http://10.0.0.12/opencms/opencms/demo/

Note: to connect apache to tomcat and do some context rewriting to get rid of opencms/opencms context/servlet path you might want to refer to the following howto on OpenCms Wiki page.

Now let us destroy our containers running and play with docker and build a new OpenCms Stack.

In the above screenshot you can see our containers running, using “docker ps -a” and deleting them with “docker rm <conatinerid1> <containerid2> …”, and creating a new image from our running mysql container named mysql-opencms-stack.
Note: if you are running the whole thing on OpenStack or whatever Cloud OS, please make a SnapShot from your running VM, to don’t lose your hard work :-)

As you can see we have destroyed all of our containers, except mysql, but we can destroy it too, since we have created a new mysql image container based on our running mysql container which contains our opencms db base installation.

Ok, delete your mysql DB container, first exit from your container, mysql will become dead.

root@539df99fdb4d:/# exit

As you can see the STATUS has changed to “Exit 0”, which means mysql is not running.

Now before deleting the container you might want to see how we can start our dead process container and attach to it:

Start the container:

root@dokku:~# docker start 539df99fdb4d
539df99fdb4d

Verify if its running:

root@dokku:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS 
539df99fdb4d mysql:latest /bin/bash 25 hours ago Up About a minute

Attach into it (attach is like to ssh into the container):

root@dokku:~# docker attach 539df99fdb4d

As you can see, mysql is not running in our container

root@539df99fdb4d:/# service mysql status
 * MariaDB is stopped.

Start mysql and destroy the running container thereafter:

root@539df99fdb4d:/# /etc/init.d/mysql start
df: Warning: cannot read table of mounted file systems: No such file or directory
 * Starting MariaDB database server mysqld [ OK ] 
 * Checking for corrupt, not cleanly closed and upgrade needing tables.
root@539df99fdb4d:/# /etc/init.d/mysql status
 * /usr/bin/mysqladmin Ver 9.1 Distrib 10.0.9-MariaDB, for debian-linux-gnu on x86_64
Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.
Server version 10.0.9-MariaDB-1~precise-log
Protocol version 10
Connection Localhost via UNIX socket
UNIX socket /var/run/mysqld/mysqld.sock
Uptime: 22 sec

Destroy the running container (don’t do that in production environments).

root@dokku:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS 
539df99fdb4d mysql:latest /bin/bash 25 hours ago Up 11 minutes

root@dokku:~# docker rm -f 539df99fdb4d
539df99fdb4d

Note: with -f you’ll force to remove a running container.

Now we don’t have any containers running, “docker ps -a” shows our cleaned environment:

 

So lets create our OpenCms Stack from our container images.

First create a new mysql container, but this time we’ll run it as a daemon and attach to it and examine its ip address, but lets see how our images repository looks like:

root@dokku:~# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
mysql-opencms-stack latest 262c262b0db3 39 minutes ago 1.161 GB
haproxy-opencms-stack latest 9234fa617729 46 minutes ago 226.7 MB
opencms-stack latest 412f6cbca898 7 hours ago 912.2 MB
apache-tomcat7-opencms latest 0bdf9abbb3c4 16 hours ago 822 MB
apache2-tomcat7 latest a3b69852067f 17 hours ago 427.5 MB
mysql latest a741dbfee89f 25 hours ago 596.8 MB
ubuntu precise 9cd978db300e 6 weeks ago 204.4 MB

Now lets dive into some problems and learn how to resolve them:

Run a new mysql container as daemon:

root@dokku:~# docker run -v="$HOME/mysqldata":"/data" -i -t -d -p 3306 mysql-opencms-stack /bin/bash 
762cf4757fa27866be8e503099dbdcaae82ce59b9c56c1b4c8d69aee076b0acc

Verify if the container si running:

root@dokku:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
762cf4757fa2 mysql-opencms-stack:latest /bin/bash 4 seconds ago Up 4 seconds 0.0.0.0:49156->3306/tcp ecstatic_pike

Attach to it:

root@dokku:~# docker attach 762cf4757fa2

Try to start mysql:

root@762cf4757fa2:/# /etc/init.d/mysql start 
df: Warning: cannot read table of mounted file systems: No such file or directory
 * Starting MariaDB database server mysqld [fail]

Brilliant, mysql can’t start, what’s going wrong?

We created an image from our running mysql container and it seems that if mysqld.sock exists, then we can’t start mysql, so feel free to delete it:

root@762cf4757fa2::/# rm /var/run/mysqld/mysqld.sock
root@762cf4757fa2:/# /etc/init.d/mysql start
df: Warning: cannot read table of mounted file systems: No such file or directory
 * Starting MariaDB database server mysqld [ OK ] :-)

Examine the IP address:

root@762cf4757fa2:/# ifconfig 
eth0 Link encap:Ethernet HWaddr 72:67:2f:e8:0e:c3 
 inet addr:172.17.0.2 Bcast:0.0.0.0 Mask:255.255.0.0

Login to mysql DB and see if our database is still there:

root@762cf4757fa2:/# mysql -uroot -p
Enter password:
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| opencms |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)

Okay, then lets run 2 OpenCms Stack Tomcat Containers and start tomcat:

root@dokku:~# docker run -i -t -d -p 10.0.0.12:81:8080 opencms-stack /bin/bash
d3a4530b6205a4bff94aacbc28a1fbeeda519ef27e026b5f6be90b8975c0
root@dokku:~# docker run -i -t -d -p 10.0.0.12:82:8080 opencms-stack /bin/bash
dcb39f82ddc58d7220d0838a6ab518e6ce101157b8a01d00a973e42d9e6da07
root@dokku:~# docker attach 1135d3a4530b
→ enter twice!
root@1135d3a4530b:/# /etc/init.d/tomcat7 status
 * Tomcat servlet engine is not running, but pid file exists.
root@1135d3a4530b:/# /etc/init.d/tomcat7 start 
 * Starting Tomcat servlet engine tomcat7

Verify if OpenCms is running:

root@dokku:~# docker attach 1135d3a4530b
root@5dcb39f82ddc:/# tail -f /var/lib/tomcat7/webapps/opencms/WEB-INF/logs/opencms.log 
….
23 Mar 2014 18:27:52,347 INFO [ org.opencms.main.OpenCmsCore:1608] . OpenCms is running! : Total startup time was 00:00:05
23 Mar 2014 18:27:52,347 INFO [ org.opencms.main.OpenCmsCore:1611] . ...............................................................
23 Mar 2014 18:27:52,347 INFO [ org.opencms.main.OpenCmsCore:1612] .

Note: OpenCms startup time was 5 Seconds :-)
As the last step we run our HAProxy container:

root@dokku:~# docker run -i -t -d -p 10.0.0.12:80:80 haproxy-opencms-stack /bin/bash
root@dokku:~# docker attach 11ef817b76bf
root@11ef817b76bf:/# /etc/init.d/haproxy start 

http://10.0.0.12/opencms/opencms/demo/about/index.html
Enjoy again and again and again :-)

Run you OpenCms Stack Cluster with a single command

To auto start tomcat and HAProxy, you can easily extend the bash.bashrc file of your haproxy-opencms-stack and opencms-stack image by adding the the start commands to them:

root@7201deadfb9b:/# vi /etc/bash.bashrc 

add this line of the beginning:

sh /etc/init.d/tomcat7 start

and in the haproxy image add the start command to your bash.bashrc file:

sh /etc/init.d/haproxy start

After doing that create a new images from your modified containers and delete the old images.

Create a file opencms-stack.sh on your VM and copy the following lines to it (adapt the IP address):

root@dokku:~# cat opencms-stack.sh 
docker run -i -t -d -p 10.0.0.12:81:8080 opencms-stack-image /bin/bash
docker run -i -t -d -p 10.0.0.12:82:8080 opencms-stack-image /bin/bash
docker run -i -t -d -p 10.0.0.12:80:80 haproxy-opencms-stack-image /bin/bash

Now you can source the file and run your OpenCms Stack Cluster with a single command:

root@dokku:~# source opencms-stack.sh 
9fa484ce71d48330d8b59c8d61fba640a2253d5207321bf8337b34e4b91eb212
7201deadfb9bd96638bd62a684bd7dfaa5ad2e1390b487a93a53883cb1967344
dfcb08c7c278a53770c1f23a57d07339d655826683c873b6d1910acff5137775

Enjoy :-)