= Linux Container (LXC) Walk-through =

+What this tutorial is:+ This tutorial demonstrates how pacemaker_remote can be used with Linux containers (managed by libvirt-lxc) to run cluster resources in an isolated environment.

+What this tutorial is not:+  This tutorial is not a realistic deployment scenario.  The steps shown here are meant to introduce users to the concept of managing Linux container environments with Pacemaker.

== Step 1: Setup LXC Host ==

This tutorial was tested with Fedora 18.   Anything that is capable of running libvirt and pacemaker v1.1.10 or greater will do though.  An installation guide for installing Fedora 18 can be found here, http://docs.fedoraproject.org/en-US/Fedora/18/html/Installation_Guide/.

Fedora 18 (or similar distro) host preparation steps.

=== SElinux and Firewall Rules ===
In order to simply this tutorial we will disable the selinux and the firewall on the host.
WARNING: These actions pose a significant security issues to machines exposed to the outside world.  Basically, just don't do this on your production system.
[source,C]
----
# setenforce 0
# sed -i.bak "s/SELINUX=enforcing/SELINUX=permissive/g" /etc/selinux/config
# firewall-cmd --add-port 3121/tcp --permanent

# systemctl disable iptables.service
# systemctl disable ip6tables.service
# rm '/etc/systemd/system/basic.target.wants/iptables.service'
# rm '/etc/systemd/system/basic.target.wants/ip6tables.service'
# systemctl stop iptables.service
# systemctl stop ip6tables.service
----

=== Install Cluster Software on Host ===

[source,C]
----
# yum install -y pacemaker pacemaker-remote corosync pcs resource-agents
----

=== Configure Corosync ===

Running the command below will attempt to detect the network address corosync should bind to.

[source,C]
----
# export corosync_addr=`ip addr | grep "inet " | tail -n 1 | awk '{print $4}' | sed s/255/0/g`
----

Display and verify the address is correct

[source,C]
----
# echo $corosync_addr
----

In most cases the address will be 192.168.1.0 if you are behind a standard home router.

Now copy over the example corosync.conf.  This code will inject your bindaddress and enable the vote quorum api which is required by pacemaker.

[source,C]
----
# cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf
# sed -i.bak "s/.*\tbindnetaddr:.*/bindnetaddr:\ $corosync_addr/g" /etc/corosync/corosync.conf
# cat << END >> /etc/corosync/corosync.conf
quorum {
           provider: corosync_votequorum
           expected_votes: 2
}
END
----

=== Verify Cluster ===

Start the cluster

[source,C]
----
# pcs cluster start
----

Verify corosync membership

[source,C]
----
# pcs status corosync

Membership information
    Nodeid      Votes Name
1795270848          1 example-host (local)
----

Verify pacemaker status.  At first the 'pcs cluster status' output will look like this.

[source,C]
----
# pcs status

 Last updated: Thu Mar 14 12:26:00 2013
 Last change: Thu Mar 14 12:25:55 2013 via crmd on example-host
 Stack: corosync
 Current DC:
 Version: 1.1.10
 1 Nodes configured, unknown expected votes
 0 Resources configured.
----

After about a minute you should see your host as a single node in the cluster.

[source,C]
----
# pcs status

 Last updated: Thu Mar 14 12:28:23 2013
 Last change: Thu Mar 14 12:25:55 2013 via crmd on example-host
 Stack: corosync
 Current DC: example-host (1795270848) - partition WITHOUT quorum
 Version: 1.1.8-9b13ea1
 1 Nodes configured, unknown expected votes
 0 Resources configured.

 Online: [ example-host ]
----

Go ahead and stop the cluster for now after verifying everything is in order.

[source,C]
----
# pcs cluster stop
----

== Step 2: Setup LXC Environment ==

=== Install Libvirt LXC software ===

[source,C]
----
# yum install -y libvirt libvirt-daemon-lxc wget
# systemctl enable libvirtd
----

At this point, restart the host.

=== Generate Libvirt LXC domains ===

I've attempted to simply this tutorial by creating a script to auto generate the libvirt-lxc xml domain definitions.

Download the script to whatever directory you want the containers to live in.  In this example I am using the /root/lxc/ directory.

[source,C]
----
# mkdir /root/lxc/
# cd /root/lxc/
# wget https://raw.github.com/davidvossel/pcmk-lxc-autogen/master/lxc-autogen
# chmod 755 lxc-autogen
----

Now execute the script.

[source,C]
----
# ./lxc-autogen
----

After executing the script you will see a bunch of directories and xml files are generated. Those xml files are the libvirt-lxc domain definitions, and the directories are used as some special mount points for each container.  If you open up one of the xml files you'll be able to see how the cpu, memory, and filesystem resources for the container are defined.  You can use the libvirt-lxc driver's documentation found here, http://libvirt.org/drvlxc.html, as a reference to help understand all the parts of the xml file.  The lxc-autogen script is not complicated and is worth exploring in order to grasp how the environment is generated.

It is worth noting that this environment is dependent on use of libvirt's default network interface. Verify the commands below look the same as your environment.  The default network address 192.168.122.1 should have been generated by automatically when you installed the virtualization software.

[source,C]
----
# virsh net-list
Name                 State      Autostart     Persistent
________________________________________________________
default              active     yes           yes

# virsh net-dumpxml default | grep -e "ip address="
<ip address='192.168.122.1' netmask='255.255.255.0'>

----

=== Generate the Authkey ===

Generate the authkey used to secure connections between the host and the lxc guest pacemaker_remote instances.  This is sort of a funny case because the lxc guests and the host will share the same key file in the /etc/pacemaker/ directory.  If in a different deployment where the lxc guests do not share the host's /etc/pacemaker directory, this key will have to be copied into each lxc guest.

[source,C]
----
# dd if=/dev/urandom of=/etc/pacemaker/authkey bs=4096 count=1
----

== Step 3: Integrate LXC guests into Cluster. ==

=== Start Cluster ===
On the host, start pacemaker.

[source,C]
----
# pcs cluster start
----

Wait for the host to become the DC. The output of 'pcs status' should look similar to this after about a minute.

[source,C]
----
Last updated: Thu Mar 14 16:41:22 2013
Last change: Thu Mar 14 16:41:08 2013 via crmd on example-host
Stack: corosync
Current DC: example-host (1795270848) - partition WITHOUT quorum
Version: 1.1.10
1 Nodes configured, unknown expected votes
0 Resources configured.


Online: [ example-host ]
----

Now enable the cluster to work without quorum or stonith.  This is required just for the sake of getting this tutorial to work with a single cluster-node.

[source,C]
----
# pcs property set stonith-enabled=false
# pcs property set no-quorum-policy=ignore
----

=== Integrate LXC Guests as remote-nodes ===

If you ran the 'lxc-autogen' script with default parameters, 3 lxc domain definitions were created as .xml files.  If you used the same directory I used for the lxc environment, the config files will be located in /root/lxc.  Replace the 'config' parameters in the following pcs commands if yours should be different.

The pcs commands below each configure a lxc guest as a remote-node in pacemaker.  Behind the scenes each lxc guest is launching an instance of pacemaker_remote allowing pacemaker to integrate the lxc guests as remote-nodes.  The meta-attribute 'remote-node=<node-name>' used in each command is what tells pacemaker that the lxc guest is both a resource and a remote-node capable of running resources.  In this case, the 'remote-node' attribute also indicates to pacemaker that it can contact each lxc's pacemaker_remote service by using the remote-node name as the hostname.  If you look in the /etc/hosts/ file you will see entries for each lxc guest. These entries were auto-generated earlier by the 'lxc-autogen' script.

[source,C]
----
# pcs resource create container1 VirtualDomain force_stop="true" hypervisor="lxc:///" config="/root/lxc/lxc1.xml" meta remote-node=lxc1
# pcs resource create container2 VirtualDomain force_stop="true" hypervisor="lxc:///" config="/root/lxc/lxc2.xml" meta remote-node=lxc2
# pcs resource create container3 VirtualDomain force_stop="true" hypervisor="lxc:///" config="/root/lxc/lxc3.xml" meta remote-node=lxc3
----


After creating the container resources you 'pcs status' should look like this.

[source,C]
----
Last updated: Mon Mar 18 17:15:46 2013
Last change: Mon Mar 18 17:15:26 2013 via cibadmin on guest1
Stack: corosync
Current DC: example-host (175810752) - partition WITHOUT quorum
Version: 1.1.10
4 Nodes configured, unknown expected votes
6 Resources configured.

Online: [ example-host lxc1 lxc2 lxc3 ]

Full list of resources:

 container3	(ocf::heartbeat:VirtualDomain):	Started example-host
 container1	(ocf::heartbeat:VirtualDomain):	Started example-host
 container2	(ocf::heartbeat:VirtualDomain):	Started example-host
----


=== Starting Resources on LXC Guests ===

Now that the lxc guests are integrated into the cluster, lets generate some Dummy resources to run on them.

Dummy resources are real resource agents used just for testing purposes. They actually execute on the node they are assigned to just like an apache server or database would, except their execution just means a file was created. When the resource is stopped, that the file it created is removed.

[source,C]
----
# pcs resource create FAKE1 ocf:pacemaker:Dummy
# pcs resource create FAKE2 ocf:pacemaker:Dummy
# pcs resource create FAKE3 ocf:pacemaker:Dummy
# pcs resource create FAKE4 ocf:pacemaker:Dummy
# pcs resource create FAKE5 ocf:pacemaker:Dummy
----


After creating the Dummy resources you will see that the resource got distributed among all the nodes.  The 'pcs status' output should look similar to this.

[source,C]
----
Last updated: Mon Mar 18 17:31:54 2013
Last change: Mon Mar 18 17:31:05 2013 via cibadmin on example-host
Stack: corosync
Current DC: example=host (175810752) - partition WITHOUT quorum
Version: 1.1.10
4 Nodes configured, unknown expected votes
11 Resources configured.


Online: [ example-host lxc1 lxc2 lxc3 ]

Full list of resources:

 container3	(ocf::heartbeat:VirtualDomain):	Started example-host
 container1	(ocf::heartbeat:VirtualDomain):	Started example-host
 container2	(ocf::heartbeat:VirtualDomain):	Started example-host
 FAKE1	(ocf::pacemaker:Dummy):	Started lxc1 
 FAKE2	(ocf::pacemaker:Dummy):	Started lxc2 
 FAKE3	(ocf::pacemaker:Dummy):	Started lxc3 
 FAKE4	(ocf::pacemaker:Dummy):	Started lxc1 
 FAKE5	(ocf::pacemaker:Dummy):	Started lxc2 
----

To witness that Dummy agents are running within the lxc guests browse one of the lxc domain's filesystem folders.  Each lxc guest has a custom mount point for the '/var/run/'directory, which is the location the Dummy resources write their state files to.

[source,C]
----
# ls lxc1-filesystem/var/run/
Dummy-FAKE4.state  Dummy-FAKE.state
----

If you are curious, take a look at lxc1.xml to see how the filesystem is mounted.

=== Testing LXC Guest Failure ===

You will be able to see each pacemaker_remoted process running in each lxc guest from the host machine.

[source,C]
----
# ps -A | grep -e pacemaker_remote*
 9142 pts/2    00:00:00 pacemaker_remot
10148 pts/4    00:00:00 pacemaker_remot
10942 pts/6    00:00:00 pacemaker_remot
----

In order to see how the cluster reacts to a failed lxc guest. Try killing one of the pacemaker_remote instances.

[source,C]
----
# kill -9 9142
----

After a few moments the lxc guest that was running that instance of pacemaker_remote will be recovered along with all the resources running within that container.
