Setting up RozoFS¶

Mutli-link Configuration¶

That configuration is the recommended one for RozoFS where there is one separate Vlan per physical port. The following diagram describes how storage nodes are connected toward the ToR switches. It is assumed that the RozoFS clients reside on nodes that are connected towards the northbound of the ToR SAN switches

LACP Configuration¶

In that case, the ports dedicated to the SAN (RozoFS and Storage) are grouped in one or two LACP groups, depending if we want to separate the RozoFS and Storage traffic or not. They can be either reside on the same or different VLANs.

Prerequisites¶

The following section informs you about system requirements, and some prerequisites for setting up a High Availability rozofs-exportd service. It also includes recommendations for this setup.

Hardware requirements

The following list specifies hardware requirements:

• At least two Linux node.
• At least two TCP/IP communication media per node.
• A STONITH mechanism. A STONITH device is a power switch which the cluster uses to reset nodes that are thought to be dead or behaving in a strange manner. This is the only reliable way to ensure that no data corruption is performed by nodes that hang and only appear to be dead.
• One empty data block device by node (partition or complete hard disk, Logical Volume LVM..) separate from OS Hard Disk.

Software requirements

• One Linux OS, Rozo Systems has only deployed this setup on the following OS:
  • CentOS 6.5, 6.6 and 7
  • Debian 7 and 8
• Enable Network Time Protocol daemon: If nodes are not synchronized, log files and cluster reports are very hard to analyze.

Other recommendations

For a useful High Availability setup, consider the following recommendations:

• Redundant Communication Paths: it’s strongly recommended to set up cluster communication (corosync) via two or more redundant paths. This can be done via:
  • Network Device Bonding.
  • A second communication channel in Corosync.
• We strongly recommend multiple STONITH devices per node.

The following diagram describes a ideal network configuration for a exportd HA setup:

• DRBD and Corosync (Primary interface: ring 0) use a bonding network device
• A second network interface (ring 1) is used by corosync if the first interface failed.
• 2 STONITH devices are used:
  • One with the Light Out Management interface (ilo, idrac, IPMI...)
  • One with a Switched PDU

Meta-data Replication with DRBD 8.4¶

About DRBD

DRBD replicates data from the primary device to the secondary device in a way which ensures that both copies of the data remain identical. Think of it as a networked RAID 1. It mirrors data in real-time, so its replication occurs continuously. Applications do not need to know that in fact their data is stored on different disks.

The following example uses two servers named node-1 and node-2, and the DRBD resource named r0. Each node use the device /dev/sdd for low level device. It sets up node-1 as the primary node. Be sure to modify the instructions relative to your own configuration.

Installing DRBD

On both servers, install DRBD packages

For installing DRBD with apt :

apt-get install drbd8-utils

For install DRBD with yum:

yum install kmod-drbd84 drbd84-utils

Configuring a DRBD resource

The DRBD configuration files are stored in the directory /etc/drbd.d/. There are two configuration files which are created:

• /etc/drbd.d/r0.res corresponds to the configuration for resource r0;
• /etc/drbd.d/global_common.conf corresponds to the global configuration of DRBD.

Create files /etc/drbd.d/global_common.conf and /etc/drbd.d/r0.res on node-1, changes the lines according to your parameters, and save them.

Examples of configuration files for DRBD 8.4:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

 global {
   usage-count no;
 }

 common {

   handlers {
       # handlers
       pri-lost "/usr/lib/drbd/notify-pri-lost.sh; \
                 /usr/lib/drbd/notify-emergency-reboot.sh; \
                 echo b > /proc/sysrq-trigger ; \
                 reboot -f";
       pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; \
                          /usr/lib/drbd/notify-emergency-reboot.sh; \
                          echo b > /proc/sysrq-trigger; \
                          reboot -f";
       pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; \
                          /usr/lib/drbd/notify-emergency-reboot.sh; \
                          echo b > /proc/sysrq-trigger; \
                          reboot -f";
       fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
       after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
       split-brain "/usr/lib/drbd/notify-split-brain.sh root";
   }

   disk {
       on-io-error detach; # If a hard drive fails
       # The handler is supposed to reach the other node over
       # alternative communication paths and call 'drbdadm outdate res' there
       fencing resource-only; # 2 rings should be configured in corosync
       ## Tuning recommendations
       al-extents 3389;
   }

   net {
       verify-alg crc32c;
       csums-alg crc32c;
       rr-conflict call-pri-lost;
       ### Automatic split brain recovery policies
       after-sb-0pri discard-zero-changes ;
       after-sb-1pri call-pri-lost-after-sb ;
       after-sb-2pri call-pri-lost-after-sb ;
       always-asbp;
       ## Tuning recommendations
       max-buffers 8000;
       max-epoch-size 8000;
       sndbuf-size 0;
       unplug-watermark 16;
   }
 }

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

 resource r0 {

   protocol C;
   on node-1 {
     device      /dev/drbd0; # Block device name
     disk        /dev/sdd;   #  Lower level device
     address     192.168.1.1:7788; # IP address:port for data transfer
     meta-disk   internal;  # Meta-data are stored on lower level device
   }
   on node-2 {
     device      /dev/drbd0;
     disk        /dev/sdd;
     address     192.168.1.2:7788;
     meta-disk   internal;
   }
 }

This file configure a DRBD resource named r0 which uses an underlying local disk named /dev/sdd on both nodes node-1 and node-2. In this example, we configure the resource to use internal meta-data (means that DRBD stores its meta data on the same physical lower-level device as the actual production data) and it uses TCP port 7788 for its network connections, and binds to the IP addresses 192.168.1.1 and 192.168.1.2, respectively. This resource is configured to use fully synchronous replication (protocol C).

Copy DRBD configuration files manually to the other node (node-2):

scp /etc/drbd.d/* node-2:/etc/drbd.d/

Enabling the DRBD resource

Each of the following steps must be completed on both nodes.

Initializes the DRBD meta-data:

drbdadm -- --ignore-sanity-checks create-md r0

Attach resource r0 to the backing device, set the replication parameters and connect the resource to its peer:

drbdadm up r0

Start the resync process and put the device into the primary role (node-1 in this case) by entering the following command only on node-1:

drbdadm --force primary r0

Creating a file system

Create desired file system on top of your DRBD device (for example ext4):

mkfs.ext4 -b 4096 -i 4096 -I 128 /dev/drbd0

Testing the metadata filesystem

If the install and configuration procedures worked as expected, you are ready to run a basic test of the DRBD functionality. Create a mount point on node-1, such as /srv/rozofs/exports:

mkdir -p /srv/rozofs/exports

Mount the DRBD device:

mount /dev/drbd0 /srv/rozofs/exports

In the following section, we will configure the management of high availability with Pacemaker. So it will be necessary to have the rozofs-exportd configuration file on both servers and this file should be identical. For that we will move this configuration file to meta-data filesystem and create symbolic links on each node.

On node-1 (the current primary node):

mv /etc/rozofs/export.conf /srv/rozofs/exports/export.conf

On node-1 and node-2:

ln -sf /srv/rozofs/exports/export.conf /etc/rozofs/export.conf

Unmount the DRBD device on node-1:

umount /srv/rozofs/exports

To verify that synchronization is performed:

cat /proc/drbd
version: 8.4.3 (api:1/proto:86-101)
srcversion: 1A9F77B1CA5FF92235C2213
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:64 nr:0 dw:24 dr:93981 al:1 bm:5 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

The two resources are now synchronized (UpToDate). The initial synchronization is performed, it is necessary to stop the DRBD service and remove the link for the initialization script not to start the service automatically DRBD. The service will be controlled by the Pacemaker service.

Disable DRBD and rozofs-exportd init script on each meta-data node (depending on your distribution):

Debian Wheezy, CentOS 6 (system V):

/etc/init.d/drbd stop
/etc/init.d/rozofs-exportd stop
insserv -vrf drbd rozofs-exportd

Debian Jessie, CentOS 7 (systemd):

systemctl stop rozofs-exportd drbd
systemctl disable rozofs-exportd drbd

It’s also necessary to create the DRBD device mountpoint directory on node-2 :

mkdir -p /srv/rozofs/exports

High Availability with Pacemaker¶

Pacemaker is an open-source high availability resource management tool suitable for clusters of Linux machines. This tool can detect machine failures with a communication system based on an exchange of UDP packets and migrate services (resource) from one server to another.

The configuration of Pacemaker can be done with the crmsh utility (Cluster Management Shell). It allows you to manage different resources and propagates changes on each server automatically. The creation of a resource is done with an entry named primitive in the configuration file. This primitive uses a script corresponding to the application to be protected.

In the case of the platform, Pacemaker manages the following resources:

• rozofs-exportd service;
• Mounting the file system used to store meta-data;
• DRBD resource (r0), roles (master or slave);
• Server connectivity.

The following diagram describes the different resources configured and controlled via Pacemaker. In this case, 2 servers are configured and node-1 is the master server.

Installing Corosync and Pacemaker packages

On both servers, install the following packages:

• pacemaker
• crmsh
• corosync
• fence-agents, ipmitool (STONITH)
• resource-agents
• fping
• rozofs-exportd

Create Cluster Authorization Key

The first component to configure is Corosync. It manages the infrastructure of the cluster, i.e. the status of nodes and their operation. For this, we must generate an authentication key that is shared by all the machines in the cluster. The corosync-keygen utility can be use to generate this key and then copy it to the other nodes.

Create key on node-1:

corosync-keygen

Copy the key manually to the other node:

scp /etc/corosync/authkey root@node-2:/etc/corosync/authkey

Configuring Corosync

Besides copying the key, you also have to modify the corosync configuration file which stored in /etc/corosync/corosync.conf.

Edit your corosync.conf with the following (example with unicast and corosync version 2):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

 totem {
     version: 2

     # How long before declaring a token lost (ms)
     token: 6000
     # How many token retransmits before forming a new configuration
     token_retransmits_before_loss_const: 10

     clear_node_high_bit: yes

     # This specifies the mode of redundant ring
     rrp_mode: passive

     # The following values need to be set based on your environment
     interface {
         ringnumber: 0
         bindnetaddr: 192.168.1.0
         mcastport: 5405
     }

     interface {
         ringnumber: 1
         bindnetaddr: 192.168.2.0
         mcastport: 5407
     }
     transport: udpu
 }

 quorum {
     provider: corosync_votequorum
     # Only valid with 2 nodes
     expected_votes: 2
     two_node: 1
 }

 nodelist {
     node {
         ring0_addr: 192.168.1.1
         ring1_addr: 192.168.2.1
         name: node-1
         nodeid: 1
     }
     node {
         ring0_addr: 192.168.1.2
         ring1_addr: 192.168.2.2
         name: node-2
         nodeid: 2
     }
 }

 logging {
     fileline: off
     to_stderr: no
     to_logfile: no
     #logfile: /var/log/corosync/corosync.log
     to_syslog: yes
     syslog_facility: daemon
     debug: off
     timestamp: on
     logger_subsys {
       subsys: QUORUM
       debug: off
     }
 }

Copy the corosync.conf manually to the other node:

scp /etc/corosync/corosync.conf root@node-2:/etc/corosync/corosync.conf

By default, the Corosync service is disabled. On both servers, change that by editing /etc/default/corosync and change the value of START to yes if needed:

1

START=yes

Starting Corosync

Corosync is started as a regular system service. Depending on your distribution, it may ship with a LSB init script, an upstart job, or a systemd unit file. Either way, the service is usually named corosync.

Examples:

/etc/init.d/corosync start
service corosync start
start corosync
systemctl start corosync

You can now check the ring status manually with corosync-cfgtool:

corosync-cfgtool -s
Printing ring status.
Local node ID 1
RING ID 0
        id      = 192.168.1.1
        status  = ring 0 active with no faults
RING ID 1
        id      = 192.168.2.1
        status  = ring 1 active with no faults

Configuring Pacemaker

Once the Pacemaker cluster is set up and before configuring the different resources and constraints of the Pacemaker cluster, it is necessary to have the OCF scripts for exportd on each server. This script is enable to start, stop and monitor the exportd daemon. This script is installed by default with the rozofs-exportd package (/usr/lib/ocf/resource.d/heartbeat/exportd).

To set the cluster properties, create cluster resources configuration file crm.conf and changes the lines according to your parameters, and save it.

property stonith-enabled="false" no-quorum-policy="ignore"

rsc_defaults migration-threshold=10 failure-timeout=60

primitive p-ping ocf:pacemaker:ping params  \
host_list="192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4" \
multiplier="100" dampen="5s" \
op start timeout="60" op monitor interval="5s" timeout="60"

clone c-ping p-ping meta interleave="true"

primitive p-drbd-r0 ocf:linbit:drbd params drbd_resource="r0" \
adjust_master_score="0 10 1000 10000" op start timeout="240" \
op stop timeout="100" op notify interval="0" timeout="90" \
op monitor interval="10" timeout="20" role="Master" \
op monitor interval="20" timeout="20" role="Slave"

ms ms-drbd-r0 p-drbd-r0 meta master-max="1" master-node-max="1" \
clone-max="2" clone-node-max="1" notify="true" \
globally-unique="false" interleave="true"

primitive p-fs-exportd ocf:heartbeat:Filesystem params device="/dev/drbd0" \
directory="/srv/rozofs/exports" fstype="ext4" options="user_xattr,noatime" \
op start timeout="60" op stop timeout="60" op monitor interval="10"

primitive exportd-rozofs ocf:heartbeat:exportd params  \
conffile="/etc/rozofs/export.conf" op monitor interval="10s"

group grp-exportd p-fs-exportd exportd-rozofs

colocation c-grp-exportd-on-drbd-r0 inf: grp-exportd ms-drbd-r0:Master

order o-drbd-r0-before-grp-exportd inf: ms-drbd-r0:promote grp-exportd:start

location loc-ms-drbd-r0-needs-ping ms-drbd-r0 \
rule -inf: not_defined pingd or pingd lt 200

Load this configuration with the following command:

crm configure load replace crm.conf

Once all the primitives and constraints are loaded, it is possible to check the correct operations of the cluster with the following command:

crm_mon -1

Last updated: Mon Jun  6 09:46:39 2016
Last change: Wed Jun  1 15:14:04 2016 by root via cibadmin on node-2
Stack: corosync
Current DC: node-1 (version 1.1.14-70404b0) - partition with quorum
2 nodes and 6 resources configured

Online: [ node-1 node-2 ]

 Resource Group: grp-exportd
     p-fs-exportd       (ocf::heartbeat:Filesystem):    Started node-1
     exportd-rozofs     (ocf::heartbeat:exportd):       Started node-1
 Master/Slave Set: ms-drbd-r0 [p-drbd-r0]
     Masters: [ node-1 ]
     Slaves: [ node-2 ]
 Clone Set: c-ping [p-ping]
     Started: [ node-1 node-2 ]

Adding a STONITH resource (example with IPMI)

Before using STONITH with IPMI, you must configure the network used by the IPMI devices and the IPMI devices on each node.

After doing this you can add the STONITH resources in the Pacemaker cluster configuration:

crm configure primitive fence-node-1 stonith:fence_ipmilan params \
pcmk_host_list="node-1" ipaddr="192.168.100.1" \
login="login" passwd="passwd" lanplus="true" \
pcmk_reboot_action="off" op monitor interval="3600s"

crm configure location loc-fence-node-1 fence-node-1 -inf: node-1

crm configure primitive fence-node-2 stonith:fence_ipmilan params \
pcmk_host_list="node-2" ipaddr="192.168.100.2" \
login="login" passwd="passwd" lanplus="true" \
pcmk_reboot_action="off" op monitor interval="3600s"

crm configure location loc-fence-node-2 fence-node-2 -inf: node-2

Set the global cluster option stonith-enabled to true:

crm configure property stonith-enabled=true

Testing your cluster configuration

Now, you can testing your cluster configuration.

Migrating rozofs-exportd resource:

# Get current location of exportd-rozofs
crm resource status exportd-rozofs
  resource exportd-rozofs is running on: node-1
# Migrate exportd-rozofs resource to node-2
crm resource migrate exportd-rozofs node-2
# Get current location of exportd-rozofs
crm resource status exportd-rozofs
  resource exportd-rozofs is running on: node-2
# Very important, remove the new constraint
crm resource unmigrate exportd-rozofs

Simulate rozofs-exportd failure:

# Manually stop rozofs-exportd service
# on the node who the resource is currently running!
# (without using crm utility)
/etc/init.d/rozofs-exportd stop

# Check that the resource agent has restarted `rozofs-exportd` service.
crm_mon -rfn1

Node node-2: online
        exportd-rozofs  (ocf::heartbeat:exportd) Started
        p-ping:1        (ocf::pacemaker:ping) Started
        p-fs-exportd    (ocf::heartbeat:Filesystem) Started
        p-drbd-r0:1     (ocf::linbit:drbd) Master
Node node-1: online
        p-ping:0        (ocf::pacemaker:ping) Started
        p-drbd-r0:0     (ocf::linbit:drbd) Slave

Inactive resources:

Migration summary:
* Node node-2:
   exportd-rozofs: migration-threshold=10 fail-count=1 last-failure='Tue Jun 28 08:55:46 2016'
* Node node-1:

Failed actions:
    exportd-rozofs_monitor_10000 (node=node-2, call=22, rc=7, status=complete): not running

Testing stonith:

crm node fence <nodename>
killall -9 corosync