Following on from the previous article, this describes a corosync configuration for three appliances configured together in a 'triangle'. OSPF/BGP is running on each appliance. With this routing configuration, I am able to apply an ip address to the loopback interface, and make each of those addresses mutually reachable from each appliance.
I think most corosync examples make the assumption that all nodes are within the same segment. This then suggests a multicast solution. As I am using routing between each appliance, I need a unicast solution.
The following is an example configuration file for the second of three nodes / appliances. Notice the bind_addr is for the loopback address, and a complete list of all three nodes taking part in the quorum. There is a 'mcastport' listed, but because of 'transport: udpu', unicast is actually used from that port number.
# described in the corosync.conf.5 manual page
totem {
version: 2
cluster_name: sheepdog
# How long before declaring a token lost (ms)
token: 3000
# How many token retransmits before forming a new configuration
token_retransmits_before_loss_const: 10
# Limit generated nodeids to 31-bits (positive signed integers)
clear_node_high_bit: yes
# Valid values for crypto_cipher are none (no encryption), aes256, aes192,
# aes128 and 3des. Enabling crypto_cipher, requires also enabling of
# crypto_hash.
crypto_cipher: none
# Valid values for crypto_hash are none (no authentication), md5, sha1,
# sha256, sha384 and sha512.
crypto_hash: none
# udp only, no multicast
transport: udpu
ip_version: ipv4
# bump up to 9000 after testing
netmtu: 1500
# interface: define at least one interface to communicate
# over. If you define more than one interface stanza, you must
# also set rrp_mode.
rrp_mode: passive
interface {
# Rings must be consecutively numbered, starting at 0.
ringnumber: 0
# configure the *host* address of the interface
# instead:
bindnetaddr: 172.16.1.22
mcastport: 5405
}
}
nodelist {
node {
ring0_addr: 172.16.1.21
nodeid: 21
}
node {
ring0_addr: 172.16.1.22
nodeid: 22
}
node {
ring0_addr: 172.16.1.23
nodeid: 23
}
logging {
# Log the source file and line where messages are being
# generated. When in doubt, leave off. Potentially useful for
# debugging.
fileline: off
# Log to standard error. When in doubt, set to no. Useful when
# running in the foreground (when invoking "corosync -f")
to_stderr: no
# Log to a log file. When set to "no", the "logfile" option
# must not be set.
to_logfile: yes
logfile: /var/log/corosync/corosync.log
# Log to the system log daemon. When in doubt, set to yes.
to_syslog: yes
# Log with syslog facility daemon.
syslog_facility: daemon
# Log debug messages (very verbose). When in doubt, leave off.
debug: on
# Log messages with time stamps. When in doubt, set to on
# (unless you are only logging to syslog, where double
# timestamps can be annoying).
timestamp: on
logger_subsys {
subsys: QUORUM
debug: off
}
}
quorum {
# Enable and configure quorum subsystem (default: off)
# see also corosync.conf.5 and votequorum.5
provider: corosync_votequorum
expected_votes: 2
}
A few troubleshooting commands I used:
# corosync-cmapctl | grep member
runtime.totem.pg.mrp.srp.members.21.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.21.ip (str) = r(0) ip(172.16.1.21)
runtime.totem.pg.mrp.srp.members.21.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.21.status (str) = joined
runtime.totem.pg.mrp.srp.members.22.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.22.ip (str) = r(0) ip(172.16.1.22)
runtime.totem.pg.mrp.srp.members.22.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.22.status (str) = joined
runtime.totem.pg.mrp.srp.members.23.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.23.ip (str) = r(0) ip(172.16.1.23)
runtime.totem.pg.mrp.srp.members.23.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.23.status (str) = joined
# corosync-cfgtool -s
Printing ring status.
Local node ID 22
RING ID 0
id = 172.16.1.22
status = ring 0 active with no faults
