关于carp+pf+pfsync的群集, 网上大多是OB的资料, 于是找到些机器将其移到FB上来. 实验上是成功,可能存在一些问题, 请大家来发现和指正!!!!

硬件: 
服务器                5台(实际只用了4台, 2台做LoadBalance(主/备各一,每台服务器至少需要三块网卡), 2台做应用服务.)
cisco2950        2台

系统: FreeBSD 7.0 release(最小化安装)

我部署环境的测试地址(不能长期保留, 群集部署是在内网中,所以外网通过重定向来carp0上的): 
测试地址就不放上来了

一 网络拓扑:
外网: 192.168.1.0/24
内网: 192.168.10.0/24
HA专用: 10.10.10.250 / 10.10.10.251



二 Load Balance配置:

A.内核设置: 
master/slave的内核配置是一样的
device carp
device pf        #启动虚拟网络设备来记录流量(经由 bpf)
device pflog     #启动虚拟网络设备来监视网络状态
device pfsync

options         ALTQ
options         ALTQ_CBQ        # 基于分类的排列 (CBQ)
options         ALTQ_RED        # 随机先期检测 (RED)
options         ALTQ_RIO        # 对进入和发出的包进行 RED
options         ALTQ_HFSC       # 带等级的包调度器 (HFSC)
options         ALTQ_PRIQ       # 按优先级的排列 (PRIQ)
options         ALTQ_NOPCC      # 在联编 SMP 内核时必须使用，禁止读时钟

重编内核. 

B.配置rc.conf
这里只写与本章节相关的配置项了. 
master部分:
gateway_enable="YES"
defaultrouter="192.168.1.1"
hostname="master.cluster.org"
cloned_interfaces="carp0 carp1"

# External Public Interface (for the secondary firewall use a different public ip.)
ifconfig_em0="inet 192.168.1.52 netmask 255.255.255.0"
# External Public Carp Interface
#ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24"
ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24 advskew 10"

# Internal Interface (for the secondary firewall change the ip address to 192.168.10.11)
ifconfig_em1="inet 192.168.10.10 netmask 255.255.255.0"
# Internal Carp Interface
ifconfig_carp1="vhid 1 pass 22222 192.168.10.100/24 advskew 10"

# Heartbeat Interface (for the secondary firewall, change the ip address to 10.10.10.251)
ifconfig_vr0="10.10.10.250 netmask 255.255.255.0"

# PFSync Interface
ifconfig_pfsync0="up syncif vr0"

pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""

slave部分:
#slave的配置与master配置大概相近了. 改动的主要是本机IP和advskew的优先值. 注意: carp 的IP是公共的必须一致. 
gateway_enable="YES"
defaultrouter="192.168.1.1"
hostname="slave.cluster.org"
cloned_interfaces="carp0 carp1"

# External Public Interface (for the primary firewall use a different public ip.)
ifconfig_em0="inet 192.168.1.53 netmask 255.255.255.0"
# External Public Carp Interface
#ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24"
ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24 advskew 20"

# Internal Interface (for the primary firewall change the ip address to 192.168.10.10)
ifconfig_em1="inet 192.168.10.11 netmask 255.255.255.0"
# Internal Carp Interface
ifconfig_carp1="vhid 1 pass 22222 192.168.10.100/24 advskew 20"

# Heartbeat Interface (for the primary firewall, change the ip address to 10.10.10.250)
ifconfig_fxp0="10.10.10.251 netmask 255.255.255.0"

# PFSync Interface
ifconfig_pfsync0="up syncif fxp0"

pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""

C.pf.conf规则
master和slave除了网卡标识不一样.其他是一致的. 
################################################################################
# Macro and lists
################################################################################
lop_if = "lo0"
ext_if = "em0"
int_if = "em1"
sync_if= "vr0"
ext_carp = "carp0"

web_ports = "{ 80, 443 }"
#web_servers = "{ 192.168.10.20, 192.168.10.21, 192.168.10.22 }"
#web_servers = "{ 192.168.10.20 }"
web_servers = "{ 192.168.10.20, 192.168.10.21 }"


################################################################################
# Options, scrub and NAT
################################################################################
set block-policy drop
set skip on $lop_if

scrub in

nat on $ext_if from $int_if:network to any -> $ext_if

################################################################################
# Redirection
################################################################################
#rdr on $ext_if proto tcp from any to any port 80 -> $web_servers round-robin sticky-address
rdr on $ext_if proto tcp from any to $ext_carp port $web_ports -> $web_servers round-robin sticky-address


################################################################################
# Filtering Rules
################################################################################

pass quick on { $sync_if } proto pfsync keep state (no-sync)
pass on { $ext_if,$int_if } proto carp keep state

D. sysctl.conf的相关配置
#master和slave 一样. 
net.inet.carp.preempt=1

net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1

net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536

E. 配置好后重启服务器.检查系统状况
master上
master# ifconfig
vr0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:05:5d:85:84:d8
        inet 10.10.10.250 netmask 0xffffff00 broadcast 10.10.10.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:c0:9f:31:25:a2
        inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:c0:9f:31:25:a3
        inet 192.168.10.10 netmask 0xffffff00 broadcast 192.168.10.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33204
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet 127.0.0.1 netmask 0xff000000 
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1460
        pfsync: syncdev: vr0 syncpeer: 224.0.0.240 maxupd: 128
carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 192.168.1.51 netmask 0xffffff00 
        carp: MASTER vhid 1 advbase 1 advskew 10
carp1: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 192.168.10.100 netmask 0xffffff00 
        carp: MASTER vhid 1 advbase 1 advskew 10

slave上:  
slave# ifconfig
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:07:e9:1b:4b:cd
        inet 10.10.10.251 netmask 0xffffff00 broadcast 10.10.10.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:c0:9f:38:bd:af
        inet 192.168.1.53 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:c0:9f:38:bd:b0
        inet 192.168.10.11 netmask 0xffffff00 broadcast 192.168.10.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet 127.0.0.1 netmask 0xff000000 
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33204
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1460
        pfsync: syncdev: fxp0 syncpeer: 224.0.0.240 maxupd: 128
carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 192.168.1.51 netmask 0xffffff00 
        carp: BACKUP vhid 1 advbase 1 advskew 20
carp1: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 192.168.10.100 netmask 0xffffff00 
        carp: BACKUP vhid 1 advbase 1 advskew 20 

二. 服务器池的配置.
统一配置,把网关指向192.168.10.100就可以了. 
defaultrouter="192.168.10.100"

我在二台服务器上部署了相同的应用(lighttpd + php-fcgi), 配置我就不写了. 大家GG一下. 
由于服务器不够,mysql我部署在其中的s2上.让两台服务器连到他上. 

三 测试. 
A.静态页面: 
在s1和s2上分别与建立个index.htm页面. 
s1上index.htm的内容是: 
"hi, this is No.1 server"


s2上index.htm的内容是: 
"hi, this is No.2 server"

访问地址:  http://192.168.1.51/index.htm
返回的内容分别s1和s2的. 注意: 调度上做的状态保持的. 所以需要关闭浏览器再开才分别看得到不同的内容. 建议最好是几个人一起访问来返回不同的结果. 

B. 动态页面:

在s1和s2上简单部署了phpwind.  假如在./bbs下.
访问地址: http://192.168.1.51/bbs/
返回的内容保持了一致性, 
这里要提到的是文件的共享我没做, 现实的方法可以是通过nfs和iscsi. 暂时调不出iscsi的存储, 又不想用nfs所以就不作了. :)


master# pfctl -s state
all carp 192.168.1.52 -> 224.0.0.18       SINGLE:NO_TRAFFIC
all pfsync 224.0.0.240 <- 10.10.10.251       NO_TRAFFIC:SINGLE
all pfsync 10.10.10.250 -> 224.0.0.240       SINGLE:NO_TRAFFIC
all carp 224.0.0.18 <- 192.168.10.10       NO_TRAFFIC:SINGLE
all carp 224.0.0.18 <- 192.168.1.52       NO_TRAFFIC:SINGLE
all carp 192.168.10.10 -> 224.0.0.18       SINGLE:NO_TRAFFIC
all tcp 192.168.10.21:65346 -> 192.168.1.52:61575 -> 130.104.5.67:25  FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62414   TIME_WAIT:TIME_WAIT
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62416   FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62423   FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62425   FIN_WAIT_2:ESTABLISHED

C. HA的测试
拨掉master的网线, slave将自动接管. 状态保持.
ifconfig你可以看到
carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 192.168.1.51 netmask 0xffffff00 
        carp: MASTER vhid 1 advbase 1 advskew 20
大家可以做些些测试了. 

最后能有些优化:)


文献参考: 
http://www.countersiege.com/doc/pfsync-carp/
http://blog.randomutterings.com/articles/2007/06/15/redundant-failover-firewall-with-pf-pfsync-and-carp-on-freebsd