Openshift开启Calico-BGP-与-OVS性能PK
Openshift网络方案选择
- 大家都知道K8S在网络插件选择上有很多种，默认的是Flannel，但是它的性能一般，互联网中使用最多的是Calico BGP，因为它的性能非常好。
- 而对于Openshift，官方只支持ovs一种网络方案，同时RedHat也表示ovs在Openshift平台上运行是最合适的。但是ovs的网络性能怎样呢？因为ovs方案对数据需要进行加包，解包的过程，性能肯定是会受影响的。同时经过实测，在万兆网络中的损耗近50%，虽然在绝大部分场景下ovs已经够用了，但是但是跟几乎无损耗的Calico BGP比起来还是逊色不少。
- 很庆幸，Openshift虽然官方不作Calico网络方案的支持，但还是很体贴地把它加入到了Openshift的安装脚本中，从而让大家都能方便地使用Calico网络方案，包括IPIP及BGP方案。
安装步骤
- 在ansible hosts中设置关闭openshift默认的sdn方案，开启calico方案
/etc/ansible/hosts1
2
3
4[OSEv3:vars]
os_sdn_network_plugin_name=cni
openshift_use_calico=true
openshift_use_openshift_sdn=false - 设置Calico网络配置
openshift-ansible/roles/calico/defaults/main.yaml配置说明（正确开启calico bgp网络的关键）：1
2
3
4
5
6
7
8
9
10
11
12
13
14calico_ip_autodetection_method: "first-found"
ip_pools:
apiVersion: projectcalico.org/v3
kind: IPPoolList
items:
- apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: default-ipv4-ippool
spec:
cidr: "{{ openshift_cluster_network_cidr }}"
ipipMode: Always #默认是为Always，为IPIP模式
natOutgoing: true
nodeSelector: "all()"
calico_ip_autodetection_methodspec.ipipMode1
2
3calico_ip_autodetection_method: "interface=eth0"
# 默认为“first-found”，如果各主机网络设备名不一样，可以使用正则
# calico_ip_autodetection_method: "interface=(eth0|eth1)"完整配置1
ipipMode: Always #默认是为Always，为IPIP模式；Never为开启BGP模式
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32---
cni_conf_dir: "/etc/cni/net.d/"
cni_bin_dir: "/opt/cni/bin/"
calico_url_policy_controller: "quay.io/calico/kube-controllers:v3.5.0"
calico_node_image: "quay.io/calico/node:v3.5.0"
calico_cni_image: "quay.io/calico/cni:v3.5.0"
calicoctl_image: "quay.io/calico/ctl:v3.5.0"
calico_upgrade_image: "quay.io/calico/upgrade:v1.0.5"
calico_ip_autodetection_method: "interface=eth0"
# 默认为“first-found”，如果各主机网络设备名不一样，可以使用正则
# calico_ip_autodetection_method: "interface=(eth0|eth1)"
use_calico_etcd: False
# Configure the IP Pool(s) from which Pod IPs will be chosen.
ip_pools:
apiVersion: projectcalico.org/v3
kind: IPPoolList
items:
- apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: default-ipv4-ippool
spec:
cidr: "{{ openshift_cluster_network_cidr }}"
ipipMode: Never #默认是为Always，为IPIP模式；Never为开启BGP模式
natOutgoing: true
nodeSelector: "all()"
# Options below are only valid for legacy Calico v2 installations,
# and have been superceded by options above for Calico v3.
calico_ipv4pool_ipip: "always" - 正常执行Openshift安装脚本
1
2ansible-playbook playbooks/prerequisites.yml
ansible-playbook playbooks/deploy_cluster.yml - 查看网络
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46[root@master1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:fc:dd:fc:ed brd ff:ff:ff:ff:ff:ff
inet 192.168.0.3/24 brd 192.168.0.255 scope global dynamic eth0
valid_lft 86262sec preferred_lft 86262sec
inet6 fe80::248:584e:2626:2269/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:46:89:5d:d0 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
4: cali252a8913dc3@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
5: cali6d8bb449db0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
6: cali9efe4d704f6@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::ecee:eeff:feee:eeee/64 scope link
valid_lft forever preferred_lft forever
[root@master1 ~]# ip route
default via 192.168.0.1 dev eth0 proto static metric 100
10.128.113.64/26 via 192.168.0.7 dev eth0 proto bird
10.128.141.128/26 via 192.168.0.4 dev eth0 proto bird
10.129.8.0/26 via 192.168.0.9 dev eth0 proto bird
10.129.182.192/26 via 192.168.0.8 dev eth0 proto bird
10.129.200.0/26 via 192.168.0.6 dev eth0 proto bird
10.130.193.128/26 via 192.168.0.10 dev eth0 proto bird
blackhole 10.131.9.192/26 proto bird
10.131.9.206 dev cali252a8913dc3 scope link
10.131.9.207 dev cali6d8bb449db0 scope link
10.131.9.208 dev cali9efe4d704f6 scope link
10.131.42.192/26 via 192.168.0.11 dev eth0 proto bird
10.131.148.0/26 via 192.168.0.5 dev eth0 proto bird
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.3 metric 100
说明：如果要部署路由反射（RR）模式，可参考OpenShift支持Calico BGP 路由反射（RR）模式
网络性能测试
测试环境为公有云平台上的虚拟机
###iperf测试Pod吞吐量
测试方法与步骤
- 部署iperf服务端
1
2
3
4
5oc new-project test
oc run iperf-server --image=registry.dcs.cmbchina.cn:9443/tools/iperf3 -- -s
oc get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
iperf-server-1-r6z2x 1/1 Running 0 3m 10.131.2.76 node1 - 部署iperf客户端
1
2
3
4
5oc run iperf-client --image=registry.dcs.cmbchina.cn:9443/tools/iperf3 -n project-e --command -- sleep 10000
oc get pod -o wide | grep qperf
NAME READY STATUS RESTARTS AGE IP NODE
iperf-client-3-gtr2l 1/1 Running 0 2h 10.130.0.70 node2
qperf-server-1-xxmhz 1/1 Running 0 4h 10.128.2.59 node1 - iperf3客户端测试iperf3(pod)吞吐量
1
2oc rsh iperf-client-3-gtr2l
iperf3 -c 10.131.2.76
测试结果
ovs网络方案测试结果
1 | Connecting to host 10.130.0.51, port 5201 |
calico bgp网络方案测试结果
1 | Connecting to host 10.129.8.3, port 5201 |
| 网络方案 | 传输数据量 | 传输速率 |
|---|---|---|
| ovs方案 | 3.01 GB | 2.59 Gb |
| calico bgp方案 | 9.73 GB | 8.36 Gb |
qperf测试网络带宽与延时
测试方法与步骤
- 部署qperf服务端
1
2
3
4oc run qperf-server --image=registry.dcs.cmbchina.cn:9443/tools/qperf
oc get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
qperf-server-1-xxmhz 1/1 Running 0 4h 10.128.2.59 node1 - 部署qperf客户端
1
2
3
4
5oc run qperf-client --image=registry.dcs.cmbchina.cn:9443/tools/qperf --command -- sleep 10000
oc get pod -o wide -n project-e | grep qperf
NAME READY STATUS RESTARTS AGE IP NODE
qperf-client-2-7jmvb 1/1 Running 0 4h 10.130.2.224 node2
qperf-server-1-xxmhz 1/1 Running 0 4h 10.128.2.59 node1 - qperf客户端测试qperf(pod)带宽与延时
1
2oc rsh qperf-client-2-7jmvb
qperf 10.128.2.59 -t 10 -oo msg_size:8:256K:*2 tcp_bw tcp_lat
测试结果
ovs网络方案qperf测试结果
1 | tcp_bw: |
calico bgp网络方案qperf测试结果
1 | tcp_bw: |
| 包大小 | ovs方案带宽 | calico bgp方案带宽 | ovs方案时延 | calico bgp方案时延 |
|---|---|---|---|---|
| msg_size | ovs tcp_bw | calico bgp tcp_bw | ovs tcp_lat | calico bgp tcp_lat |
| 8bytes | 15 MB/sec | 17 MB/sec | 34.2 us | 95.8 us |
| 16bytes | 26.4 MB/sec | 32.1 MB/sec | 34.4 us | 71.5 us |
| 32bytes | 40.7 MB/sec | 39.4 MB/sec | 33.9 us | 69.1 us |
| 64bytes | 59.5MB/sec | 81.7 MB/sec | 33.4 us | 69.6 us |
| 128bytes | 76.1 MB/sec | 141 MB/sec | 34.1 us | 72.7 us |
| 256bytes | 194 MB/sec | 297 MB/sec | 34.1 us | 84 us |
| 512bytes | 239 MB/sec | 703 MB/sec | 34.2 us | 93.3 us |
| 1KiB | 256 MB/sec | 790 MB/sec | 34.8 us | 86.3 us |
| 2KiB | 258 MB/sec | 845 MB/sec | 46.3 us | 145 us |
| 4KiB | 262 MB/sec | 708 MB/sec | 56 us | 139 us |
| 8KiB | 259 MB/sec | 830 MB/sec | 86.5 us | 158 us |
| 16KiB | 250 MB/sec | 884 MB/sec | 133 us | 171 us |
| 32KiB | 272 MB/sec | 768 MB/sec | 219 us | 198 us |
| 64KiB | 291 MB/sec | 787 MB/sec | 435 us | 459 us |
| 128KiB | 272 MB/sec | 749 MB/sec | 733 us | 593 us |
| 256KiB | 282 MB/sec | 780 MB/sec | 1.27 ms | 881 us |
结果总结
从测试的数据中可以看到对于小包传输，Calico BGP的优势并不明显，同时它的网络延时甚至会更高,而对于大包传输，Calico BGP网络方案明显好于ovs方案。
文章已结束，以下并没有内容了。
#
本博客所有文章除特别声明外，均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自 Michael Blog！
相关推荐
评论