测试20万qps的web接口二本篇文章主要描述如何测试两台Linux服务器之间的带宽和延迟。动机在前面的一篇文章中通过压测分析了web服务端能够提供的最大qps此时cpu达到了饱和状态而网络带宽没有。在这里我需要确认测试端和web服务端之间的网络带宽和延迟用于验证前面得出的结论。工具这里使用的工具有iperf3、sockperf。iperf3可以测试吞吐量sockperf可以测试吞吐量和延迟。确认网络连接状态在测试之前一定要提前确认好网络的连接状态要不然会得出一些奇怪的测试结果。刚开始使用iperf3测试带宽时发现丢包率特别高查找之后发现web-服务端的网络接口速率只有100Mb/s。使用命令调整速率失败通过更换网线和路由器端口后恢复到1000Mb/s。rootranger2-pc:~# ethtool enp3s0Settingsforenp3s0: Supported ports:[TP]Supportedlinkmodes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertisedlinkmodes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Speed: 100Mb/s Duplex: Full Auto-negotiation: on Port: Twisted Pair PHYAD:0Transceiver: internal MDI-X: Unknown Current message level: 0x000060e4(24804)linkifuprx_err tx_err hw wol Link detected:yes测试带宽iperf3和sockperf都支持带宽测试。在这里我测试了测试端和web服务端之间的带宽单向和双向发现它们之间的带宽可以跑满。sockperf测试带宽在web服务端运行server在测试端运行client。sockperf只能测试单向的带宽如果我想确认双向是否能同时跑满带宽就会用到iperf3。# web服务端运行sockperf-server。sockperf server--ip192.168.31.54--port8888# 测试端运行sockperf-client。sockperf throughput--ip192.168.31.54--port8888--msg-size1472--mpsmax--time100# 这里是client运行3次的结果。sockperf: Total of8066133messages sentin100.000sec sockperf: Summary: Message Rate is80661[msg/sec]sockperf: Summary: BandWidth is113.233MBps(905.861Mbps sockperf: Total of8066270messages sentin100.000sec sockperf: Summary: Message Rate is80663[msg/sec]sockperf: Summary: BandWidth is113.235MBps(905.883Mbps)sockperf: Total of8066225messages sentin100.000sec sockperf: Summary: Message Rate is80663[msg/sec]sockperf: Summary: BandWidth is113.235MBps(905.883Mbps)# 查看测试端的网络收发状况发现发送方向的带宽的确是跑满了。06:20:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 06:20:02 PM lo0.000.000.000.000.000.000.000.0006:20:02 PM eno11.0080674.000.06119590.910.000.000.0097.9706:20:02 PM wlp1s00.000.000.000.000.000.000.000.0006:20:02 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 06:20:03 PM lo0.000.000.000.000.000.000.000.0006:20:03 PM eno11.0080640.000.06119540.510.000.000.0097.9306:20:03 PM wlp1s00.000.000.000.000.000.000.000.00# 指定大包时带宽可以跑满指定小包时带宽跑不满。# 可以看到支持每秒发送43万个小包。rootranger2-mini:~# sockperf throughput --ip 192.168.31.54 --port 8888 --msg-size 64 --mpsmax --time 100sockperf:version#3.7-no.git sockperf[CLIENT]send on:[0]IP192.168.31.54 PORT8888# UDPsockperf: Warmup stage(sending a few dummy messages)... sockperf: Starting test... sockperf: Test end(interrupted by timer)sockperf: Test ended sockperf: Total of42678085messages sentin100.000sec sockperf: NOTE:testwas performed, using msg-size64. For getting maximum throughput consider using --msg-size1472sockperf: Summary: Message Rate is426780[msg/sec]sockperf: Summary: BandWidth is26.049MBps(208.389Mbps)# 测试端的网络收发状况。04:32:58 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 04:32:59 PM lo0.000.000.000.000.000.000.000.0004:32:59 PM eno12.00435405.000.1246772.710.000.000.0038.3204:32:59 PM wlp1s01.000.000.060.000.000.000.000.0004:32:59 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 04:33:00 PM lo0.000.000.000.000.000.000.000.0004:33:00 PM eno13.00435626.000.1946798.380.000.000.0038.3404:33:00 PM wlp1s00.000.000.000.000.000.000.000.00# web服务端的cpu状况。# 可以看到有个内核线程使用89.5%的cpu时间处理软中断。# 如果有多个内核线程来处理网络消息的接收是否可以达到更高的测试结果top-16:52:20 up41min,3users, load average:1.64,1.11,0.73任务:270total,3running,267sleeping,0stopped,0zombie %Cpu0:6.8us,10.6sy,15.4ni,60.4id,6.8wa,0.0hi,0.0si,0.0st %Cpu1:9.4us,19.9sy,0.0ni,62.2id,8.6wa,0.0hi,0.0si,0.0st %Cpu2:3.0us,5.1sy,0.0ni,91.9id,0.0wa,0.0hi,0.0si,0.0st %Cpu3:0.7us,7.0sy,0.0ni,64.9id,27.4wa,0.0hi,0.0si,0.0st %Cpu4:0.3us,1.3sy,0.0ni,94.7id,3.7wa,0.0hi,0.0si,0.0st %Cpu5:0.4us,0.4sy,0.0ni,9.7id,0.0wa,0.0hi,89.5si,0.0st %Cpu6:3.4us,5.7sy,0.3ni,84.1id,5.7wa,0.0hi,0.7si,0.0st %Cpu7:3.3us,4.3sy,0.0ni,88.0id,4.3wa,0.0hi,0.0si,0.0st MiB Mem:15417.5total,8331.2free,1495.1used,5924.5buff/cache MiB Swap:4096.0total,4096.0free,0.0used.13922.3avail Mem 进程号USERPR NI VIRT RES SHR %CPU %MEM TIME COMMAND49root200000S60.10.08:12.05 ksoftirqd/54046root2001342078604344R52.50.07:14.98 sockperf27979root391912940010458474632R17.90.70:00.54 apt-check# 查看web服务端的网卡队列情况发现不支持。rootranger2-pc:~# ethtool -l enp3s0netlink error: Operation not supported# 只开启了一个接收队列。rootranger2-pc:~# ls /sys/class/net/enp3s0/queues/rx-0 tx-0 tx-1 tx-2 tx-3 rootranger2-pc:~# cat /proc/interruptsCPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU70:320000000IR-IO-APIC2-edge timer8:00001000IR-IO-APIC8-edge rtc09:00000000IR-IO-APIC9-fasteoi acpi17:0394000000IR-IO-APIC17-fasteoi snd_hda_intel:card118:160000000IR-IO-APIC18-fasteoi i801_smbus23:0000010900IR-IO-APIC23-fasteoi ehci_hcd:usb2, ehci_hcd:usb424:00000000DMAR-MSI0-edge dmar025:00080000IR-PCI-MSI-0000:00:01.00-edge PCIe bwctrl26:00000000IR-PCI-MSI-0000:00:14.00-edge xhci_hcd34:000000282770IR-PCI-MSI-0000:00:1f.20-edge ahci[0000:00:1f.2]35:000000023IR-PCI-MSI-0000:00:16.00-edge mei_me36:0060500000IR-PCI-MSI-0000:00:1b.00-edge snd_hda_intel:card037:000254480000IR-PCI-MSI-0000:01:00.00-edge nvidia38:00003139000IR-PCI-MSIX-0000:03:00.00-edge enp3s039:0000072475700IR-PCI-MSIX-0000:03:00.01-edge enp3s0-TxRx-040:00000010460IR-PCI-MSIX-0000:03:00.02-edge enp3s0-tx-141:00000002655IR-PCI-MSIX-0000:03:00.03-edge enp3s0-tx-242:1340000000IR-PCI-MSIX-0000:03:00.04-edge enp3s0-tx-3iperf3测试带宽在web服务端运行server在测试端运行client。iperf3可以测试单向带宽也可同时测量双向带宽。# web服务端运行iperf3-server。iperf3-s-p8888# 测试端运行iperf3-client测试单向带宽。iperf3-u-c192.168.31.54-p8888-t30-P10-b100M# 测试端运行iperf3-client测试双向带宽。iperf3-u-c192.168.31.54-p8888-t30-P10-b100M--bidir# 单向带宽的3次测试结果。[SUM]0.00-30.00 sec3.32GBytes949Mbits/sec0.000ms0/2458699(0%)sender[SUM]0.00-30.00 sec3.32GBytes949Mbits/sec0.242ms30/2458310(0.0012%)receiver[SUM]0.00-30.00 sec3.32GBytes949Mbits/sec0.000ms0/2458867(0%)sender[SUM]0.00-30.00 sec3.32GBytes949Mbits/sec0.263ms23/2458358(0.00094%)receiver[SUM]0.00-30.00 sec3.32GBytes949Mbits/sec0.000ms0/2458680(0%)sender[SUM]0.00-30.01 sec3.32GBytes949Mbits/sec0.235ms1/2458312(4.1e-05%)receiver# 查看测试端的网络收发状况发现发送方向的带宽跑满了。06:45:49 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 06:45:50 PM lo0.000.000.000.000.000.000.000.0006:45:50 PM eno12.0081946.000.12119554.160.000.000.0097.9406:45:50 PM wlp1s00.000.000.000.000.000.000.000.0006:45:50 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 06:45:51 PM lo0.000.000.000.000.000.000.000.0006:45:51 PM eno12.0081945.000.12119552.510.000.000.0097.9406:45:51 PM wlp1s00.000.000.000.000.000.000.000.00# 双向带宽的3次测试结果。[SUM][TX-C]0.00-30.00 sec3.32GBytes949Mbits/sec0.000ms0/2459020(0%)sender[SUM][TX-C]0.00-30.00 sec3.31GBytes949Mbits/sec0.235ms393/2458490(0.016%)receiver[SUM][RX-C]0.00-30.00 sec3.33GBytes954Mbits/sec0.000ms0/2469581(0%)sender[SUM][RX-C]0.00-30.00 sec3.29GBytes943Mbits/sec0.175ms27797/2469141(1.1%)receiver[SUM][TX-C]0.00-30.00 sec3.32GBytes951Mbits/sec0.000ms0/2463459(0%)sender[SUM][TX-C]0.00-30.00 sec3.31GBytes949Mbits/sec0.292ms4829/2462942(0.2%)receiver[SUM][RX-C]0.00-30.00 sec3.33GBytes955Mbits/sec0.000ms0/2472409(0%)sender[SUM][RX-C]0.00-30.00 sec3.29GBytes943Mbits/sec0.191ms30577/2471960(1.2%)receiver[SUM][TX-C]0.00-30.00 sec3.32GBytes950Mbits/sec0.000ms0/2459145(0%)sender[SUM][TX-C]0.00-30.00 sec3.31GBytes949Mbits/sec0.164ms385/2458537(0.016%)receiver[SUM][RX-C]0.00-30.00 sec3.31GBytes949Mbits/sec0.000ms0/2456521(0%)sender[SUM][RX-C]0.00-30.00 sec3.29GBytes943Mbits/sec0.189ms14642/2455996(0.6%)receiver# 查看测试端的网络收发状况发现两个方向的带宽都跑满了。06:47:58 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 06:47:59 PM lo0.000.000.000.000.000.000.000.0006:47:59 PM eno181383.0081942.00118730.94119546.640.000.000.0097.9306:47:59 PM wlp1s00.000.000.000.000.000.000.000.0006:47:59 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 06:48:00 PM lo0.000.000.000.000.000.000.000.0006:48:00 PM eno181383.0081942.00118732.34119548.060.000.000.0097.9306:48:00 PM wlp1s00.000.000.000.000.000.000.000.00测试延迟在这里我测试了测试端和web服务端之间的延迟大约是102微秒。sockperf测试延迟sockperf有两种场景来测试延迟under-load和ping-pong这里使用了ping-pong模式。根据sockperf文档描述延迟的2倍就是RTT。# web服务端运行sockperf-server。sockperf server--ip192.168.31.54--port8888# 测试端运行sockperf-client。sockperf ping-pong--ip192.168.31.54--port8888--msg-size64--mpsmax--time100# 这里是client运行3次的结果。sockperf:avg-latency102.390(std-dev3.403)sockperf:# dropped messages 0; # duplicated messages 0; # out-of-order messages 0sockperf: Summary: Latency is102.390usec sockperf: Total486044observations;each percentile contains4860.44observations sockperf: ---MAXobservation467.910sockperf: ---percentile99.999261.082sockperf: ---percentile99.990175.892sockperf: ---percentile99.900124.611sockperf: ---percentile99.000112.973sockperf: ---percentile90.000104.562sockperf: ---percentile75.000103.164sockperf: ---percentile50.000102.255sockperf: ---percentile25.000101.474sockperf: ---MINobservation46.862sockperf:avg-latency102.384(std-dev4.813)sockperf:# dropped messages 0; # duplicated messages 0; # out-of-order messages 0sockperf: Summary: Latency is102.384usec sockperf: Total486054observations;each percentile contains4860.54observations sockperf: ---MAXobservation311.665sockperf: ---percentile99.999259.503sockperf: ---percentile99.990153.888sockperf: ---percentile99.900123.267sockperf: ---percentile99.000114.124sockperf: ---percentile90.000109.601sockperf: ---percentile75.000103.411sockperf: ---percentile50.000102.195sockperf: ---percentile25.000100.776sockperf: ---MINobservation49.797sockperf:avg-latency102.389(std-dev4.837)sockperf:# dropped messages 0; # duplicated messages 0; # out-of-order messages 0sockperf: Summary: Latency is102.389usec sockperf: Total486040observations;each percentile contains4860.40observations sockperf: ---MAXobservation807.367sockperf: ---percentile99.999259.533sockperf: ---percentile99.990157.034sockperf: ---percentile99.900125.547sockperf: ---percentile99.000114.138sockperf: ---percentile90.000109.364sockperf: ---percentile75.000103.374sockperf: ---percentile50.000102.204sockperf: ---percentile25.000100.881sockperf: ---MINobservation55.326redis-benchmark知道了测试端和web服务端之间的RTT为102*2微秒想看看redis操作的耗时情况。这里没有对redis做一些参数优化使用的是默认参数。# web服务端运行redis-server。redis-server redis.conf# 测试端运行redis-benchmark 。redis-benchmark-h192.168.31.54-p6379# 运行3次的测试结果。Summary: throughput summary:84104.29requests per second latency summary(msec): avg min p50 p95 p99 max0.5680.2000.5670.7350.8073.303Summary: throughput summary:72674.41requests per second latency summary(msec): avg min p50 p95 p99 max0.5890.2240.5750.8311.0473.791Summary: throughput summary:82781.46requests per second latency summary(msec): avg min p50 p95 p99 max0.5680.1840.5670.7270.7994.247# 查看测试端的网络使用情况网络使用率为38%。# 这里有点奇怪网络和cpu都没有占满是redis线程处理不过来了03:59:44 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 03:59:45 PM lo0.000.000.000.000.000.000.000.0003:59:45 PM eno1961.00432648.00104.5846476.530.000.006.0038.0703:59:45 PM wlp1s00.000.000.000.000.000.000.000.0003:59:45 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 03:59:46 PM lo0.000.000.000.000.000.000.000.0003:59:46 PM eno1956.00431228.00102.6546324.070.000.000.0037.9503:59:46 PM wlp1s00.000.000.000.000.000.000.000.00短连接测试在上一篇文章中测试端和web服务端之间使用了长连接来测试qps大约为22万。然而在真实场景中大部分请求都是短连接。在本次测试中没有对tcp参数做优化。# 使用短连接方式测试tps发现请求数很低。rootranger2-mini:~# wrk -t4 -c150 -d60s -T5s -H Connection: Close --latency http://192.168.31.54:8080/Running 1mtest http://192.168.31.54:8080/4threads and150connections Thread Stats Avg Stdev Max /- Stdev Latency21.13ms20.36ms414.10ms78.27% Req/Sec738.04522.741.99k77.61% Latency Distribution50%13.96ms75%29.68ms90%56.22ms99%76.72ms82057requestsin1.00m,14.63MBreadSocket errors: connect0,read0,write0,timeout1Requests/sec:1364.41Transfer/sec:249.17KB# 查看web服务端的网络连接状况发现有大量连接处于TIME_WAIT状态。rootranger2-pc:~/work/conf# ss -ant sport :8080 | awk {S[$1]} END {for (i in S) print i, S[i]}SYN-RECV7State1LISTEN1TIME-WAIT16379rootranger2-pc:~/work/conf# netstat -ant |grep 8080 |awk {S[$NF]} END {for (a in S) print a, S[a]}TIME_WAIT16379SYN_RECV7LISTEN1结论使用wrk达到22万qps时网络使用率还不到一半主要瓶颈还是在web服务端的cpu上毕竟是2012年的e3-1230v2 4核8线程。后续规划使用短连接方式发起请求看看如何达到20万qps。这3台电脑估计很难达到这个目标准备使用浪潮5212M5和PVE来做测试。