Kafka 压力测试

测试目的

本地单机环境对 Kafka 处理 MQ 消息能力进行压力测试。测试包括对 Kafka 写入 MQ 消息和消费 MQ 消息进行压力测试,根据 10w、100w 和 1000w 级别的消息处理结果,评估 Kafka 的处理性能是否满足项目需求

测试范围及方法

测试使用 Kafka 自带的测试脚本,通过命令对 Kafka 发起写入 MQ 消息和 Kafka 消费 MQ 消息的请求。
模拟不同数量级的 MQ 消息写入和 MQ 消息消费场景,根据 Kafka 的处理结果,评估 Kafka 处理消息的能力

测试场景:

本机信息

操作系统:Windows 7 Enterprise Edition Service Pack 1 (Build 7601), 64-bit
处理器:Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz 四核
内存:16.00 GB
硬盘:西部数据 WDCWD10EZEX-75WN4A0(1000GB)
kafka版本: 2.11-2.1.1

写入压力测试
测试场景 MQ 消息数 每秒写入消息数 记录大小 (单位:字节) 测试命令
Kafka 消息写入测试 10W 2000 条 1000 kafka-producer-perf-test.bat –topic test_perf –num-records 100000 –record-size 1000 –throughput 2000 –producer-props bootstrap.servers=127.0.0.1:9092
10W 5000 条 1000 kafka-producer-perf-test.bat –topic test_perf –num-records 1000000 –record-size 2000 –throughput 5000 –producer-props bootstrap.servers=127.0.0.1:9092
1000W 5000 条 1000 kafka-producer-perf-test.bat –topic test_perf –num-records 10000000 –record-size 2000 –throughput 5000 –producer-props bootstrap.servers=127.0.0.1:9092
消费压力测试
测试场景 MQ 消息数 测试命令
Kafka 消息消费测试 10W kafka-consumer-perf-test.bat –broker-list localhost:9092 –topic test_perf –fetch-size 1048576 –messages 100000 –threads 1
100W kafka-consumer-perf-test.bat –broker-list localhost:9092 –topic test_perf –fetch-size 1048576 –messages 1000000 –threads 1
1000W kafka-consumer-perf-test.bat –broker-list localhost:9092 –topic test_perf –fetch-size 1048576 –messages 10000000 –threads 1

测试结果

写入测试
  • 10W写入
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    D:\dev\kafka_2.11-2.1.1\bin\windows
    λ kafka-producer-perf-test.bat --topic test_perf --num-records 100000 --record-size 1000 --throughput 2000 --producer-props bootstrap.servers=127.0.0.1:9092
    10002 records sent, 1999.6 records/sec (1.91 MB/sec), 1.2 ms avg latency, 147.0 max latency.
    10008 records sent, 2001.2 records/sec (1.91 MB/sec), 0.3 ms avg latency, 17.0 max latency.
    10004 records sent, 2000.8 records/sec (1.91 MB/sec), 0.2 ms avg latency, 5.0 max latency.
    10049 records sent, 2009.4 records/sec (1.92 MB/sec), 0.2 ms avg latency, 24.0 max latency.
    10003 records sent, 2000.6 records/sec (1.91 MB/sec), 0.2 ms avg latency, 4.0 max latency.
    10006 records sent, 2000.8 records/sec (1.91 MB/sec), 0.2 ms avg latency, 3.0 max latency.
    10014 records sent, 2002.4 records/sec (1.91 MB/sec), 0.2 ms avg latency, 6.0 max latency.
    10009 records sent, 2001.4 records/sec (1.91 MB/sec), 0.2 ms avg latency, 9.0 max latency.
    10005 records sent, 2000.6 records/sec (1.91 MB/sec), 0.2 ms avg latency, 3.0 max latency.
    100000 records sent, 1999.800020 records/sec (1.91 MB/sec), 0.29 ms avg latency, 147.00 ms max latency, 0 ms 50th, 1 ms 95th, 2 ms 99th, 25 ms 99.9th.

本例中写入 100w 条 MQ 消息为例,每秒平均向 kafka 写入了 1.91MB 的数据,大概是 1999.800条消息 / 秒,每次写入的平均延迟为 0.29 毫秒,最大的延迟为 147 毫秒。

  • 100w条写入
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    D:\dev\kafka_2.11-2.1.1\bin\windows
    λ ckafka-producer-perf-test.bat --topic test_perf --num-records 1000000 --record-size 1000 --throughput 5000 --producer-props bootstrap.servers=127.0.0.1:9092
    25002 records sent, 4997.4 records/sec (4.77 MB/sec), 1.7 ms avg latency, 155.0 max latency.
    25020 records sent, 5004.0 records/sec (4.77 MB/sec), 0.4 ms avg latency, 12.0 max latency.
    25015 records sent, 5002.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 5.0 max latency.
    25015 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 6.0 max latency.
    25025 records sent, 5002.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 6.0 max latency.
    25020 records sent, 5002.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 12.0 max latency.
    25005 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 8.0 max latency.
    25018 records sent, 5003.6 records/sec (4.77 MB/sec), 0.3 ms avg latency, 5.0 max latency.
    25007 records sent, 4999.4 records/sec (4.77 MB/sec), 0.3 ms avg latency, 13.0 max latency.
    25010 records sent, 5002.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 4.0 max latency.
    25015 records sent, 5001.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 6.0 max latency.
    25020 records sent, 5002.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 5.0 max latency.
    25015 records sent, 5003.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 3.0 max latency.
    25015 records sent, 5002.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 15.0 max latency.
    25015 records sent, 5003.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 3.0 max latency.
    25020 records sent, 5003.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 4.0 max latency.
    25005 records sent, 5001.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 5.0 max latency.
    25010 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 7.0 max latency.
    25010 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 5.0 max latency.
    25020 records sent, 5004.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 4.0 max latency.
    25005 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 5.0 max latency.
    25005 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 14.0 max latency.
    25010 records sent, 5001.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 6.0 max latency.
    25020 records sent, 5003.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 7.0 max latency.
    25025 records sent, 5003.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 4.0 max latency.
    25004 records sent, 5000.8 records/sec (4.77 MB/sec), 0.3 ms avg latency, 4.0 max latency.
    25016 records sent, 5003.2 records/sec (4.77 MB/sec), 0.2 ms avg latency, 5.0 max latency.
    25010 records sent, 5002.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 3.0 max latency.
    25005 records sent, 5001.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 3.0 max latency.
    25010 records sent, 5002.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 13.0 max latency.
    25025 records sent, 5003.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 4.0 max latency.
    25010 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 3.0 max latency.
    25024 records sent, 5004.8 records/sec (4.77 MB/sec), 0.3 ms avg latency, 4.0 max latency.
    25001 records sent, 5000.2 records/sec (4.77 MB/sec), 0.3 ms avg latency, 10.0 max latency.
    25005 records sent, 5000.0 records/sec (4.77 MB/sec), 0.6 ms avg latency, 46.0 max latency.
    25010 records sent, 5002.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 6.0 max latency.
    25020 records sent, 5002.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 5.0 max latency.
    25020 records sent, 5001.0 records/sec (4.77 MB/sec), 0.3 ms avg latency, 7.0 max latency.
    25015 records sent, 5003.0 records/sec (4.77 MB/sec), 0.2 ms avg latency, 3.0 max latency.
    1000000 records sent, 4999.875003 records/sec (4.77 MB/sec), 0.31 ms avg latency, 155.00 ms max latency, 0 ms 50th, 1 ms 95th, 2 ms 99th, 19 ms 99.9th.

本例中写入 100w 条 MQ 消息为例,每秒平均向 kafka 写入了 4.77MB 的数据,大概是 4999.875 条消息 / 秒,每次写入的平均延迟为 0.31 毫秒,最大的延迟为 155 毫秒。

  • 1000w条数据写入测试
    1
    10000000 records sent, 4999.982500 records/sec (4.77 MB/sec), 0.93 ms avg latency, 682.00 ms max latency, 0 ms 50th, 1 ms 95th, 3 ms 99th, 209 ms 99.9th.

本例中写入 1000w 条 MQ 消息为例,每秒平均向 kafka 写入了 4.77MB 的数据,大概是 4999.982 条消息 / 秒,每次写入的平均延迟为 0.93 毫秒,最大的延迟为 682 毫秒。

消费测试
  • 消费 10w 消息压测结果
1
2
3
4
kafka-consumer-perf-test.bat --broker-list localhost:9092 --topic test_perf --fetch-size 1048576 --messages 100000 --threads 1

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2019-03-20 17:32:23:660, 2019-03-20 17:32:36:593, 95.8204, 7.4090, 100475, 7768.8858, 3014, 9919, 9.6603, 10129.5493

以本例中消费 10w 条 MQ 消息为例总共消费了 95.8204M 的数据,每秒消费数据大小为 7.4090M,总共消费了 100475 条消息,每秒消费 7768.8858 条消息。

  • 消费 100w 消息压测结果
    1
    2
    3
    4
    D:\dev\kafka_2.11-2.1.1\bin\windows
    λ kkafka-consumer-perf-test.bat --broker-list localhost:9092 --topic test_perf --fetch-size 1048576 --messages 1000000 --threads 1
    start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
    2019-03-20 17:37:17:229, 2019-03-20 17:37:48:082, 953.8565, 30.9162, 1000191, 32417.9496, 3016, 27837, 34.2658, 35930.2727

以本例中消费 100w 条 MQ 消息为例总共消费了 953.8565M 的数据,每秒消费数据大小为 30.9162M,总共消费了 1000191 条消息,每秒消费 32417.9496 条消息。

  • 消费 1000w 消息压测结果
1
2
3
4
D:\dev\kafka_2.11-2.1.1\bin\windows
λ kafka-consumer-perf-test.bat --broker-list localhost:9092 --topic test_perf --fetch-size 1048576 --messages 10000000 --threads 1
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2019-03-20 17:41:59:064, 2019-03-20 17:46:47:322, 9536.9205, 33.0847, 10000186, 34691.7900, 3013, 285245, 33.4341, 35058.2341

以本例中消费 1000w 条 MQ 消息为例总共消费了 9536.9205 的数据,每秒消费数据大小为 33.0847M,总共消费了 10000186 条消息,每秒消费 35058.2341 条消息。

总结

一般写入 MQ 消息设置 5000 条 / 秒时,消息延迟时间小于等于 1ms,在可接受范围内,说明消息写入及时。

Kafka 消费 MQ 消息时,1000W 待处理消息的处理能力如果在每秒 20w 条以上,那么处理结果是理想的。

根据 Kafka 处理 10w、100w 和 1000w 级的消息时的处理能力,可以评估出 Kafka 集群服务,是否有能力处理上亿级别的消息。

本次测试是在单台服务器上进行,基本不需要考虑网络带宽的影响。

分享到: