VMware Tanzu RabbitMQ Stream Performance Validation
This solution testing showcases Tanzu RabbitMQ running on Tanzu Kubernetes Grid for streaming performance.
Performance Testing Scope and Result
Unlike traditional queues, streams are persistent data structures that are designed to store large amounts of data with minimal in-memory overhead. Stream queues can be used with traditional AMQP 0.9.1 clients, but when used with a new, blazing fast binary protocol (the streams protocol), developers can achieve message publishing rates orders of magnitude faster than AMQP.
We used RabbitMQ PerfTest and RabbitMQ Streams Java Client to generate loads for the Tanzu RabbitMQ cluster to compare stream queues with different protocols. In the performance testing, we provisioned a cluster of three Tanzu RabbitMQ pods, each with 16 vCPUs, 64GB of RAM, and 500GB storage.
See Tanzu RabbitMQ Stream for more information about the performance tool,
Scenario 1: We compared the quorum queue with stream on traditional AMQP 0.9.1 clients and stream with protocol. Each queue with the default message size (12 bytes) by one producer and one consumer. The performance results are shown as below:
We ran the test with different stream protocols:
- Each quorum queue published rate average at 38,305 message/s, consumed average at 37,981 message/s
- Tanzu RabbitMQ stream with AMQP protocol published rate average at 48,872 message/s, consumed average at 48,804 message/s
- Tanzu RabbitMQ streams with stream protocol published 1037598 message/s, consumed 1035672 msg/s, written bytes 30.1 MB/s, read bytes 30.0 MB/s
Scenario 2: We set the number of streams to get high throughput as the number grows. The figure of publishing rate shows near-linear scaling as the streams increase.
The detailed published, consumer, written, and read results are shown below:
- Single Tanzu RabbitMQ streams published 1037598 message/s, consumed 1035672 msg/s, written bytes 30.1 MB/s, read bytes 30.0 MB/s
- Three (3) streams published 2735823 msg/s, confirmed 2729862 msg/s, consumed 2729862 msg/s, written bytes 79.4 MB/s, read bytes 79.2 MB/s
- Five (5) streams published 4347910 msg/s, confirmed 4341437 msg/s, consumed 4341437 msg/s, written bytes 126.2 MB/s, read bytes 126.0 MB/s
During the test, we set the number of messages placed in a sub-entry to increase throughput. A sub-entry is one "slot" in a publishing frame, meaning the outbound messages are not only batched in publishing frames, but in sub-entries as well.
Based on 1 stream with 1KB message size, the average rate achieved 218,249 msg/s (~18 billion messages per day), while on 3 streams with 1KB message size, the average rate climbed to 481,637 msg/s (~42 billion messages per day).
Check out the solution Home Page for more information.
Previous page: VMware Tanzu RabbitMQ Best Practices
Next page: VMware Tanzu RabbitMQ Warm Standby Replication