RabbitMQ Stream Performance Validation

Application Validation

Introduction

This solution testing showcases Tanzu RabbitMQ running on Tanzu Kubernetes Grid for streaming performance.

Performance Testing Scope and Result

Unlike traditional queues, streams are persistent data structures that are designed to store large amounts of data with minimal in-memory overhead. Stream queues can be used with traditional AMQP 0.9.1 clients, but when used with a new, blazing fast binary protocol (the streams protocol), developers can achieve message publishing rates orders of magnitude faster than AMQP.

We used RabbitMQ PerfTest and RabbitMQ Streams Java Client to generate loads for the RabbitMQ cluster to compare stream queues with different protocols. In the performance testing, we provisioned a cluster of three Tanzu RabbitMQ pods, each with 16 vCPUs, 64GB of RAM, and 500GB storage.

See RabbitMQ Stream for more information about the performance tool,

Scenario 1: We compared the quorum queue with stream on traditional AMQP 0.9.1 clients and stream with protocol. Each queue with the default message size (12 bytes) by one producer and one consumer. The performance results are shown as below:

We ran the test with different stream protocols:

  • Each quorum queue published rate average at 38,305 message/s, consumed average at 37,981 message/s
  • RabbitMQ stream with AMQP protocol published rate average at 48,872 message/s, consumed average at 48,804 message/s
  • RabbitMQ streams with stream protocol published 1037598 message/s, consumed 1035672 msg/s, written bytes 30.1 MB/s, read bytes 30.0 MB/s

Scenario 2: We set the number of streams to get high throughput as the number grows. The figure of publishing rate shows near-linear scaling as the streams increase.

 

The detailed published, consumer, written, and read results are shown below:

  • Single RabbitMQ streams published 1037598 message/s, consumed 1035672 msg/s, written bytes 30.1 MB/s, read bytes 30.0 MB/s
  • Three (3) streams published 2735823 msg/s, confirmed 2729862 msg/s, consumed 2729862 msg/s, written bytes 79.4 MB/s, read bytes 79.2 MB/s
  • Five (5) streams published 4347910 msg/s, confirmed 4341437 msg/s, consumed 4341437 msg/s, written bytes 126.2 MB/s, read bytes 126.0 MB/s

During the test, we set the number of messages placed in a sub-entry to increase throughput. A sub-entry is one "slot" in a publishing frame, meaning the outbound messages are not only batched in publishing frames, but in sub-entries as well.

Based on 1 stream with 1KB message size, the average rate achieved 218,249 msg/s (~18 billion messages per day), while on 3 streams with 1KB message size, the average rate climbed to 481,637 msg/s (~42 billion messages per day).

 

 


 

Filter Tags

Modern Applications vSphere 7 vSphere with Tanzu Kubernetes Document Activity Path