VMware RabbitMQ Stream Performance Validation

Application Validation

Introduction

This solution testing showcases VMware RabbitMQ running on Tanzu Kubernetes Grid for streaming performance.

Performance Testing Scope and Result

Unlike traditional queues, streams are persistent data structures that are designed to store large amounts of data with minimal in-memory overhead. Stream queues can be used with traditional AMQP 0.9.1 clients, but when used with a new, blazing fast binary protocol (the streams protocol), developers can achieve message publishing rates orders of magnitude faster than AMQP.

We used RabbitMQ PerfTest and RabbitMQ Streams Java Client to generate loads for the VMware RabbitMQ cluster to compare stream queues with different protocols. In the performance testing, we provisioned a cluster of three VMware RabbitMQ pods, each with 16 vCPUs, 64GB of RAM, and 500GB storage.

See VMware RabbitMQ Stream for more information about the performance tool,

Scenario 1: We compared the quorum queue with stream on traditional AMQP 0.9.1 clients and stream with protocol. Each queue with the default message size (12 bytes) by one producer and one consumer. The performance results are shown as below:

We ran the test with different stream protocols:

  • Each quorum queue published rate average at 38,305 message/s, consumed average at 37,981 message/s
  • VMware RabbitMQ stream with AMQP protocol published rate average at 48,872 message/s, consumed average at 48,804 message/s
  • VMware RabbitMQ streams with stream protocol published 1037598 message/s, consumed 1035672 msg/s, written bytes 30.1 MB/s, read bytes 30.0 MB/s

Scenario 2: We set the number of streams to get high throughput as the number grows. The figure of publishing rate shows near-linear scaling as the streams increase.

 

The detailed published, consumer, written, and read results are shown below:

  • Single VMware RabbitMQ streams published 1037598 message/s, consumed 1035672 msg/s, written bytes 30.1 MB/s, read bytes 30.0 MB/s
  • Three (3) streams published 2735823 msg/s, confirmed 2729862 msg/s, consumed 2729862 msg/s, written bytes 79.4 MB/s, read bytes 79.2 MB/s
  • Five (5) streams published 4347910 msg/s, confirmed 4341437 msg/s, consumed 4341437 msg/s, written bytes 126.2 MB/s, read bytes 126.0 MB/s

During the test, we set the number of messages placed in a sub-entry to increase throughput. A sub-entry is one "slot" in a publishing frame, meaning the outbound messages are not only batched in publishing frames, but in sub-entries as well.

Based on 1 stream with 1KB message size, the average rate achieved 218,249 msg/s (~18 billion messages per day), while on 3 streams with 1KB message size, the average rate climbed to 481,637 msg/s (~42 billion messages per day).

Check out the solution Home Page for more information.

Previous page: VMware RabbitMQ Best Practices

Next page:  VMware RabbitMQ Warm Standby Replication

 

 

 

 

Filter Tags

Modern Applications vSphere 7 vSphere with Tanzu Kubernetes Document Activity Path