]

Solution

  • Modern Applications

Type

  • Document

Category

  • Reference Architecture

Product

  • Cloud Foundation 4
  • vSAN

Running Modern Applications with VMware Cloud Foundation with Tanzu on Dell EMC VxRail

Executive Summary

Business Case

With the rapid expansion of event stream processing, decisions must be made within a matter of seconds. Event streaming is a new paradigm where data is seen as a continuous stream of events. As the event stream data expands rapidly, Elasticsearch and Apache Kafka share a tight-knit relationship in the log/event processing realm. Several companies use Kafka as a transport layer for storing and processing large volumes of data. In many deployments in the industry, Kafka plays an important role in staging data before making its way into Elasticsearch for fast search and analysis. The Confluent Platform is an enterprise-ready platform that complements Kafka with advanced capabilities designed to help accelerate application development and connectivity, enable event transformations through stream processing, and simplify enterprise operations at scale.

Operating a Kafka and Elasticsearch environment within traditional IT infrastructure can be challenging, since the demand for resources can fluctuate with business needs, leaving the Kafka cluster either under-powered or over-provisioned. IT needs a more flexible, scalable, and secure infrastructure to handle the ever-changing demands of Kafka. With a single architecture that is easy to deploy, VMware Cloud Foundation™ can provision compute, network, and storage on demand. VMware Cloud Foundation protects network and data with micro-segmentation and satisfies compliance requirements with data-at-rest encryption. Policy-based management delivers business-critical performance. VMware Cloud Foundation delivers flexible, consistent, secure infrastructure and operations across private and public clouds and is ideally suited to meet the demands of Confluent Kafka and Elasticsearch.

Dell EMC VxRail™, powered by Dell EMC PowerEdge server platforms and VxRail HCI System Software, features next-generation technology to future proof your infrastructure and enables deep integration across the VMware ecosystem. Advanced VMware hybrid cloud integration and automation simplifies the deployment of a secure VxRail cloud infrastructure.

VMware Cloud Foundation on Dell EMC VxRail, the Dell Technologies Cloud Platform, builds upon native VxRail, and VMware Cloud Foundation capabilities with unique integration features jointly engineered between Dell EMC and VMware that simplify, streamline, and automate the operations of your entire SDDC from Day 0 through Day 2 operations. The full stack integration with VMware Cloud Foundation on VxRail allows both the HCI infrastructure layer and VMware cloud software stack lifecycle to be managed as one complete, automated, turnkey hybrid cloud experience, significantly reducing risk and increasing IT operational efficiency. This solution also allows us to perform cloud operations through a familiar set of tools, offering a consistent experience, with a single vendor support relationship and consistent SLAs across all your traditional and modernized workloads, public and private cloud, as well as edge deployments.

In this solution, we provide general design and deployment guidelines for Confluent Kafka and Elasticsearch on VMware Cloud Foundation on Dell EMC VxRail.

Technology Overview

Solution technology components are listed below:

  • VMware Cloud Foundation
    • VMware Cloud Foundation with Tanzu
    • VMware vSphere
    • VMware vSAN
    • VMware NSX Data Center
  • Dell EMC VxRail Hyperconverged Integrated System
    • VxRail HCI System Software
  • Confluent Kafka
  • Elasticsearch and Kibana

VMware Cloud Foundation

VMware Cloud Foundation is an integrated software stack that combines compute virtualization (VMware vSphere®), storage virtualization (VMware vSAN™), network virtualization (VMware NSX®), and cloud management and monitoring (VMware vRealize® Suite) into a single platform that can be deployed on-premises as a private cloud or run as a service within a public cloud. This documentation focuses on the private cloud use case. VMware Cloud Foundation bridges the traditional administrative silos in data centers, merging compute, storage, network provisioning, and cloud management to facilitate end-to-end support for application deployment. 

VMware Cloud Foundation with VMware Tanzu

VMware Cloud Foundation with VMware Tanzu™ is a hybrid cloud platform that automates infrastructure deployment and lifecycle management of complex Kubernetes clusters alongside mission critical enterprise applications, including an embedded Kubernetes runtime environment that accelerates the development of modern applications.

VMware Cloud Foundation with Tanzu automates full-stack deployment and operation of Kubernetes clusters through integration with VMware Tanzu Kubernetes Grid™. VMware Cloud Foundation with Tanzu helps to eliminate manual steps for host configuration, creating logical relationships, managing hypervisors for faster deployment of applications at scale. VMware Cloud Foundation with Tanzu provides a comprehensive hybrid cloud platform that bridges the gap between app developers and IT administrators. VMware Cloud Foundation can be deployed on-premises on a broad range of vSAN ReadyNode servers, on engineered systems like Dell EMC VxRail, or consumed as a service in the public cloud from VMware Cloud on AWS, Azure VMware Solution, Google Cloud VMware Engine, and select VMware Cloud™ Providers.

VMware Cloud Foundation with Tanzu is a major architectural upgrade to the industry’s most advanced hybrid cloud platform. The most exciting feature added to the VMware Cloud Foundation architecture is the integration of Kubernetes directly into the vSphere Hypervisor, which delivers an entirely new set of VMware Cloud Foundation services, a new Kubernetes and RESTful API surface that empowers developers to have self-service access to Kubernetes clusters, vSphere Pods, virtual machines, persistent volumes, stateful services, networking resources, etc. These services include VMware Tanzu Kubernetes Grid plus infrastructure and automation services that provide the basis for the cloud infrastructure and container ecosystems to boost developer productivity. VMware Cloud Foundation with Tanzu represents a major advancement in cloud-native compute, storage, networking, and management to seamlessly support containers and VMs all within the same automated hybrid cloud infrastructure.

VMware vSphere

VMware vSphere is VMware's virtualization platform, which transforms data centers into aggregated computing infrastructures that include CPU, storage, and networking resources. vSphere manages these infrastructures as a unified operating environment and provides operators with the tools to administer the data centers that participate in that environment. The two core components of vSphere are ESXi™ and vCenter Server®. ESXi is the hypervisor platform used to create and run virtualized workloads. vCenter Server is the management plane for the hosts and workloads running on the ESXi hosts.

VMware vSAN

VMware vSAN is the industry-leading software powering VMware’s software defined storage and HCI solution. vSAN helps customers evolve their data center without risk, control IT costs, and scale to tomorrow’s business needs. vSAN, native to the market-leading hypervisor, delivers flash-optimized, secure storage for all of your critical vSphere workloads, and is built on industry-standard x86 servers and components that help lower TCO in comparison to traditional storage. It delivers the agility to scale IT easily and offers the industry’s first native HCI encryption.

vSAN simplifies Day 1 and Day 2 operations, and customers can quickly deploy and extend cloud infrastructure and minimize maintenance disruptions. vSAN helps modernize hyperconverged infrastructure by providing administrators a unified storage control plane for both block and file protocols and provides significant enhancements that make it a great solution for traditional virtual machines as well cloud-native applications. vSAN helps reduce the complexity of monitoring and maintaining infrastructure and enables administrators to rapidly provision a file share in a single workflow for Kubernetes-orchestrated cloud native applications.

VMware NSX Data Center

VMware NSX Data Center is the network virtualization and security platform that enables the virtual cloud network, a software-defined approach to networking that extends across data centers, clouds, and application frameworks. With NSX Data Center, networking and security are brought closer to the application wherever it’s running, from virtual machines to containers to bare metal. Like the operational model of VMs, networks can be provisioned and managed independently of the underlying hardware. NSX Data Center reproduces the entire network model in software, enabling any network topology—from simple to complex multitier networks—to be created and provisioned in seconds. Users can create multiple virtual networks with diverse requirements, leveraging a combination of the services offered via NSX or from a broad ecosystem of third-party integrations ranging from next-generation firewalls to performance management solutions to build inherently more agile and secure environments. These services can then be extended to a variety of endpoints within and across clouds.

Dell EMC VxRail Hyperconverged Integrated System

The only fully integrated, pre-configured, and pre-tested VMware hyperconverged integrated system optimized for VMware vSAN and VMware Cloud Foundation, VxRail transforms HCI networking and simplifies VMware cloud adoption while meeting any HCI use case - including support for many of the most demanding workloads and applications. Powered by Dell EMC PowerEdge server platforms and VxRail HCI System Software, VxRail features next-generation technology to future proof your infrastructure and enables deep integration across the VMware ecosystem. The advanced VMware hybrid cloud integration and automation simplifies the deployment of a secure VxRail cloud infrastructure.

VxRail HCI System Software

VxRail HCI system software is integrated software that delivers a seamless and automated operational experience, offering 100% native integration between VxRail Manager and vCenter. Intelligent lifecycle management automates non-disruptive upgrades, patching, and node addition or retirement while keeping VxRail infrastructure in a continuously validated state to ensure that workloads are always available. The HCI System Software includes SaaS multi-cluster management and orchestration for centralized data collection and analytics that uses machine learning and AI to help customers keep their HCI stack operating at peak performance and ready for future workloads. IT teams can benefit from the actionable insights to optimize infrastructure performance, improve serviceability, and foster operational freedom.

Figure 1. VxRail Manager and SDDC Manager Integration

Confluent Kafka

Apache Kafka is a community distributed event streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since being created and open sourced by LinkedIn in 2011, Kafka has quickly evolved from a messaging queue to a full-fledged event streaming platform.

Founded by the original developers of Apache Kafka, Confluent delivers the most complete distribution of Kafka with the Confluent Platform. The Confluent Platform improves Kafka with additional community and commercial features designed to enhance the streaming experience of both operators and developers in production at massive scale.

Elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine‑tuned relevancy, and powerful analytics that scale with ease.

Kibana

Kibana is an open source frontend application that sits on top of the Elastic Stack, providing search and data visualization capabilities for data indexed in Elasticsearch. Commonly known as the charting tool for the Elastic Stack (previously referred to as the ELK Stack after Elasticsearch, Logstash, and Kibana), Kibana also acts as the user interface for monitoring, managing, and securing an Elastic Stack cluster — as well as the centralized hub for built-in solutions developed on the Elastic Stack. Developed in 2013 from within the Elasticsearch community, Kibana has grown to become the window into the Elastic Stack itself, offering a portal for users and companies.

Test Tools

We leverage the following monitoring and benchmark tools in the scope of our functional validation of Kafka and Elasticsearch on VMware Cloud Foundation.

Monitoring Tools

vSAN Performance Service

vSAN Performance Service is used to monitor the performance of the vSAN environment through the vSphere Client. The performance service collects and analyzes performance statistics and displays the data in a graphical format. You can use the performance charts to manage your workload and determine the root cause of problems.

vSAN Health Check

vSAN Health Check delivers a simplified troubleshooting and monitoring experience of all things related to vSAN. Through the vSphere client, it offers multiple health checks specifically for vSAN including cluster, hardware compatibility, data, limits, and physical disks. It is used to check the vSAN health before the mixed-workload environment deployment.

Workload Generation and Testing Tools

HCIBench

HCIBench is an automation wrapper around the popular and proven open-source benchmark tools, Vdbench, and Fio. Its simplified deployment makes it easier to automate testing across an HCI cluster.

The Confluent Platform Kafka CLI testing tools

The standard Confluent Platform Kafka application deployment provides several command-line utilities that allow for the simulation of message production and consumption. We use the kafka-topics, kafka-producer-perf-test, and kafka-consumer-perf-test command-line utilities to generate our test workloads.

Rally

Rally is an Elasticsearch benchmarking tool developed by the Elasticsearch team. It can be used to benchmark against an existing Elasticsearch cluster, manage benchmark configurations, run and compare results, and find potential performance issues.

Solution Configuration

This section introduces the resources and configurations:

  • Architecture diagram
  • Hardware resources
  • Software resources
  • Network configuration
  • vSAN configuration

Architecture Diagram

The VMware Cloud Foundation test environment was composed of a management domain and a workload domain. We deployed Confluent Kafka and the Elasticsearch stack as pods running in Tanzu Kubernetes Grid cluster of workload domain, and all other infrastructure VMs were in the separate management workload domain (figure 2).

Figure 2. TKG Cluster on VMware Cloud Foundation Solution Architecture

In our solution, we created a 4-node VxRail P570F cluster for the VMware Cloud Foundation management domain, running management virtual machines and appliances.

Table 1. Management Domain VMs

VM Role

vCPU

Memory (GB)

VM Count

Management Domain vCenter Server

4

16

1

SDDC Manager

4

16

1

Management Domain NSX-T Manager

6

24

3

Workload Domain NSX-T Manager

12

48

3

Workload Domain vCenter Server

8

28

1

VxRail Manager Appliance

2

8

1

Cloud Builder

4

4

1

For the workload domain, we created another 4-node VxRail P570F cluster with a separate NSX-T Fabric, deployed an NSX Edge Cluster and deployed a Tanzu Kubernetes Grid cluster after enabling the Kubernetes functionality in vSphere.

Table 2 shows the deployment of the workload domain edge nodes and TKG. For workload domain edge node, we recommend that NSX Edge transport nodes are deployed with “Large” form factor. In the YAML manifest file, we defined the size of Tanzu Kubernetes Grid workers as extra-large.

Table 2. Workload Domain VMs

VM Role

vCPU

Memory (GB)

Storage

 Deployment Size

VM Count

Workload Domain Edge node

8

32

    200 GB

Large

2

Supervisor Control Plane VM

16

32

22GB

Large

3

Tanzu Kubernetes Grid cluster – Control Plane

4

32

16GB

xlarge

3

Tanzu Kubernetes Grid cluster – Worker

4

32

16GB

xlarge

10

Linux Client

4

8

200GB

-

1

For Kafka and Elasticsearch pods, we can customize the parameters of yaml file for better performance. In this solution, we use 3 Kafka Brokers to validate. Each Kafka Broker pod was requested with 2 vCPU and 16GB.  And we also use 3 Elasticseach pods as a cluster, each Elasticsearch pod was requested with 2 vCPUs and 16G vRAM. Details of the yaml files that we used for our testing are demonstrated in Appendix 1 and 2.

Hardware Resources

In this solution, we used a total of four VxRail R570F nodes. Each server was configured with two disk groups, and each disk group consisted of one cache-tier write-intensive SAS SSD and four capacity-tier read-intensive SAS SSDs.

Each VxRail node in the cluster had the following configuration, as shown in table 3.

Table 3. Hardware Configuration for VxRail

PROPERTY

SPECIFICATION

Server model name

 VxRail P570F

CPU

2 x Intel(R) Xeon(R) Platinum 8180M CPU @ 2.50GHz, 28 core each

RAM

512GB

Network adapter

2 x Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller

Storage adapter

1 x Dell HBA330 Adapter

Disks

Cache - 2 x 800GB Write Intensive SAS SSDs

Capacity - 8 x 3.84TB Read Intensive SAS SSDs

Software Resources

Table 4 shows the software resources used in this solution.

Table 4. Software Resources

Software

Version

Purpose

VMware Cloud Foundation on Dell EMC VxRail

4.0

A unified SDDC platform on Dell EMC VxRail that brings together VMware vSphere, vSAN, NSX, and optionally, vRealize Suite components, into a natively integrated stack to deliver enterprise-ready cloud infrastructure for the private and public cloud.

See BOM of VMware Cloud Foundation on VxRail for details.

Dell EMC VxRail

7.0.000

Turnkey Hyperconverged Infrastructure for hybrid cloud

VMware vSphere

7.0

VMware vSphere is a suite of products: vCenter Server and ESXi.

VMware vSAN

7.0

vSAN is the storage component in VMware Cloud Foundation to provide low-cost and high-performance next-generation HCI solutions.

NSX-T

3.0

NSX-T is the key network component in VMware Cloud Foundation on VxRail and deployed automatically. It is designed for networking management and operation.

Kubernetes

1.16.8

Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.

Confluent Kafka

5.5.0

Apache Kafka is a community distributed event streaming platform capable of handling trillions of events a day.

Elasticsearch

7.8.0

Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases.

Kibana

7.9.0

Kibana is an open source frontend application that sits on top of the Elastic Stack, providing search and data visualization capabilities for data indexed in Elasticsearch.

Rally

2.0.0

Use Rally to benchmark against an existing Elasticsearch cluster, manage benchmark configurations, run and compare results, and find potential performance issues.

 

Network Configuration

Figure 3 shows the VMware vSphere Distributed Switch™ network configuration for TKG cluster in the workload domain of the VMware Cloud Foundation on VxRail. The networking used for TKG clusters is a combination of NSX-T that underlies the vSphere with Tanzu infrastructure and Calico that provides networking for pods, services, and ingress. To enable workload management for TKG cluster, an NSX-T edge cluster is required to deploy and BGP peering and route distribution of upstream network is required to configure. For more details, refer to VMware Cloud Foundation 4.0 on VxRail Planning and Preparation Guide.

Figure 3. Tanzu Kubernetes Cluster Networking

vSAN Configuration

The solution validation was conducted using the default vSAN datastore storage policy of RAID 1 FTT=1, checksums enabled, deduplication and compression disabled, and no encryption. This storage policy offers the best performance with the ability to tolerate up to one device failure (Figures 4 and 5).

A screenshot of a cell phone

Description automatically generated

Figure 4. vSAN Storage Policy Availability Settings

A screenshot of a cell phone

Description automatically generated

Figure 5. vSAN Storage Policy Advanced Policy Rules

Validation

To install the various components of the Confluent Platform/Kafka on TKG cluster, we used an edited “private.yaml” cluster specification that is fully listed in Appendix 1. With the help of Helm, you can install a prebuilt chart that will configure Elasticsearch resources by running one simple command. We can use an edited value.yaml file containing the specifications for Elasticsearch components listed in Appendix 2.

After the successful deployment, a list of healthy pods can be seen in the following output.

$ kubectl get pods

NAME                            READY   STATUS   RESTARTS    AGE

cc-operator-6895c8d746-mwp74                                       1/1                Running             1                               19h

connectors-0                                                                       1/1                Running             0                               19h

connectors-1                                                                       1/1                Running             0                               19h

controlcenter-0                                                                   1/1                Running             0                               19h

kafka-0                                                                               1/1                Running             0                               19h

kafka-1                                                                               1/1                Running             0                               19h

kafka-2                                                                               1/1                Running             0                               19h

ksql-0                                                                                 1/1                Running              0                              19h

ksql-1                                                                                 1/1                Running              0                              19h

replicator-0                                                                         1/1               Running              0                              19h

replicator-1                                                                         1/1               Running              0                              19h

schemaregistry-0                                                                1/1               Running              0                              19h

schemaregistry-1                                                                1/1               Running              0                              19h

zookeeper-0                                                                        1/1               Running              0                              19h

zookeeper-1                                                                        1/1               Running              0                              19h

zookeeper-2                                                                        1/1               Running              0                              19h

Elasticsearch-master-0                                                       1/1                Running             0                              19h

Production Criteria Recommendations

Kafka provides application-level fault tolerance through native application clustering. When deployed on VMware Cloud Foundation, it is best to consider the following recommended settings within Storage Policy Based Management (SPBM) and the vCenter vSAN Cluster level settings:

SPBM 

Availability:

FTT

The Number of Failures to Tolerate capability addresses the key customer and design requirement of availability. With FTT, availability is provided by maintaining replica copies of data, to mitigate the risk of a host failure resulting in lost connectivity to data or potential data loss. The FTT policy works in conjunction with VMware vSphere High Availability to maintain availability and provide consistent and near continuous uptime to workloads.

Recommendation: 1 Failure (FTT=1) 

RAID

vSAN has the ability to use RAID1 for mirroring or RAID5/6 for Erasure Coding. Erasure coding can provide the same level of data protection as mirroring (RAID 1), while using less storage capacity.

Recommendation: RAID-1 (Mirroring)

We recommend both FTT=1 and RAID1 from a performance and cost perspective. Using RAID1 will provide the best level of performance in conjunction with FTT=1, provides operational efficiency and availability coupled with Kafka clustering.

vCenter vSAN Cluster 

Deduplication/Compression

Deduplication and compression can greatly enhance space savings capabilities. However, for optimal performance with the Confluent Platform and Apache Kafka, we do not recommend enabling deduplication and compression. 

                        Recommendation: Disable Dedupe/Compression

Encryption

vSAN can perform data at rest encryption. Data is encrypted after all other processing, such as deduplication, is performed. Data at rest encryption protects data on storage devices, in case a device is removed from the cluster. Use encryption as per your company’s Information Security requirements. 

                         Recommendation: Enable Encryption as required per your InfoSec

High Availability

vSphere HA provides high availability for virtual machines. Hosts in the cluster are monitored, and in the event of a failure, the virtual machines on a failed host are restarted on remaining available alternate hosts.

Recommendation: HA Enabled

DRS

VMware vSphere® Distributed Resource Scheduler™ (DRS) is the resource scheduling and load balancing solution for vSphere. DRS works on a cluster of ESXi hosts and provides resource management capabilities like workload balancing and virtual machine (VM) placement. DRS also enforces user-defined resource allocation policies at the cluster level, while working with system-level constraints. 

Recommendation: DRS – partially automated

Note: With partially automated mode, DRS will handle the initial placement of virtual machines. However, any further migration recommendations will be surfaced up to the administrator to decide whether or not to move the virtual machine. The administrator can check the recommendation and may decide not to migrate the virtual machine. Recommendations should be for hosts on the same site.

Kafka Compression

We recommend using a Kafka supported type of compression (GZIP, LZ4, Snappy, Zstandard). Using compression will greatly save sent throughput to improve latency and minimize the impact to other workloads residing on the same vSAN cluster.  

Recommendation: Use compression

Kafka Partitions and Replication Factors

Environment and use case will dictate the optimal number of Kafka partitions and/or replication factors. We recommend consulting Confluent Best Practices to determine the number of partitions and/or replication factors that will meet your use case needs.

                         Recommendation: Refer to Confluent Best Practices

               Elasticsearch Level Replicas

Environment and use case will dictate the optimal number of Elasticsearch replicas. In a production environment, we recommend setting Elasticsearch level replicas to 1 in which two copies of data will be stored.

                        Recommendation: Number of Replicas=1                             

                         

Use Case

Elasticsearch + Kibana

When the Elasticsearch cluster is up and running on Kubernetes, you can use Kibana to manage and monitor it.

Rally supports the ability to store benchmark metrics in an Elasticsearch index for analysis in Kibana. After you ran a race, you noticed Rally metrics aggregating the data and shipping it into Elasticsearch, as shown in Figure 7.

$ curl http://172.31.11.110:9200/_cat/health\?v

Figure 7. Rally Metrics

If all is working as expected, you can see an elasticlogs_q-* index as listed. After the Kibana was defined, a list of all the parsed and mapped fields was displayed. Open the Discover page to analyze your data.

Figure 8. Kibana Visualization

Conclusion

VMware Cloud Foundation on VxRail delivers flexible, consistent, secure infrastructure and operations across private and public clouds. It is ideally suited to meet the demands of modern applications like Confluent Kafka and Elasticsearch Stack running on container platform such as Tanzu Kubernetes Grid. The real-time data streaming solution allows enterprises to make business critical decisions instantaneously using a single hybrid cloud platform that accelerates the deployment of application development lifecycle.

With VMware Cloud Foundation with Tanzu, admins get unified visibility of virtual machines (VMs), containers, and Kubernetes clusters all within vCenter Server. The Kubernetes concept of a namespace is integrated into vSphere and becomes the unit of management. By grouping resource objects such as VMs and containers into logical applications via namespaces, admins who used to manage thousands of VMs can now manage just dozens of applications, resulting in a massive increase in scale and reduction in cognitive load.

With Kubernetes embedded into the control plane of vSphere, VI admins can create both supervisor clusters and guest clusters. Developers consume cloud resources such as Kubernetes clusters, disks, and networks using familiar Kubernetes CLI and API tools, while the admins can manage systems at scale through vCenter Server. VMware Cloud Foundation with Tanzu automates infrastructure provisioning and scaling so that developers can focus on building and deploying apps while infrastructure teams become more strategic, maintaining centralized visibility and control of their global cloud infrastructure and operations.

Appendix

Appendix 1: The Edited Helm/Providers/private.yaml file from the Confluent Operator Fileset

Entries in the private.yaml file used for Operator installation that required edits to be done to customize the contents for this validation test are shown in red font below.

## Overriding values for Chart's values.yaml

## Example values to run Confluent Operator in Private Cloud

global

provider:

  name: private

  ## if any name which indicates regions

  ##

  region: anyregion

  kubernetes:

   deployment:

   ## If kubernetes is deployed in multi zone mode then specify availability-zones as appropriate

   ## If kubernetes is deployed in single availability zone then specify appropriate values

   ## For the private cloud, use kubernetes node labels as appropriate

   zones:

   - myzones

  ## more information can be found here

  ## https://kubernetes.io/docs/concepts/storage/storage-classes/

  storage:

  ## Use Retain if you want to persist data after CP cluster has been uninstalled

  reclaimPolicy: Delete

  provisioner: <strong>csi.vsphere.vmware.com</strong>

  parameters: {

   svstorageclass: default

  }

##

## Docker registry endpoint where Confluent Images are available.

##

registry:

  fqdn: docker.io

  credential:

   required: false

sasl:

plain:

  username: test

  password: test123

## Zookeeper cluster

##

zookeeper:

name: zookeeper

replicas: 3

resources:

  requests:

   cpu: 2000m

   memory: 8Gi

## Kafka Cluster

##

kafka:

name: kafka

replicas: 3

resources:

  requests:

   cpu: 2000m

   memory: 16Gi

loadBalancer:

enabled: true

domain: "cpbu.lab"

tls:

  enabled: false

  fullchain: |-

  privkey: |-

  cacerts: |-

metricReporter:

  enabled: false

 

## Connect Cluster

##

connect:

name: connectors

replicas: 2

tls:

  enabled: false

  ## "" for none, "tls" for mutual auth

  authentication:

   type: ""

  fullchain: |-

  privkey: |-

  cacerts: |-

loadBalancer:

  enabled: false

  domain: ""

dependencies:

  kafka:

   bootstrapEndpoint: kafka:9071

   brokerCount: 3

  schemaRegistry:

   enabled: true

   url: <a href="http://schemaregistry:8081">http://schemaregistry:8081</a>

## Replicator Connect Cluster

##

replicator:

name: replicator

replicas: 2

tls:

  enabled: false

  authentication:

   type: ""

  fullchain: |-

  privkey: |-

  cacerts: |-

loadBalancer:

  enabled: false

  domain: ""

dependencies:

  kafka:

   brokerCount: 3

   bootstrapEndpoint: kafka:9071

 

##

## Schema Registry

##

schemaregistry:

name: schemaregistry

tls:

  enabled: false

  authentication:

   type: ""

  fullchain: |-

  privkey: |-

  cacerts: |-

loadBalancer:

  enabled: false

  domain: ""

dependencies:

  kafka:

  brokerCount: 3

  bootstrapEndpoint: kafka:9071

 

##

## KSQL

##

ksql:

name: ksql

replicas: 2

tls:

  enabled: false

  authentication:

   type: ""

  fullchain: |-

  privkey: |-

  cacerts: |-

loadBalancer:

  enabled: false

  domain: ""

dependencies:

  kafka:

  brokerCount: 3

  bootstrapEndpoint: kafka:9071

  brokerEndpoints: kafka-0.kafka:9071,kafka-1.kafka:9071,kafka-2.kafka:9071

schemaRegistry:

  enabled: false

  tls:

   enabled: false

   authentication:

    type: ""

  url: http://schemaregistry:8081

 

## Control Center (C3) Resource configuration

##

controlcenter:

name: controlcenter

license: ""

##

## C3 dependencies

##

dependencies:

  c3KafkaCluster:

   brokerCount: 3

   bootstrapEndpoint: kafka:9071

   zookeeper:

    endpoint: zookeeper:2181

   connectCluster:

    enabled: true

    url: http://connectors:8083

   ksql:

    enabled: true

    url: http://ksql:9088

   schemaRegistry:

    enabled: true

    url: http://schemaregistry:8081

  ##

  ## C3 External Access

  ##

  loadBalancer:

   enabled: false

   domain: ""

  ##

  ## TLS configuration

  ##

  tls:

   enabled: false

   authentication:

    type: ""

   fullchain: |-

   privkey: |-

   cacerts: |-

  ##

  ## C3 authentication

  ##

  auth:

   basic:

    enabled: true

    ##

    ## map with key as user and value as password and role

    property:

     admin: Developer1,Administrators

     disallowed: no_access

Appendix 2: The Edited helm-charts/elasticsearch/values.yaml file

See link: https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml

About the Author

Yimeng Liu, Solutions Architect in the Solutions Architecture team of the Cloud Platform Business Unit, wrote the original version of this paper.

The following reviewers also contributed to the paper contents: 

  • William Leslie, Sr. Manager of VxRail Technical Marketing in Dell EMC 
  • Vic Dery, Sr. Principal Engineer of VxRail Technical Marketing in Dell EMC
  • Linwood Zoller, Sr. Principal Engineer, Kubernetes and Cloud-Native Apps on VxRail
  • Jason Marques, Sr. Principal Engineer of VxRail Technical Marketing in Dell EMC 
  • Ka Kit Wong, Staff Solutions Architect in the Solutions Architecture team of the Cloud Platform Business Unit in VMware 

 

 

 

Filter Tags

  • Modern Applications
  • Reference Architecture
  • Document
  • Cloud Foundation 4
  • vSAN