Pivotal Cloud Foundry on Dell EMC VxRail Appliance

Executive Summary

This section covers the Business Case, Solution Overview, Document Purpose and the Audience for the Pivotal Cloud Foundry on Dell EMC VxRail Appliance document.

Business Case

The digital transformation is powering a deep shift in platforms and processes aimed at driving continuous delivery of software. Developers increasingly work in agile teams and demand a platform that allows them to continuously deliver updates and to horizontally scale their applications with no downtime. They seek standardized ways to plug in leading data services and perform deep user analytics on top of massive datasets to drive rapid iteration based on customer needs. These changes and disruptive technologies are forcing companies to evolve how they develop and bring products to Market. Achieving state-of-theart application development and delivery lies at the heart of this transformation and accelerates your time-to-market!

When your IT infrastructure impedes your ability to mobilize and adapt to new ways of doing business, it often means you cannot meet Service Level Agreements (SLAs), or new business initiatives have to wait. Today's consumers of IT resources often run advanced application suites and require access to those applications anytime, from anywhere, on any device. The lack of a simple, repeatable, infrastructure lifecycle can inhibit your organization’s ability to innovate and respond to market demands. You know you need to be less reactive and find an IT infrastructure solution that gets your organization to the next level.

Now you can extend the benefits of hyperconverged infrastructure to applications and workloads that start small, even if you do not know how fast and large they will grow. Hyperconverged infrastructures appliances (HCIA) tightly integrate compute, storage, networking and virtualization resources on a hardware platform support by a single vendor. Cloud Native Application Development is about increasing agility and enablement of organizations to rapidly respond to business demands.

VMware is teaming up with Pivotal to enable customers to quickly deploy a modern application development platform into their environment while leveraging their existing and proven investment in VMware infrastructure. PCF as an on-premises structured platform as a service (PaaS) in combination with Dell EMC VxRail Appliances enables the IT organizations to be ready for building and delivering Cloud Native Applications (CNA). Developers gain the ability to push applications in single command and deploy code complete in seconds with instant access to application services. No effort is wasted to build or configure the environment which dramatically increases the productivity of developers and operators.

Solution Overview

Pivotal Cloud Foundry (PCF) is a Cloud Native Solution that supplies developers with a ready-to-use cloud computing environment and application services, all hosted by virtualized servers on your existing private infrastructure. Built for continuous delivery and horizontal scale, Pivotal Cloud Foundry allows developers to focus on the application as the unit of deployment and scale. It delivers a turnkey experience for scaling and updating applications with no downtime.

Dell EMC VxRail™ Appliances are a hyperconverged infrastructure (HCI) solution that consolidates compute and storage into a single, highly available, network-ready turnkey unit. VxRail is not a server. It is an appliance. VxRail has a simple, scale-out architecture, leveraging VMware vSphere^® for server virtualization and VMware vSAN™ for virtualized software-defined storage.  A VxRail cluster can scale up to 64 nodes.

Document purpose

This document describes the reference architecture for deploying PCF using Dell EMC VxRail Appliances powered by VMware vSAN 6.2 and VMware vSphere 6.0. It also describes how VxRail can simplify and optimize your onpremises native cloud platform powered by PCF.

This document is not a comprehensive guide to every aspect of this solution, but rather a high level overview of the configuration.

Audience

This reference architecture guide is for development and operations (DevOps) cloud architects, DevOps cloud operators, or software development managers who want to design, manage, and deploy cloud native applications with PCF on VxRail Appliances.

Knowledge of the Dell EMC VxRail Appliance

This section describes the pre-requisite knowledge the reader should have for fully understanding this document.

Knowledge of the Dell EMC VxRail Appliance

The reader should have:

• Knowledge of VMware vSphere
• Knowledge of VMware vSAN
• Knowledge of Pivotal Cloud Foundry (PCF)
• Familiarity with virtualization, networking, storage technology, and general IT functions.

Technology Overview

This section provides an overview of the technologies used in this solution.

Dell EMC VxRail Appliance

The Dell EMC VxRail Appliance is the only fully integrated, pre-configured, and tested HyperConverged appliance jointly developed with VMware. Powered by VMware vSAN and vSphere, the appliance provides a fast, easy way to extend your VMware environment. Scale-out is easy—you simply add a new appliance to join an existing VxRail Appliance cluster. Select from a broad set of configuration options at a variety of price and scale points, including all-flash VxRail Appliances to achieve exceptional scalability. The VxRail Appliance automatically discovers and non-disruptively adds each new node and appliance and rebalances resources and workloads across the cluster, creating a single resource pool for compute and storage.

VxRail Appliance is the only hyperconverged appliance that provides kernel layer integration between VMware vSAN and the vSphere hypervisor, delivering unique and unmatched performance and efficiency benefits. VxRail seamlessly integrates with off-the-shelf VMware tools, providing a familiar experience. VxRail is the new standard in hyperconverged appliances.

The VxRail Appliance combines virtualization, compute, storage, management, and data protection with a single point of support for the hardware and software in an efficient 2U/4-node building block. The VxRail Appliance delivers resiliency, quality of service (QoS), and centralized management functionality—enabling faster, better, and simpler management of consolidated workloads, virtual desktops, business-critical applications, and remote office infrastructure.

VxRail is available in multiple models and configurations, including all-flash options that feature enterprise-class data efficiency services, including deduplication, compression, and erasure coding, which enhance performance and offer greater effective capacity.

Figure 1. VxRail G Series All-Flash and Hybrid Configurations

The VxRail Appliance enables power-on to virtual machine creation in minutes, radically easy deployment, one-click non-disruptive patches and upgrades, and extremely simplified management.

VxRail Appliances also seamlessly extend to more than 20 public cloud environments, including vCloud Air, Amazon Web Services, and Microsoft Azure, to securely expand storage capacity without limits and providing an additional 10 TB of on-demand cloud storage per appliance.

Customers can purchase additional cloud-storage capacity as needed.

VxRail is backed by a single point of contact which includes 24/7 support for both hardware and software on the appliance.

The software bundled with VxRail pertinent to this solution is:

• VMware vSphere —A common virtualization layer to host the infrastructure server and virtual desktop environment. It provides high availability in the virtualization layer with vSphere features such as VMware High Availability (VMHA) clusters and VMware vMotion and Storage vMotion.
• VMware vCenter Server —A centralized platform for managing all vSphere hosts and their virtual machines.
• VxRail Manager —A user-friendly GUI to monitor and maintain the health of all individual appliance and nodes in the entire VxRail cluster.
• VMware vSAN – Software fully integrated with vSphere that provides full-feature and cost-effective software-defined storage.
• VMware vRealize Log Insight—Software that delivers real-time monitoring to the VxRail Manager.

Pivotal Cloud Foundry

Pivotal Cloud Foundry (PCF) is a commercial version of Cloud Foundry Platform as a Service (PaaS) that supplies developers with a ready-to-use cloud computing environment and application services, all hosted by virtualized servers on your existing on-premises private or off-premises public infrastructure. It is a structured cloud-native platform that is made up of standardized components. It enables developers with a rich set of services for deploying, scaling and managing the application life cycle. It enables operator visibility and control to manage the platform. This base functionality is enabled through:

• Ops Manager
• Application deployment and management
• Polyglot (Frameworks, Languages, and Services)
• Container management
• User Management with role based access
• Performance Monitoring and Reporting
• Centralized Log aggregation
• Security
• Fault Tolerance and Multiple Resilience levels

The essential elements of PCF pertinent to this solution is:

• PCF Ops Manager for Pivotal Cloud Foundry installs and manages PCF products and services. It is a web application used to deploy and manage Pivotal CF and associated services such as SSO and Pivotal CF Metrics. It is the industry’s first turn-key enterprise CNP management platform with Infrastructure-as-a-Service (IaaS) integration. 

• Pivotal Elastic Runtime is the framework that hosts running applications, manages system health, and provides client access to the runtime environment, including running applications through the cloud controller API endpoint.
• PCF Metrics stores metrics data from applications running on PCF. It provides real-time data on container metrics; network metrics; and application events such as create, update etc.
• Pivotal Marketplace provides developers with a robust self-service user experience that boosts productivity with on-demand access to a large marketplace of mobile and data services, an intuitive console, and open APIs. The Marketplace's managed and user-provided services can be used without the need for IT action.

Since PCF is based on Cloud Foundry, an open source platform, developers have flexibility to leverage open source tools such as:

• Eclipse
• Jenkins, Bamboo, Concourse
• Artifactory
• LogInsight, VROPS, Loggly, Splunk, DataDog, Papertrail­
• NewRelic, AppDynamics
• Spring ecosystem

Deploying PCF on Dell EMC VxRail Appliances

This section provides information on deploying PCF on Dell EMC VxRail Appliances.

Deploying PCF on Dell EMC VxRail Appliances

The purpose of this reference architecture is to demonstrate the functionality, performance, and scalability of cloud native applications on a Dell EMC VxRail Appliance with Pivotal Cloud Foundry (PCF) and VMware vSphere 6.0.

VxRail and PCF are designed for maximum availability. Both offer zero downtime upgrades to both platform and applications. Pivotal CF enables developers to code in mul

Reference Architecture Requirements

Follow all VxRail appliance requirements found in the VxRail Network Guide.  It should be noted that vSphere is deployed on VxRail with High Availability (HA) and Distributed Resource Scheduler (DRS) enabled. It is a requirement for DRS be fully automated for the solution to work.

The following are the minimum resources required for maintaining a Pivotal Cloud Foundry® (PCF) deployment with Ops Manager and Elastic Runtime on the VxRail Appliance:

• Public IP address for PCF Ops Manager
• Public IP address for PCF Elastic Runtime
• A wildcard DNS record to point to your PCF router
• A DNS entry for the IP address for Ops Manager, with a fully qualified domain name (FQDN)
• The Ops Manager Director must have https access to vCenter Server and each ESXi hosts on TCP ports 443, 902 and 903
• A network without DHCP for deploying the Elastic Runtime VMs
• A Security Assertion Markup Language (SAML) identify provider

Hardware resources

A VxRail Appliance consists of four independent nodes. There are four different model types, and several configurations available within each of these in terms of memory and disk sizes.

For this reference architecture, a VxRail model G410 Appliance was used, with each node having the following core hardware components:

• 2 Intel Xeon E5-2630 v3 eight-core CPUs
• 256 GB of memory
• Dual ports 10GbE NICs
• Five SAS 10K RPM 1.2 TB HDD for the VMware vSAN Datastore
• One 400 GB MLC enterprise-grade SSD for the vSAN read/write cache

Refer to the VxRail specifications data for more details on the hardware configuration of each model of VxRail Appliances.

With 64 physical CPU cores in total, this solution supports all the infrastructure servers running on the VxRail G410 appliance and the additional load from other virtual machines used in PCF solution.

Memory plays a key role in ensuring application functionality and performance. Each group of desktops will have different targets for the amount of available memory that is considered acceptable. With total of 1024 GB RAM (or optionally 2048 GB RAM) available on VxRail model G410 appliance, it can easily support the infrastructure virtual servers and virtual desktops running in this solution.

Details of the VxRail Virtual Machine resources is found in Appendix D.

Software resources

The VxRail software bundle is preloaded and consists of the following components:

A PCF solution will have a foundation of key components (tiles) and additional tiles can be added depending on the design requirements of the Native Cloud application being deployed. The following tiles represent the more common choices for application development with PCF. Configuration details for each tile is show in Appendix A.

All software binaries are available via pivotal cloud foundry website portal, network.pivotal.io.

Figure 3: Pivotal Cloud Foundry Portal

Solution Management

This section focusses on how PCF on Dell EMC VxRail Appliance provides ease of management using the infrastructure provided.

Dell EMC VxRail Management

PCF on Dell EMC VxRail Appliance provides ease of management using the infrastructure provided.

The VxRail Appliance comes with VxRail Manager for complete life cycle management of hardware and software components.

Figure 4: VxRail Manager

PCF Management

PCF has a flexible management framework. Both GUI and CLI options are available.

The Operations Manager (Ops Manager) is a GUI web application used to deploy and manage a Pivotal Cloud Foundry (PCF). It is available on all major browsers. PCF foundational components and additional tiles can be deployed and configured here.

Various binaries are available for cli administration via Linux, Windows and Mac OS X at http://docs.cloudfoundry.org/cf-cli/

Figure 5: PCF Ops Manager

Storage Configuration

This section provides details on the storage configurations used for this solution.

Storage Configuration

The storage layer is a key component of any SDDC solution that serves data generated by applications and operating systems in a data center storage processing system. VxRail uses VMware vSAN storage to provide virtualization at the storage layer. vSAN is integrated in your VxRail Appliance to provide Software-Defined Storage (SDS). vSAN is not a VMware vSphere Storage Appliance, but is embedded in the ESXi hypervisor kernel's I/O data path. As a result, vSAN can deliver higher performance with minimum CPU and memory overhead.

The VxRail Appliance pools internal SSDs and HDDs on the ESXi hosts to present a single Datastore, vSAN for all hosts in the cluster. vSAN uses a highly available, distributed, object-based architecture. vSAN mirrors and distributes the individual virtual disk (VMDK) files across the Datastore.

Figure 6: vSAN Cluster Datastore

vSAN lets virtualization infrastructure administrators manage storage on a pervirtual machine basis via Storage Based Policy Management (SBPM). This virtual machine-centric approach allows for storage policies to be defined at virtual machine-level granularity for provisioning and load balancing vSAN is fully integrated with vSphere, which simplifies setting up the availability, capacity, and performance policies.

PCF VxRail Storage

The powerful simplicity of vSAN in the VxRail Appliance enables PCF, which enables quick configuration and reducing time to value. You are productive more rapidly. There is no complex configuration of storage required. No FC zoning, no LUNS to configure etc, VxRail Appliance provisioning provides vSAN storage ready to utilize. PCF configuration requires declaration of an Ephemeral and Persistent Datastore. During VxRail configuration the single vSAN Datastore will serve to provide both. Your PCF installation and functionality is dependent on this and modification (renaming or removal) will disrupt PCF functionality. Datastore names are provisioned at the onset of VxRail deployment. The example given in the table is specific to the unit for the RA. Your VxRail vSAN Datastore name will have a similar format, but not be exactly the same.

Another recommendation is to separate virtual machine inventory according to function.

Creation of PCF specific VM folders, Template folders and Disk path folders will enable you to organize and if necessary correlate application function with vSAN objects for better management.

Figure 7: PCF vCenter Disk Configuration Options

Network Configuration

This section provides details on the network configuration used for this solution.

The VxRail Appliance nodes come with sufficient network adapters to provide ample bandwidth for storage, management, virtual machine, and client access traffic. All that is required to set it up from networking stand point is sufficient switching capacity.

The solution tested in this paper used two interconnected Brocade switches to provide a fully redundant network architecture, eliminating single points of failure. Any similar enterprise-grade switches with sufficient capacity can be used with your PCF implementation on VxRail, including from Cisco, Arista etc.

This solution uses virtual local area networks (VLANs) to separate network traffic of various types to improve throughput, manageability, application separation, high availability, and security.

This solution requires a minimum of four VLANs:

• Client access network—Virtual machine networking  (public customer-facing networks, which can be separated if needed)
• Storage network—vSAN networking (private network)
• vMotion network (private network)
• Management network—vSphere management (private network)

Below is the VxRail Distributed Port Configuration.

Figure 8 shows the network connectivity between the switches and the VxRail Appliance. We recommend using at least two network switches to avoid a single point of failure in the network layer.

Figure 8: Network Layout Diagram

Figure 9 shows all network interfaces on the vSphere servers in this solution use 10-gigabit Ethernet connections. VxRail creates a distributed switch for the cluster by using the two 10 GbE NICs in each node.

Figure 9: Distributed Virtual Switch, VxRail

PCF on VxRail Sizing Guidance

This section provides information on the PCF Sizing deployment.

PCF on VxRail Sizing Guidance

The Pivotal Sizing tool can be found at http://pcfsizer.cfapps.io/. The tool takes inputs for sizing a PCF deployment and provides a high level guidance on infrastructure requirements to run PCF. Infrastructure would be the combined resources that a single VxRail provides: Shared Storage - vSAN, RAM and Compute.

Specifics that are related as your application will be combined with advanced guidance from Pivotal. In general, defer to Pivotal solution experts for exact PCF sizing to match your organization requirements. AI sizing is dependent on your organization’s application. The sizing tool will help determine the total number of Application Instances possible hosted on specific VxRail.

PCF Sizing Background Information

PCF sizing is explained at a high level in the table below:

Once the AIs and PCF Services are sized, a total infrastructure summary is provided within the PCF Sizing tool.

Figure 10: PCF Sizing tool

Solution Sizing Results

For the Reference architecture configuration described in this document, a single VxRail Series G Model G410 appliance with 64 physical cores and 1TB of RAM was used. The appliance was able to handle both vertical and horizontal scaling. Vertical scaling is for larger applications with larger memory and storage allocations, while horizontal scaling represents increased number of application instance numbers. The sizing numbers suggested by the PCF sizing tool were validated.

The appliance could run over 500 AI’s configured to 0.5G average RAM and Storage as suggested in the Fig.9 above. Likewise, the appliance could run about 100 App Instances configured to 2G average RAM and Storage.

The Reference Architecture focused on horizontal scaling. Appendix C explains vertical scaling for PCF.

These numbers can be safely extrapolated to scale PCF with additional VxRail nodes and appliances. Multi-appliance scalability study was not done as part of this RA study.

VxRail utilization is balanced between high availability, failover and maximizing utilization of available resources. The utilization cannot be allocated beyond what is practically manageable. While over-allocation is possible for any given node in the VxRail appliance, consideration must be given for utilization to not exceed failover limitations. The VxRail appliance and vSAN are designed to tolerate a FTT =1.

Certain assumptions are made to ensure that high availability is maintained from the vSphere level. Balancing workload per host is designed to ensure running hot with high availability. Specifically, workloads are designed to not overload and to allow for enough flexibility in workload placement in case of host failure, hardware component failure or PCF application resilience. Ideally the high water mark is not to exceed 75% of total utilization per host for RAM and/ or CPU. vSAN is set to FTT level of 1, meaning it allows for the allowance of a single host failure with minimal interruption. The PCF-VxRail solution by design leverages vMotion, DRS and HA as well as the PCF components, deployed with High Availability and with built in PCF application high availability and recovery.

Conclusion

This section summarizes on how Dell EMC VxRail Appliances exceeds all minimum infrastructure requirements for PCF on vSphere for a standard mid to large PCF deployment.

The VMware reference architecture for Pivotal Cloud Foundry on Dell EMC VxRail is a prescriptive and pre-tested blueprint designed for fast deployment, ease of scale, and simple operation of Pivotal Cloud Foundry infrastructure.  It is a simple, scalable and agile cloud native solution, with enhanced security and automation. With VxRail, Pivotal customers can achieve:

• Rapid Time to Value with greater agility and security
• Infrastructure on demand with predictable cost and performance
• Mission critical data services for ultimate efficiency

Dell EMC VxRail Appliances exceeds all minimum infrastructure requirements for PCF on vSphere for a standard mid to large PCF deployment.  The total number of application instances will depend on the total available resources and requirements of the individual application.  However, the use of VxRail as hardware platform of choice simplifies and accelerates the time to value and easily allows you to quickly scale up to support larger PCF deployments.

Appendix A: PCF configurations used for this RA

This section provides information on the PCF configurations used for this Reference Architecture instance.

Below are the screen shots for the PCF tiles that were used in this specific RA instance.

Figure 11: Pivotal Elastic Runtime

Figure 12. Spring Cloud Services

Figure 13: MySql – multi node cluster

Figure 14: PCF Metrics

Figure 15: RabbitMQ - single node

Figure 16: Redis

Appendix B: PCF Availability Zone

This section describes the PCF Availability Zone.

A PCF Availability Zone (AZ) is a functionally independent segment of the network infrastructure. The deployment of a VxRail Appliance is configured as a single AZ. Pivotal recommends a minimum of 3 AZ’s to provide high availability. In order to achieve this with VxRail, you need 3 VxRail Appliances which are configured to use an external vCenter Server so each Appliance has its own separate vSAN Datastore.

During product updates and platform upgrades, the VMs in a deployment restart in succession, rendering them temporarily unavailable. During outages, VMs go down in a less orderly way. Spreading components across Availability Zones and scaling them to a sufficient level of redundancy maintains high availability during both upgrades and outages and can ensure zero downtime.

Deploying PCF across three or more AZs and assigning multiple component instances to different AZ locations lets a deployment operate uninterrupted when entire AZs become unavailable. PCF maintains its availability as long as a majority of the AZs remain accessible. For example, a three-AZ deployment stays up when one entire AZ goes down, and a five-AZ deployment can withstand an outage of up to two AZs with no impact on uptime.

Appendix C: Vertical Scaling

This section describes how PCF on VxRail supports both Vertical and Horizontal Scaling.

PCF on VxRail support both Vertical and Horizontal

PCF on VxRail supports both Vertical and Horizontal Scaling

You can scale PCF platform capacity vertically by adding memory and disk, or horizontally by adding more VMs running instances of Cloud Foundry components. However, this is by increasing the default configuration options for the PCF component instances.

Vertical scaling, to ensure that you allocate and maintain enough of the following:

• Free space on host VMs, whether they are Diego cells or DEAs, so that apps expected to deploy can successfully be staged and run.
• Disk space and memory in your deployment such that if one host VM is down, all instances of apps can be placed on the remaining Host VMs.
• Free space to handle one AZ going down if deploying in multiple AZs.

Horizontal Scaling

Horizontally scaling up the following components also increases your capacity to host applications. The nature of the applications you host on Cloud Foundry should determine how you should scale vertically vs. horizontally.

Horizontally scaling an application creates or destroys instances of your application. . Adding more instances allows your application to handle increased traffic and demand.

Incoming requests to your application are automatically load balanced across all instances of your application, and each instance handles tasks in parallel with every other instance.

Appendix D:VxRail Machine Resource Configuration

This section provides information on the VxRail Machine Resource Configuration.

References

This section lists the relevant references used for this document.

White Paper

For additional information, see the following white paper:

• What’s New with VMware vSAN 6.2

Product Documentation

VMware vSphere 6.0 Documentation Center

The following documents, available on the VMware website, provide additional and relevant information:

• VMware vSphere Installation and Setup Guide
• VMware vSphere Networking
• VMware vSphere Resource Management
• VMware vSphere Storage Guide
• VMware vSphere Virtual Machine Administration
• VMware vSphere Virtual Machine Management
• VMware vCenter Server and Host Management
• Installing and Administering VMware vSphere Update Manager
• Preparing the Update Manager Database
• Preparing vCenter Server Databases
• Understanding Memory Resource Management in VMware vSphere 6.0

Pivotal Documentation

• Installing Pivotal Cloud Foundry on vSphere
• Developer Marketplace
• Deploying Operations Manager to vSphere

Dell EMC documentation

The following documents provide additional and relevant information:

• Dell EMC VxRail Network Guide
• Dell EMC VxRail Appliance: HyperConverged Infrastructure Appliance from Dell EMC and VMware Tech Book

Other documentation

The following documents, available on the Microsoft website, provide additional and relevant information:

• Installing Windows Server 2012 R2
• SQL Server Installation (SQL Server 2012)

About the Author

This section provides a brief background on the author and contributors of this document.

Vuong Pham, Solution Architect in the Storage and Availability, Product Enablement team wrote the original version of this paper. Thanks to Prasad Bopardikar, Partner Solution Architect, Pivotal for his expertise and guidance with Pivotal Cloud Foundry.