Holodeck: Getting Started with NSX Microsegmentation

Module 2: Changing the Security Game with Microsegmentation

Over the past 10+ years traffic in a datacenter has changed.  More and more traffic remains within the datacenter, moving between distributed application components.  This traffic, known as East-West, is difficult to secure using traditional perimeter firewalls, which were predominantly designed for traditional North-South traffic.

Microsegmentation enables administrators to increase the agility and efficiency of the datacenter while maintaining an acceptable security posture. Microsegmentation decreases the level of risk and increases the security posture of the modern data center.

Microsegmentation with NSX in VCF is applied at the vNIC to VDS interface. Packets are inspected as they enter and leave each virtual machine. Microsegmentation is effectively a centralized packet filtering solution that acts on every machine.

 

Lab 1: Tagging VMs and Grouping Workloads based on Tags

This lab explores the use of tagging to create groups of VMs to apply specific distributed firewall rules to. In small environments, creating groups based on VM name may suffice. However, as an environment grows, tagging may be a better alternative.

Terminology and definitions:

• Tags – A virtual machine is not directly managed by NSX, however, NSX allows attachment of tags to a virtual machine. This tagging enables tag-based grouping of objects. For example, a tag called AppServer can be associated to all application servers). 
• Security Groups – A security group is a collection of assets or grouping objects from your vSphere inventory.

Security Groups are containers that can contain multiple object types including logical switch, vNIC, IPset, and Virtual Machine (VM). Security groups can have dynamic membership criteria based on security tags, VM name or logical switch name. For example, all VMs that have the security tag web will be automatically added to a specific security group destined for Web servers. After creating a security group, a security policy is applied to that group.

• Security Policies – A security policy is a set of Guest Introspection, firewall, and network introspection services that can be applied to a security group. The order in which security policies are displayed is determined by the weight associated with the policy. By default, a new policy is assigned the highest weight so that it is at the top of the table. However, you can modify the default suggested weight to change the order assigned to the new policy.  Policies can be stateful or stateless.

Note: Tagging in NSX is distinct from tagging in vCenter Server. At this time, vCenter Server tags cannot be used to create groupings in NSX. In larger, more automated environments, customers use a solution such as vRealize Automation to deploy virtual machines and containers with security tagging set at time of creation.

Given that the OpenCart application used in this lab only has two web servers and one database server, we’re going to create two tags as criteria for two groups. This might seem somewhat redundant, creating one tag per group, however it’s essential to remember:

• This is a small sample two-tier application. For applications leveraging micro-services, you’ll be able to group more than one machine under one tag, and better leverage the security groups
• The advantage of using tags and groups is also an operational one. Once you create your infrastructure around Security Groups that contain tags, the moment you tag a machine with a specific tag, it immediately inherits the specific Security Group, firewall rules and so on. This brings us closer to the cloud delivery model.
• The downside is that a certain level of caution needs to be implemented when working with tags and Security Groups, meaning that it’s just as easy to add a machine to an existing Security Group and avoid the complication that comes with setting up the firewall rules, but it is also just as easy to evade good security by giving the new machine too many permissions due to old tags/security group configurations.

To show the capability of tags we will set up OC-Apache-A with the appropriate Tags and Security Group. And then we’ll have OC-Apache-B inherit the web tag and see how easy it is to apply all the appropriate rules to a new machine.  The VM->Tag->Group mapping is as follows

VM	IP Address	Tag	Security Group
OC-MySQL	10.1.1.50	OC-DB-Tag	OC-DB-Group
OC-Apache-A	10.1.1.18	OC-Web-Tag	OC-Web-group
OC-Apache-B	10.1.1.19	OC-Web-Tag	OC-Web-Group

 

[Step 1] Create Tags

1. On the Holo-Console, open a new tab in the Chrome browser
2. Click the Managed bookmarks folder in the bookmark bar then select Mgmt Domain->Mgmt NSX
3. Log into NSX Manager as the user admin with the password VMware123!VMware123!
4. Navigate to Inventory->Tags
5. Click on ADD TAG

 

6. We will first create a new tag for the web servers. Add the name OC-Web-Tag in the Tag field. Don’t hit save yet. As you can see, the note says the tag must be assigned to at least one object first.

 

 

7. Assign this tag to the OC-Apache-A virtual machine. Click on the Set Virtual Machines link field and select the OC-Apache-A virtual machine (you may need to scroll). For now, do not select OC-Apache-B as it will added later

8. Click APPLY
9. Note the “Assigned To” value has incremented to 1
10. Click SAVE to save the new tag

11. Click Add Tag
12. Add the name “OC-DB-Tag”

13. Click Set Virtual Machines then select OC-MySQL
14. Click Apply
15. Click Save

Step 2: Verify Tags

1. Click on Inventory->Tags->Filter by Name, Path and more
2. Click Tag

 

3. Search for Tags with the string “OC” in the name and click OK

4. Verify the two tags created earlier are displayed

 

 

Step 3: Verify virtual machines are mapped to tags.

1. Select Inventory->Virtual Machines and click in the Filter area
2. Scroll down in Basic Detail and select Tag

 

3. Select our two tags and click Apply

 

4. Verify OC-Apache-A and OC-MySQL are present

 

Step 4: Create Groups

Group mapping is as follows

Tag	Security Group
OC-DB-Tag	OC-DB-Group
OC-Web-Tag	OC-Web-Group

Step 5: Create OC-Web-Group

1. Click Inventory->Groups->ADD GROUP

2. Add group name OC-Web-Group

 

3. Click on “Set” in the Compute Members column to add group members. In this example we will use the tags we just created to populate the group.
4. Click Add Criterion

5. Select Virtual Machine, that filters on Tag Equals “OC-Web-Tag” 

 

6. Click Apply
7. Click Save

Step 6: Create OC-DB-Group

1. Click Inventory->Groups->ADD GROUP

 

2. Add group name OC-DB-Group
3. Click on Set to add group members. In this example we will use the tags we just created to populate the group

 

 

4. Click Add Criterion
5. Click on the Scope and select the OC-DB-Tag tag

 

 

6. Click Apply
7. Click Save

Step 7: Verify Groups

1. On the Inventory->Groups panel click in the Filter by Name, Path and More field

2. Click on Name in the Basic Detail column

 

3. Type OC to filter for our group names
4. Select the OC-Web-Group and OC-DB-Group groups
5. Click Apply

6. Click View Members for each group
7. Each group should have a single VM as a member (we can ignore IP addresses, Ports and VIF for now).

 

The following example shows the view when looking at the View Members details for the OC-DB-Group

 

8. At this point we have implemented the following:

 

VM	Tag	Group	Configured
OC-Apache-A 10.1.1.18	OC-Web-Tag	OC-Web-Group	Yes
OC-Apache-B 10.1.1.19			No
OC-MySQL 10.1.1.50	OC-DB-Tag	OC-DB-Group	Yes

 

[Lab 1 Summary]

Lab 1 shows how to create tagging and grouping in NSX. This capability allows creation and management of a scalable set of distributed firewall rules.

Lab 2: Applying Distributed Firewall Rules Based on Tagging on a segment

This lab will show configuring the distributed firewall to limit access in our Opencart Application. For the purposes of this lab, we will create the following rules.

Name	Source	Destination	Port/Protocol	Allowed	Notes
Inbound-web-80	Any	OC-Web-Group	HTTP (80)	Allow	Outside to web port 80
Inbound-web-8080	Any	OC-Web-Group	HTTP (8080)	Reject	Outside to web port 8080
Web-DB	OC-Web-Group	OC-DB-Group	3306 (MySQL)	Allow	Web to DB comms
ssh-admin	10.0.0.0/24	OC-DB-Group OC-Web-Group	SSH	Allow	SSH from lab console only
ICMP-Admin	10.0.0.0/24	OC-DB-Group OC-Web-Group	ICMP ALL	Allow	Allow ICMP only from lab console
Deny-All-Inbound	Any	OC-DB-Group OC-Web-Group	Any	Reject	Reject all else inbound

 

Keep in mind that this all happening at the distributed firewall level, where firewall rules are implemented at the VM switch port versus needing the services of a routed (perimeter) firewall to implement. Since we have created groups in the previous lab, now we can create access rules based on these groups. 

Step 1: Create New Policy

1. If necessary, open a new tab in the Chrome browser
2. Click the Management NSX-T shortcut in the bookmark bar (click advanced / proceed to nsx-mgmt.vcf.sddc.lab, if required to accept the certificate
3. )
4. Log into NSX Manager as user: admin with the password: VMware123!VMware123!
5. Navigate to Security > Distributed Firewall in the NSX-T Console
6. Click on Add Policy > New policy, Name the policy “Opencart”

 

7. Hover over DFW in the Opencart policy row, then click the pencil twice.
8. Note the default is the entire distributed firewall, however we want this rule to apply to the groups we created in the previous labs.
9. Click the Groups radio button
10. Search for OC then select OC-DB-Group and OC-Web-Group

11. Click Apply

Step 2: Add inbound-web-80 rule

1. Click on the three vertical dots on the left of the Opencart policy.
2. Click Add Rule

3. Click on Add Rule and change name to inbound-web-80
4. Hover over Destinations, then click the pencil icon twice
5. This brings up the Set Destination pop-up. Type OC in the search box then click on OC-Web-Group checkbox, then apply

 

6. Hover over Services, then click the pencil icon
7. Type HTTP in the services area then select HTTP

 

8. Click Apply
9. The result should be the following:

 

Step 3: Add inbound-web-8080 rule (clone inbound-web-80)

1. Click on the three vertical dots on the left of the inbound-web-80 rule.
2. Click Clone Rule

 

3. Click on Copy of inbound-web-80 and change name to inbound-web-8080
4. Hover over Services, then click the pencil icon
5. Uncheck the box for the HTTP service
6. Click on Raw Port Protocols
7. Click Add Service Entry
8. Set Service Type TCP and Destinations Port 8080

 

9. Click Apply
10. The result should be the following:

 

Step 4: Test inbound-web-XX rules

1. Notice at this point we have two rules in place that are defaulted to Allow, and we have not yet published the rule changes. Leave this as is for the moment. Next, we will test that both ports are currently active on our web server

 

2. Open a tab on the browser and connect to OC-Apache-A 10.1.1.18

 

3. Open a second tab and connect to OC-Apache-A 10.1.1.18:8080

 

4. Return to the NSX Manager tab.  Click the arrow next to Allow on inbound-web-8080 and select Reject

 

5. Click PUBLISH to make the rules active
6. Notice that the Publish button is greyed out, showing there are no uncommitted changes.  The green Success indicator is set at the policy level, and our two rules now have ID numbers showing they have been activated. You may need to click the refresh icon next to In Progress to show success.

 

7. Go to the OC-Apache-A 10.1.1.18 browser tab and refresh. The web page should load normally

8. Go to the OC-Apache-A 10.1.1.18:8080 browser tab and refresh. The web page should fail to load

Step 5: Extend security policies to new VM based on tag

1. Test operations of the OC-Apache-B 10.1.1.19 webserver on port 80

2. Test operations of the OC-Apache-B 10.1.1.19 webserver on port 8080

3. Observe that the Reject rule on port 8080 does not extend to OC-Apache-B. 
4. Return to the NSX tab and click on Inventory then Tags
5. Click on Filter, select Tag and search for by Name and type OC-Web

6. Click the 3 dots next to OC-Web-Tag, then click Edit
7. Click on the number 1 under Assigned To
8. Search for OC-Apache and select OC-Apache-B then click Apply

9. Click Save

10. Test operations of the OC-Apache-B 10.1.1.19 webserver on port 8080

 

 

11. Notice that as soon as the tag was applied to OC-Apache-B VM, it immediately became a part of the Opencart Policy, because it became a member of the web-group that the rules applied to

Step 6: Implement Web-DB rule

The step will Allow communications from only the Apache web servers to MySQL

1. On the NSX Manager tab, go to Security -> Distributed Firewall
2. Click on the 3 dots next to OpenCart and Add Rule

3. Name the rule Web-DB.

4. Set Sources to OC-Web-Group and click Apply

 

5. Set Destinations to OC-DB-Group and click Apply

6. Set Services to MySQL and click Apply

7. Your rule should look like this

8. Click Publish
9. Test access to OC-Apache-A on 10.1.1.18. You should get a normal web page load

 

10. Return to the NSX tab and set the Web-DB rule to Reject and click Publish (this will allow us to see the impact of the firewall blocking access from Apache to MySQL which will be useful later in the lab)

 

11. Test access to OC-Apache-A on 10.1.1.18. Your web page load should fail

12. Reset the Web-DB rule to Allow and Publish before moving on to the next step

Step 7: Implement Deny All Inbound rule

The step will implement a Deny All Inbound rule which will deny all inbound traffic we have not explicitly allowed.  This step will also show the order of rule evaluation within a security policy

1. On the NSX Manager tab, go to Security -> Distributed Firewall
2. Click on the 3 dots next to OpenCart and Add New Rule
3. Name the rule Deny-All-Inbound.

4. Leave Sources at Any
5. Set Destinations to groups OC-DB-Group and OC-Web-Group (Hint: Type OC- and enter to quickly show OC groups) Check the two groups and then Apply

6. Set Action to Reject, then click Publish

7. Test access to OC-Apache-A on 10.1.1.18. Your web page load should fail. This is because the Deny-All rule is evaluated prior to our Allow rules

8. Return to the NSX tab
9. Move the deny-all rule down by clicking the mouse and holding down with the cursor anywhere on the Deny-All rule line and dragging the rule to below our inbound-web-80 rule then clicking Publish

10. Test access to OC-Apache-A on 10.1.1.18. You should get a normal web page load

11. Return to the NSX tab
12. Click on the tree dots to the left of the inbound-web-8080 line and Delete Rule (since we have a hard Deny-All, unless port 8080 is explicitly allowed, it will be blocked, rendering this rule no longer needed.)
13. Publish the rules. You may need to click Refresh on the bottom of the policy screen

14. Go to the OC-Apache-A 10.1.1.18:8080 The web page should fail to load.

Step 8: Implement ssh rule

The step will implement a rule to allow ssh connections to our Apache and MySQL VM’s, but only from our inside admin network (10.0.0.0/24)

1. Attempt to SSH to OC-Apache-A. Click on PuTTY and connect to 10.1.1.18
2. Your connection should time out

3. Return to the NSX tab
4. Click on the three dots to the left of the Opencart policy and Add Rule
5. Name the rule ssh-admin

6. Click on the pencil for Sources. Then IP Addresses. Set to 10.0.0.0/24, then click Apply

7. Click on the pencil for Services. Select SSH, then click Apply

8. Publish your new rule

9. Attempt to SSH to OC-Apache-A. Click on PuTTY and connect to 10.1.1.18
10. Your connection should succeed.
11. Login as ocuser, password VMware123!

12. Attempt to ssh from OC-Apache-A to OC-Apache-B 10.1.1.19
13. Your connection should be refused

14. Observe that we have blocked ssh within the Web servers but allow between our admin network and the web servers.

Step 9: Implement ICMP-Admin rule

The step will implement a rule to allow ICMP (Ping) to our Apache and MySQL VM’s, but only from our inside admin network (10.0.0.0/24), as Ping is used to determine host accessibility in many security threat situations.

1. Open a command prompt on the Windows desktop by clicking the window icon -> Windows System ->  Command Prompt

 

2. Attempt to ping to OC-Apache-A 10.1.1.18. Your connection should time out

3. On the NSX Manager tab, go to Security -> Distributed Firewall
4. Click on the 3 dots next to Opencart and Add New Rule
5. Name the rule ICMP-Admin

6. Click on the pencil for Sources. Then IP Addresses. Set to 10.0.0.0/24, then click Apply

7. Click on the pencil for Services. Select ICMP-ALL, then click Apply

8. Publish your new rule

9. Return to your Windows command prompt
10. Attempt to ping to OC-Apache-A 10.1.1.18. Your connection should succeed

11. From the NSX-T Manager interface click the Plan & Troubleshoot tab
12. Click Traceflow in the left navigation panel
13. Configure Traceflow from OC-Apache-A to OC-MySQL and click Trace

 

14. Traceflow should show the ICMP packet dropped at the first firewall point (OC-Apache-A) before being placed on the segment

15. Review the observations panel and notice the packet was dropped at OC-Apache A between the VM NIC and OC-Web-Segment

16. On the Traceflow screen, click EDIT to reconfigure the trace then click Proceed on the warning pop up
17. Change Protocol Type from ICMP to TCP, with Destination Port 3306

18. Click Trace.  Your Traceflow should succeed

19. In the Observations panel, review the following
    • We show 1 packet delivered
    • The packet was injected at the network adapter for OC-Apache-A virtual machine
    • It is then received at the distributed firewall at the VDS port for OC-Apache-A
    • With no rule blocking, the packet is then forwarded on from the sending VDS port
    • The packet then hits OC-T1 router and gets forwarded to the OC-DB-Segment
    • Since OC-Apache-A and OC-MySQL are running on different ESXi hosts, you notice the physical hop between TEP’s
    • The packet is then received on the distributed firewall at the receiving VDS port for OC-MySQL
    • With no rule blocking forwarding, the packet is then forwarded to the destination, the last step shows the packet being delivered to the network adapter for the OC-MySQL VM

20. Notice the flexibility of Traceflow to allow us to troubleshoot our distributed firewall and distributed routing using appropriate communications protocols.

[Lab 2 Summary]

Lab 2 shows the power of the distributed firewall capability in NSX. Using tagging a grouping, we were able to create a scalable set of rules for our Opencart application that only allow necessary communications for application operation, while blocking all other traffic. This was all done directly at the vSphere VDS switch port level, versus a piece of hardware elsewhere in the datacenter.