Configuring Distributed Switches in vCenter 6

Introduction to Distributed Switches

Version: vSphere 6.5 Update 1

Distributed vSwitches were first introduced in vSphere 4.0 - and since then various enhancements have been made as each subsequent release of vSphere has been released. At their heart, Distributed vSwitches are method of centralizing the management of the virtual network into single plane. Every VMware ESXi host added to Distributed vSwitch inherits its configuration, and those settings are stored within vCenter, rather than on the ESXi host itself. This means adding new portgroups for a new VLAN for a cluster of ESXi hosts are relatively trivial affair. The VMware ESXi host "caches" its Distributed vSwitch to local storage so if the vCenter is unavailable for whatever reason network communications are unaffected. However, no management of the Distributed vSwitch is possible until it is restored. For this reason some virtualization admins prefer that infrastructure VMs such as vCenter, SQL, Domain Controller and other VMware services and appliance remain on standard vSwitch to allow for continued management even if vCenter is offline.
Distributed vSwitch also off some features which are easier to configure than with Standard vSwitches, such as adjusting the MTU for Jumbo Frame support - in addition there are some unique features these include:

Part 5 : VMWare VCenter 6.7 : VCenter 6.7 VCHA configuration

This post will cover the vCenter High Availability part (VCHA). VCHA is pretty easy to enable as its available under the configuration section of vCenter server, with just 2 pre-requisites.
After enabling VCHA we will be having 2 extra VM’s. One is the vCenter passive node which will act as vCenter in the event that Active vCenter is down and Other VM is vCenter witness which will simply act as witness for split brain or cluster majority.
How Many DNS records and Management IP’s the vCenter will have after VCHA ?
the Management IP in our case vcenter-vc01.sslab.com (192.168.1.213) is the only one which will serve vCenter purposes. In the event the Active vCenter goes down, the passive vCenter will claim the management IP and it will serve the vCenter management part.
Can vCenter work if both active and passive fails with witness ?
the witness VM will not hold any data , it is just a quorum majority. so at any point vCenter can afford failure of either active or passive node only.

Contents of the Post

Part 4 : VMWare VCenter 6.7 : VCenter 6.7 Installation with External PSC

This post will cover the installation of vCenter server and pointing it to the external load balanced PSC which we had implemented in the last three posts of this series.
Just to recap First we had installed 2 PSC nodes in same SSO domain and site, Then we installed certificates on the PSC server. After that we had configured Netscaler load balancer for load balancing both PSC nodes. With this we are ready to start vCenter server installation.

Contents of the Post

Part 3 : VMWare VCenter 6.7 : PSC loadBalancing with NetScaler

In this post will cover the load balancing of PSC servers with Netscaler. If you are new to Netscaler or wanted to refresh on some basic concepts of Netscaler please view my Netscaler series here.

Contents of the Post

• Pre-requisites for Load Balancer
• Add PSC servers in Netscaler
• Create Services for PSC
• Create Load Balancing Virtual Servers
• Create Persistency Group for virtual servers
• Verify virtual servers and its services
• Quick Links for vCenter 6.7 Installation Series

Pre-requisites for Load Balancer

1. Names and IP of First and Second PSC nodes. vcenter-psc01.sslab.com , vcenter-psc02.sslab.com
2. One free IP used for load balanced virtual IP for PSC nodes.
3. DNS record to be created for PSC LB pointing to this virtual IP. vcenter-psclb01.sslab.com
4. Communication from Netscaler SNIP to both PSC nodes on ports 443, 389, 636, 2012, 2014 and 2020.
5. No certificate needs to be installed on Netscaler for this activity as we are using only TCP virtual servers.

Add PSC servers in Netscaler

First we need to both PSC nodes as servers in Netscaler.

Navigate to Configuration > Traffic Management > Load Balancing > Servers.
Select Add.
Provide Name and IP of First PSC
Click Create.

Similarly create the second PSC server. Both the PSC servers needs to be added as shown below.

Create Services for PSC

VCenter PSC nodes will work on ports 443, 389, 636, 2012, 2014 and 2020. Hence we need to create services for both PSC servers for all the 6 ports. So by the end of this section a total of 12 services will be created for both PSC nodes.

Navigate to Configuration > Traffic Management > Load Balancing > Services
Click ADD
Enter Service Name
Select Existing Server
Select the First PSC server
Click Protocol and select TCP
enter port no 443
Click OK to create the service, Click Done in the service overview page.

Note: Similarly create separate services with ports 389, 636, 2012, 2014 and 2020 with first PSC.

Click Done.

Now all the 6 services for PSC1 will show for ports 443, 389, 636, 2012, 2014 and 2020.

Similarly create services for Second PSC for ports 443, 389, 636, 2012, 2014 and 2020. At the end a total of 12 services will be visible with status UP as below.

Create Load Balancing Virtual Servers

As each PSC is working on ports 443, 389, 636, 2012, 2014 and 2020, so we need to create 6 virtual servers with same IP for each of these ports. Respective services created before for both PSC will be added to the virtual servers.
We need on Free IP which will be called the Virtual IP used for creating virtual servers. Same IP will be used with different ports for creating the virtual servers.  The Subnet IP of the netscaler should be able to reach the PSC nodes on ports 443, 389, 636, 2012, 2014 and 2020 for successful communication.

Navigate to Configuration > Traffic Management > Load Balancing > Virtual Servers.
Click Add
Enter a Name for virtual server
Click Protocol and then select TCP
Enter the Virtual IP Address
Click Port , enter 443
Click OK

Select load balancing virtual service binding

Select the respective services from PSC1 and PSC2. Like for virtual server 443 add services with port 443 only.

Click Bind to bind the services to virtual server.

Click Continue to create the virtual server

Review and Click Done.

Similarly create virtual servers for ports 389, 636, 2012, 2014 and 2020 and respective services. All the virtual servers status will show up and greeen as below.

Create Persistency Group for virtual servers

We need to create a persistency group with higher value like 1440 to make the vcenter to PSC session stable.

Navigate to Configuration > Traffic Management > Load Balancing > Persistency Groups
Click Add.
provide a name
Persistence select SOURCEIP
Time-out enter 1440.
Beside virtual server click the ADD or + button
Click the > button to move all six PSC virtual Servers to right as shown
Click Create.

Persistency group is created as below.

Verify virtual servers and its services

Now verify all the virtual servers are up and correct services with ports are bound to it. Click on each virtual server and verify the persistency is present as well.

With with load balancing PSC servers is completed will mode to next step of Installing vCenter server and pointing to psc load balanced DNS record.

Part 2 : VMWare VCenter 6.7 : PSC External Certificate Installation

Once Both the PSC are installed and configured we need to Replace the Certificate on Both PSC nodes with subnet alternate DNS records having First PSC, Second PSC and Load balanced PSC Name. We will cover the load balancer part in Part 3.

Contents of the Post

• Step 1: Create CSR request File
• Step 2: Download CSR and Request Certificate from CA
  • Fix : WINSCP connecting issues with PSC or vCenter
  • Option 1: Generating a certificate from the VMCA
  • Option 2: Generate Certificate from Microsoft CA or External CA
• Step 3: Upload Certificates and Create Certificate Chaining
• Step 4: Replace Current SSL certificates in First PSC
• Step 5: Replace SSL certificates in Second PSC & Verify
• Quick Links for vCenter 6.7 Installation Series

Step 1: Create CSR request File

This step will cover how to create a request file then CSR for SSO SSL certificate which needs to be requested either from the PSC iteself or from external third party CA. These certificates eventually needs to be installed on Both PSC.
CSR generation is covered under vmware article https://kb.vmware.com/s/article/2147627

1. First we need to connect to any PSC using putty or terminal
2. run shell to login to shell
3. create certs folder  under root directory using mkdir certs command
4. then change directory to certs using cd certs
5. create the config file using cat > filename -> enter -> paste the config file contents -> enter -> ctrl + z

Configuration file (psc_ha_csr_cfg.cfg) contents are below

[ req ]
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:false
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = DNS:vcenter-psc01.sslab.com, DNS:vcenter-psc02.sslab.com, DNS:vcenter-psc-lb.sslab.com
[ req_distinguished_name ]
countryName = AE
stateOrProvinceName = State
localityName = City
0.organizationName = SSLAB
organizationalUnitName = Department
commonName = vcenter-psc-lb.sslab.com

Note: subjectAltName = DNS:vcenter-psc01.sslab.com, DNS:vcenter-psc02.sslab.com, DNS:vcenter-psc-lb.sslab.com , these are the names of First PSC, Second PSC and Load balanced VIP Name.
Note: Common name should be load balanced VIP name: commonName = vcenter-psc-lb.sslab.com
Sample Output for all steps are given below.

Command> shell
Shell access is granted to root
root@vcenter-psc01 [ ~ ]# pwd
/root
root@vcenter-psc01 [ ~ ]# mkdir certs
root@vcenter-psc01 [ ~ ]# ls
certs
root@vcenter-psc01 [ ~ ]# cd certs

root@vcenter-psc01 [ ~/certs ]# cat > psc_ha_csr_cfg.cfg
[ req ]
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:false
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = DNS:vcenter-psc01.sslab.com, DNS:vcenter-psc02.sslab.com, DNS:vcenter-psc-lb.sslab.com
[ req_distinguished_name ]
countryName = AE
stateOrProvinceName = State
localityName = City
0.organizationName = SSLAB
organizationalUnitName = Department
commonName = vcenter-psc-lb.sslab.com

root@vcenter-psc01 [ ~/certs ]# cd ..

Run below command to create the certificate request file psc-ha-vip.csr with keyfile using the above created config file.
openssl req -new -nodes -out /certs/psc-ha-vip.csr -newkey rsa:2048 -keyout /certs/psc-ha-vip.key -config /certs/psc_ha_csr_cfg.cfg
If you receive any directory /certs not found errors, remove / before certs and run command as shown below.
openssl req -new -nodes -out certs/psc-ha-vip.csr -newkey rsa:2048 -keyout certs/psc-ha-vip.key -config certs/psc_ha_csr_cfg.cfg
Sample output is given below.

Step 2: Download CSR and Request Certificate from CA

Fix : WINSCP connecting issues with PSC or vCenter

We need to connect to PSC with WINSCP however there are known errors that you might get SFTP server or buffer size errors then follow below steps to enable bash shell so that PSC will allow secure SFPT connections.
First enable bash shell on the PSC appliance

• Bash shell can be enabled form vm console -> F2 -> trouble shooting options  or
• login to PSC using https://fqdn:5480 ->access -> edit -> Bash shell -> enable and give some no like 90 min. ( as shown below)

Then from the putty or terminal SSH session -> shell -> run below command to change to bash shell
chsh -s /bin/bash

After fixing the WINSCP issue, connect and download the CSR file from PSC.

Option 1: Generating a certificate from the VMCA

Run this command to create the certificate from the psc-ha-vip.csr and the the psc_ha_csr_cfg.cfg file outputting a psc-ha-vip.crt file.
openssl x509 -req -days 3650 -in /certs/psc-ha-vip.csr -out /certs/psc-ha-vip.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile /certs/psc_ha_csr_cfg.cfg
Run this command to copy the current VMCA root certificate and rename it to cachain.crt.
cp /var/lib/vmware/vmca/root.cer /certs/cachain.crt
Run this command to create Machine SSL Certificate that contains the newly created certificate and the VMCA root certificate named psc-ha-vip-chain.crt.
cat /certs/psc-ha-vip.crt >> /certs/psc-ha-vip-chain.crt
cat /certs/cachain.crt >> /certs/psc-ha-vip-chain.crt

Option 2: Generate Certificate from Microsoft CA or External CA

The certificate can be requested form an external CA as well, in my case microsoft CA is in use. Select Advanced Certificate request and request the certificate as shown below.

• Once certificate is requested select Base 64 and download the certificate – Rename it to psc-ha-vip.crt
• Click on Home and download the ROOT certificate – Rename it to RootCA.crt

Step 3: Upload Certificates and Create Certificate Chaining

Using WINSCP upload server certificate psc-ha-vip.crt and Root certificate RootCA.crt to /certs folder in PSC appliance.
This command will create psc-ha-vip-chain.crt file, which contains all the certificates in chain from server certificate , then intermediate and ROOT certificates. Our case no intermediate certifiacate authorities are present.

cat /certs/psc-ha-vip.crt >> /certs/psc-ha-vip-chain.crt
cat /certs/RootCA.crt >> /certs/psc-ha-vip-chain.crt

This command will create cachain.crt file, which includes the ROOT and all intermediate CA certificates.

cat /certs/RootCA.crt >> /certs/cachain.crt

If in case there are intermediate certificates are present then below will help.
cat /certs/psc-ha-vip.crt >> /certs/psc-ha-vip-chain.crt
cat /certs/CustomInterCA1.crt >> /certs/psc-ha-vip-chain.crt
cat /certs/CustomInterCA2.crt >> /certs/psc-ha-vip-chain.crt
cat /certs/CustomRootCA.crt >> /certs/psc-ha-vip-chain.crt
If there is intermediate certificates, run these commands to create a cachain.crt of the intermediate certificates and the root certificate.
cat /certs/CustomInterCA1.crt >> /certs/cachain.crt
cat /certs/CustomInterCA2.crt >> /certs/cachain.crt
cat /certs/CustomRootCA.crt >> /certs/cachain.crt

Step 4: Replace Current SSL certificates in First PSC

verify using ls command under /certs folder  for psc-ha-vip-chain.crt , psc-ha-vip.key and cachain.crt files are present.
To replace the SSL Certificate SSH to the PSC node, login to shell and run below command as shown.

/usr/lib/vmware-vmca/bin/certificate-manager

Select First Option 1 – Then Option 2

When Prompted for file path provide below information as shown.

• custom certificate file : /certs/psc-ha-vip-chain.crt
• custom key file : /certs/psc-ha-vip.key
• signing certificate (CA) file : /certs/cachain.crt

Note: If getting any path errors remove the / before certs, as we are running it under root as shown below.

It will update all the services, stop and start them as shown below.

Restart the PSC appliance after this.

Step 5: Replace SSL certificates in Second PSC & Verify

Copy the complete certs folder in First PSC to your computer  using WINSCP , then copy the complete certs folder to Second PSC appliance.
Login to Second PSC appliance using putty or terminal and repeat Step 4 for the second PSC.
Once the certificate assigning is completed and PSC is restarted, open the fqdn of both psc from browser and verify that certificate is in place and without any errors.
Note: if the CA root certificate is not installed on the machine from which you are opening in browser you might get cert error. in that case verify cert and ignore warning.

Next Load balancer configuration needs to be completed.

Part 1 : VMWare VCenter 6.7 : External PSC for LB Step-By-Step Installation

Having External PSC and load balancing them has become a common trend from vCenter 6.5. This post will cover the installation and configuration of 2 vCenter 6.7 PSC appliances, with one Single Sign On Domain and One site.

Contents of the Post

• Pre-requisites for this post
• First PSC installation and Configuration
  • Stage 1 of First PSC : PSC appliance installation and basic configuration
  • Stage 2 of First PSC : SSO Configuration
• Second PSC installation and Configuration
  • Stage 1 of Second PSC : PSC appliance installation and basic configuration
  • Stage 2  of Second PSC: Joining to SSO domain & Site
• Quick Links for vCenter 6.7 Installation Series

Pre-requisites for this post