Restoring vCenter access after being blocked by a Deny All rule in NSX DFW

Restoring vCenter access after being blocked by a Deny All rule in NSX DFW

The issue

I recently had an issue to solve with an NSX(-v) instance in which the default firewall rule had been set to reject without an exception defined for the vCenter Server.
As a result, the communication between the vCenter Server appliance and NSX Manager was interrupted, which blocked vCenter Server completely from accessing the network.
I decided to document the procedure, which was necessary to get it working again, just in case I would have to solve it again, but I also thought that sharing it would do no harm.

Basically the following procedure is based on VMware KB 2079620: vCenter Server access is blocked after creating a Deny All rule in DFW (2079620), but as I faced additional issues during troubleshooting, I document the whole procedure which I had to do.

Unblock vCenter

The first thing I did was get vCenter Server to communicate again. vCenter and NSX Manager were completely unavailable, which is why the first thing I had to do was to remove the NSX vibs from the underlying host.

  1. First shut down vCenter Server appliance, and other remaining VMs on the ESXi host.
  2. Then put host in maintenance mode, which is required to remove the NSX vibs.
  3. Connect via ssh to the host (e.g. using PuTTY). Note that if ssh is not enabled on the host, it has to be enabled first.
  4. Check for the NSX vibs installed on the host using the command esxcli software vib list | grep nsx

[root@hostname:~] esxcli software vib list | grep nsx
esx-nsxv         6.7.0-0.0.8590012         VMware     VMwareCertified   2018-11-13

  • Remove the NSX vib:

esxcli software vib remove --vibname=esx-nsxv

Removal Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed:
VIBs Removed: VMware_bootbank_esx-nsxv_6.7.0-0.0.8590012
VIBs Skipped:

  • Exit the host from maintenance mode
  • Restart vCenter

 

—————–

Side note:

Depending on the situation you might be facing, or the point you step in the troubleshooting, the situation can be slightly different. Sometimes somebody else has already begun troubleshooting, and vCenter Server can be disconnected already from the vDS.

I already posted about this in this article.

In the case that the portgroup is a non-ephemeral one, no new ports can be allocated as the vCenter Server is down. In that particular case a new standard vSwitch has to be created, vCenter needs to be connected to the VSS for the boot process, until it can be reconnected to the vDS again.

If you encounter the same issue, simply follow the procedure from the above linked article.

—————–

 

Remove the blocking firewall rule in NSX Manager via an API call

 

Now the vibs are removed and vCenter Server can be accessed again. However, as in this case there is no exclusion defined in the distributed firewall of the vCenter Server, the default firewall rule has to be reset to default.
In order to do so, you will have to work with REST API. If you have already a favourite REST API client, you are free to use what you want. If not, you can download some for free, as for instance Postman, which also exists as extension for Chrome. (The standalone version of Postman can be downloaded here: https://www.getpostman.com).

Note: If you want to know more about Postman, feel free to check Romain’s blogpost here.

Here is the procedure I followed:

  1. If not already done, download and install Postman, or use your favourite REST API client. Launch your REST API client.
  2. Form a request using the method GET to test connectivity to the NSX Manager. Using the URL below you will be able to download the actual firewall configuration, which might help in identifying the blocking rule.
  • Choose Basic Auth as authentication type
  • Enter Username and password for your NSX Manager
  • Choose GET as method
  • Enter following URL: https://NSX-MANAGER-IP/api/4.0/firewall/globalroot-0/config
  • Hit Send

Note the return code is “200 OK”, meaning the GET the request was successful.

  1. Verify the output. If you want to keep a copy of your firewall configuration export you just pulled, you can save it from here before continuing.
  2. Now form the request to reset the default configuration of the default firewall rule. Form the same request as before using the same parameters, but this time using the method DELETE instead of GET:
  • Choose Basic Auth as authentication type
  • Enter Username and password for your NSX Manager
  • Choose GET as method
  • Enter following URL: https://NSX-MANAGER-IP/api/4.0/firewall/globalroot-0/config
  • Hit Send

If the return code is 204, the request was successful.

  1. Now your blocking firewall rule should be gone. You can test it by connecting to the vCenter Server Appliance.
  • Go to Menu –> Networking and Security
  • Note that the connection to NSX Manager has been re-established if the NSX Manager can be found.
  • Now you can reconfigure the firewall rules as you wish.

Additional Note:

To avoid this situation in future, it is recommended to add the vCenter Server in the exclusion list of the NSX distributed firewall settings.
This can be done by going to Firewall Settings à Exclusion List à User Excluded VM(s).

Click ADD to add the vCenter to the exclusion list.

Select your vCenter VM, then click the à arrow to ad dit to the selected objects. Then hit OK.
Now the vCenter will not be blocked anymore by the distributed firewall rules of the NSX Manager.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.