Be careful upgrading NSX-T with vDS

Hit a good issue this week upgrading a customer from 3.2.1 to 3.2.2, but the issue also occurs if you're upgrading to 4.

There's a new vDS property which is set if you do a deployment of NSX-T onto vDS with a current version, com.vmware.nsx.vdsSecurity.enabled, the problem is that it isn't set during the upgrade process, until too late I think (I think the Managers add it after they are upgraded, but add you do the Edges then hosts, then Managers that doesn't help a whole lot).  So as each host is upgraded in turn and the Upgrade Manager takes each out of maintenance mode, the newly upgraded hosts are not in a state where they can receive VMs migrated by DRS when the next host is placed in maintenance mode.

Result, your VMs gradually get concentrated onto fewer and fewer hosts until the process fails.  Hopefully your remaining hosts aren't so overloaded that you/DRS can no longer migrate VMs back off them.

Workaround is to run on each host prior to upgrade:

net-dvs  -s com.vmware.nsx.vdsSecurity.enabled -p hostPropList <vDS name>

Confirm it's present with 

net-dvs -l | grep com.vmware.nsx.vdsSecurity.enabled

The host may require a reboot after setting it.

KB:

https://kb.vmware.com/s/article/90298