Friday, May 1, 2020

vCSA 7.0 upgrade ate my STS_INTERNAL_SSL_CERT store

So a week after upgrading to vSphere 7 I can't login to vCenter anymore, I get

"HTTP Status 400 - An error occurred while sending an authentication request to the PSC Single Sign-On server"

Google confirms this is a thing that can occur post an upgrade and there are blog posts and KB articles on fixing it.
Including this one: https://kb.vmware.com/s/article/52541
which describes how to put the cert back into the store for the machine store when it has become corrupt.  I had the additional wrinkle in that my whole store was gone, when I tried any of the commands except for backing up the certs I got,

"Error: Failed to open the store.
vecs-cli failed. Error 4312: Possible errors:
LDAP error: Unknown (extension) error
Win Error: Operation failed with error ERROR_OBJECT_NOT_FOUND (4312)"

I'm pleased to say creating a new store with:

/usr/lib/vmware-vmafd/bin/vecs-cli store create --name STS_INTERNAL_SSL_CERT

was all it took, then following the procedure in the KB, restarting all services and I'm back.  I was also finally motivated to make use of the vSCA backup process, which is super easy and supports every protocol under the sun.  My 4 node cluster creates so few DB entries that a backup is only a gigabyte, so having it backup every night to my Synology at home and retain 7 of them is something long overdue.  I'm using NFS, the only wrinkle in setting it up being remembering where in Synology to add another host to the permitted list.