Supermicro IPMI yet again

So I broke one of my ESXi hosts by installing 7.0U2 as a patch baseline instead of an upgrade baseline in Lifecycle Manager (formerly Update Manager).  Failed to boot getting stuck at 'loading crypto...'

Easy fix supposedly, boot from CD and upgrade install over the top of the existing install, boot right back into the cluster.

Forums abounded with other people hitting the same thing and using iDRAC, iLO etc to mount the image and recover, I tried to do the same with Supermicro IMPI.  That's how I installed 6.7 on these in first place so I knew I had the capability to mount an ISO from an SMB share.  However install was at home where I was mounting the images off a Synology.  I dimly remembered having to mess with Synology but couldn't remember just how.

After wasting a long time trying to mount the ISO of a Windows box I gave up and installed a fresh Ubuntu VM to use, figuring correctly that Samba logging would help me figure it out.  Supermicro's SMB client not only speaks only SMB 1, 'server min protocol = NT1' but also doesn't support any decent authentication methods.  So after also adding 'ntlm auth = yes' the mount worked and I could recover.  The Samba VM got 150 random SMB hits from the Internet during its brief lifespan too, though all either zero length log files or ones filled with auth failures.  (My IPMI ports are out on the internet but with ACLs to limit access to just some static IPs I have access to, I'm not completely crazy)

[global]

server min protocol = NT1

ntlm auth = yes

[shared]

path = /home/simon/shared

valid users = simon

read only = no

To get IPMI settings of local system from ESXi:

localcli hardware ipmi bmc get

esxcli hardware ipmi bmc get

What to do when VCSA 7 runs out of space

 In my case ‘var/log’ was full, it being one of the smaller 10GB virtual disks.

The beauty of vCSA having 16 disks all in separate files is the ease with which you can grow one.

Get onto the console via virtual console or SSH, run a shell, then you can 'df -h' to confirm the full mount point, then use 'lsblk' to trace that back from it’s ‘Dev/wrapper’ mountpoint to an actual device like ‘Dev/sde’.  E being the 5th letter of the alphabet correlates with it being a 10GB device here and also my disk 5 in the VM settings.

Now take a backup.  Of course you're already doing nightly backups but then check that they're actually working, mine hadn't been for six weeks without my noticing due to an NFS permissions issue.  

Gracefully shutdown vCSA taking note of which host it’s on.  Connect to that host and edit settings for the vCSA, edit that virtual disk to increase its size, feel free to expand any other disks while you're there, it's not like most virtual storage isn't thin provisioned anyhow.  I took the opportunity to increase my RAM and CPU count too as I’m not resource constrained and I figured 4 vCPUs and 24GB would make my vCenter snappier.  Power back on and get a coffee while it boots/starts services.  

If you get 'editing host resources is disabled because this host is managed by vCenter' you can workaround by SSHing to the host and restarting vpxa and hostd - this will kick you out of the GUI, but then once you have re-authenticated you can make changes.

Console or SSH in again, open a shell and run the ‘/usr/lib/applmgmt/support/scripts/autogrow.sh‘ script, it should find your extra space and grow both the partition and the file system.

Done.  

vCSA 7.0 upgrade ate my STS_INTERNAL_SSL_CERT store

So a week after upgrading to vSphere 7 I can't login to vCenter anymore, I get

"HTTP Status 400 - An error occurred while sending an authentication request to the PSC Single Sign-On server"

Google confirms this is a thing that can occur post an upgrade and there are blog posts and KB articles on fixing it.
Including this one: https://kb.vmware.com/s/article/52541
which describes how to put the cert back into the store for the machine store when it has become corrupt.  I had the additional wrinkle in that my whole store was gone, when I tried any of the commands except for backing up the certs I got,

"Error: Failed to open the store.
vecs-cli failed. Error 4312: Possible errors:
LDAP error: Unknown (extension) error
Win Error: Operation failed with error ERROR_OBJECT_NOT_FOUND (4312)"

I'm pleased to say creating a new store with:

/usr/lib/vmware-vmafd/bin/vecs-cli store create --name STS_INTERNAL_SSL_CERT

was all it took, then following the procedure in the KB, restarting all services and I'm back.  I was also finally motivated to make use of the vSCA backup process, which is super easy and supports every protocol under the sun.  My 4 node cluster creates so few DB entries that a backup is only a gigabyte, so having it backup every night to my Synology at home and retain 7 of them is something long overdue.  I'm using NFS, the only wrinkle in setting it up being remembering where in Synology to add another host to the permitted list.

10 Gigabit home networking

Yes I know it's mostly absurd / just for bragging rights, but the Microtik CRS305-1G-4S+IN boxes made it possible for not too crazy an expenditure.  These have one gigabit copper port plus 4 SFP+ sockets for 10 gigabit modules - I'm using a mixture of 10Gtek 10GBase-TX modules and DAC cables to my Synology / Supermicro box which have SPF+ connections themselves.

I have a pair of Synology NAS (one of which is running VMs within via KVM), plus a single ESXi server and a Mac with 10 gig, plus all the usual TVs and home Internet devices.  My office is in the garage, a 100M run from the wiring closet (alright coat closet, but also where all the Cat 5 runs go to) in the house.  What I wanted the flexibility to do was site one or both Synologies in the house while retaining 10 gig access to the bigger one (actual transfer tops out at just over 2 gig) and not impacting other services over the wire.  I ran a Cat 7 cable which I knew would need to be right at 100M so purchased a 125M one - the run ended up being 110M.  I spent a lot of effort finding copper SFP+ that stated they would run at 5 and 2.5 in addition to 10, expecting to need the option and 2.5 or 5 be all that was reliable (and more than sufficient for me anyhow).  In the end it works fine at 10 gig, no errors.

AWS Advanced Networking exam notes

I recently completed the AWS Solution Architect - Associate exam, which was a medium degree of effort - I watched 40 hours or so of Linux Academy classes, tried some things in their lab and my own AWS environment (finally redeeming my $100 credit from attending last years re:invent), and passed the exam first time in about an hour flat.
There were a lot of database questions, which were my weakest area, so where I concentrated my study.

I debated going directly to the Professional SA cert, but why struggle with all those DB (and many other product) questions in greater depth, when I could instead do the Advanced Networking exam to get an equivalent (in the partner qualification sense) level certification.  In theory this is right in my wheelhouse as a long time CCIE etc.  In practice it's a harder exam than I was expecting - partially because it's more niche than the sysadmin ones so there's much less preparation material available, and what there is can be dated; AWS is an ever evolving beast where old limitations are eliminated and new features add frequently.  The official prep guide will quote a restriction or state a 3rd party product is needed to do something for which AWS introduced a native product for several months ago.  As usual it would help if AWS (and other vendors) had a clear message at the beginning of the test giving a date when it was last revised.

Davis' notes are helpful, and I'll reiterate his point that this is a tougher exam than you may expect. You really need to know Direct Connect well, which is hard as it's not something you can lab.
Interconnecting multiple VPCs, instantiating them with CloudFormation, lots and lots of questions to ensure you know transitive routing isn't a thing usually.  Route53 and load balancing and when to use which functions of each.  The example questions from the AWS site are as expected highly representative of the exam.  The Linux Academy and Udemy practice tests are pretty good but the questions are fairly wordy so it takes time to run through them - I ran through them many times with an open book (Google) on another screen until I was getting 90% odd reliably.  I can't speak to the SA Pro exam but the distractors on the Networking exam are much more plausible than the ones at associate level where a lot of questions could be answered by applying common sense.

Module 'CPUID' power on failed

When building my shiny new homelab I had enabled nested virtualization globally on all the hosts,

echo 'vhv.allow = "TRUE"' >> /etc/vmware/config

This seemed like a great idea and installing nested ESXi VMs went smoothly.  Then I wanted shared storage for those nested labs and had the bright idea of using NetApp OnTap virtual appliances or simulators to provide that - the OnTap appliance eval provides more storage than I need for 60 days, while the simulator is limited to 210GB but not in duration - in my lab I think the simulator is plenty and use in the lab is exactly what it is provided for.

Anyhow deploy the OVF, power on, error 'module CPUID power on failed'.  Deploy the OnTap virtual appliance, same result.

Some Googling, figure out it's to do with the vhv setting, turn that off on one host and reboot, sure enough both VMs now work fine.  Migrate a powered off ESXi VM to the modified host, that works fine too as the VMX file has the nested virtualization settings anyway.

Now for the pain - decide to remove vhv.allow from all my hosts, click maintenance mode, vmotions all fail with 'Failed to receive migration' - because the destinations have different CPU capabilities than the originating hosts.  I understand this fully for the VMs using nested virtualization but this is all VMs...such fun now going through every host powering down all the VMs in order to cold migrate them...all done now, but the lesson is DO NOT globally turn on vhv.allow anymore, it's better to turn on VT passthrough on individual VMs and not have other VMs that won't power on.  

Supermicro IPMI - Redux

The X10 IPMI support on my new servers is great - no more Java!
HTML5 virtual console for the win, that plus H5 vSphere and NSX (increasingly) means the days of needing Java or Flash are numbered.
  I did have one hiccup though; right when I thought my cluster was ready to go I tested remote access, as I wanted to secure it with an ACL in addition to a non-standard username & strong password.
  I'd used the IPMI virtual CD-ROM to install ESXi onto these so was surprised to find I couldn't access two of the four anymore.  After many reboots, trying both static and DHCP I concluded something had become wedged in the firmware, as the ports were showing as up on the switches, but though frames were being sent to them the inbound counters were all zeros.
  My theory is that in the bit of code that decides between the dedicated IPMI LAN interface and sharing LAN1 there was a bug.  This is the default 'failover' mode, where it uses the dedicated port if it is determined to be connected when power is applied, but once it's failed over to using LAN1 it never recovers without a hard reset - which is a huge pain in my new 2U boxes with shared power for the two nodes, the only way to power cycle just one is to physically pull it from the chassis, making my remote switched PDUs pointless.  Don't ever apply power to these until your switches are fully booted - which in my case is several minutes, so in the event of a power loss it would break.  I did try putting the LAN1 ports on the lights out VLAN without any change amongst many other experiments  that on my workbench at home I was happy to do for curiosities sake, where in a datacenter I'd just want the boxes back up ASAP.
  Anyhow, I built a DOS boot USB key (this is useful) and put Supermicro's IPMICFG tool as well as the latest IPMI firmware on it (already a release newer than when I started setting these servers up in December).  After upgrading from 3.77 to 3.78 and setting a static IP again I was back in business and once back in the web interface I changed their LAN mode from 'failover' to 'dedicated' which will hopefully prevent the issue from reoccurrence.

Postscript -
Managed to screw up another one by upgrading to current release, but then it never came back after rebooting.  Querying it from the Linux command line tools just gave errors. 
The AlUpdate tool was able to re-flash it - after which it worked, but be warned this needs a hard power cycle which would've been hard if the box had been off in a colo somewhere.
'AlUpdate -f REDFISH_X10_380.bin -kcs -r n'
(Update via KCS channel without preserving config)