Be careful upgrading NSX-T with vDS

Hit a good issue this week upgrading a customer from 3.2.1 to 3.2.2, but the issue also occurs if you're upgrading to 4.

There's a new vDS property which is set if you do a deployment of NSX-T onto vDS with a current version, com.vmware.nsx.vdsSecurity.enabled, the problem is that it isn't set during the upgrade process, until too late I think (I think the Managers add it after they are upgraded, but add you do the Edges then hosts, then Managers that doesn't help a whole lot).  So as each host is upgraded in turn and the Upgrade Manager takes each out of maintenance mode, the newly upgraded hosts are not in a state where they can receive VMs migrated by DRS when the next host is placed in maintenance mode.

Result, your VMs gradually get concentrated onto fewer and fewer hosts until the process fails.  Hopefully your remaining hosts aren't so overloaded that you/DRS can no longer migrate VMs back off them.

Workaround is to run on each host prior to upgrade:

net-dvs  -s com.vmware.nsx.vdsSecurity.enabled -p hostPropList <vDS name>

Confirm it's present with 

net-dvs -l | grep com.vmware.nsx.vdsSecurity.enabled

The host may require a reboot after setting it.

KB:

https://kb.vmware.com/s/article/90298

Recovering Dell OS10 switch from boot to ONIE

 Issue - received two Dell switches that booted into ONIE install O/S mode.

Interrupting the ONIE boot menu by pressing x, then exit, enabled it to boot into OS10 that time, confirming OS10 and most importantly the license was intact.  I then SCPed the license file off to somewhere safe - I didn't / don't have access to the customers Dell Digital Locker, so I had no way to recover it.

Reboot and let it boot into ONIE O/S install mode again, install OS10 - in my case this rebuild both OS10 partitions but did leave the license intact, but if I didn't back that up first I would have been distressed to see it repartition and format the flash...

Confirmed it now boots directly into OS10.

Sonic Networking

Sonic is an open source networking operating system, which runs on a variety of platforms.  Azure and other hyperscalers use it as an abstraction layer - they can manage multiple vendors network hardware the same - this may have been especially useful during the last couple of years.  Thanks to the supply chain issues of all the major vendors the plentiful availability of 32X100G used switches made them a candidate for a short notice project, and I figured I'd replace the ancient H3C 10G switch in my lab too.

They work well, as you would expect as the hardware is very similar to anything else based on the Broadcom Trident, be it white box or major vendor.  Sonic is not user friendly though, there are multiple release streams, with a variety of bugs in each.  User friendliness is not a high priority with all the big Sonic users out there heavily into automation.

You're going to want to read this huge thread through:
https://www.reddit.com/r/homelab/comments/n5opo2/initial_configuration_of_a_celestica_dx010_100ge/
Between that thread, the EdgeCore Sonic docs and the official docs, these are my notes...

Initial configuration:

Change the device to layer2 - default is layer3, which makes sense for a switch with no Spanning-Tree, be absolutely sure there are no loops before doing this (I created a nice 400G broadcast storm which was enough to kill the management plane access)

sudo sonic-cfggen --preset l2 -p -H -k Seastone-DX010 >/etc/sonic/config_db.json

Then reboot - in theory sudo config reload should work, I've had mixed results.  There are multiple warm reboot options for upgrading code without interrupting forwarding or rebuilding the config without reloading the O/S.  I will go back and experiment when I have time.

Hostname & management port (which is DHCP by default)

sudo config hostname switch01

sudo config interface ip add eth0 192.168.10.10/24 192.168.10.1

The default gateway doesn't really show up anywhere, don't be surprised, it does still work.

The management port also needs the following added into config_db.json, it works without it but a couple of the commands fail, specifically when you try to breakout ports dynamically - breaking out ports by editing config_db.json and restarting always works though.

"MGMT_PORT": {

        "eth0": {

            "admin_status": "up",

            "alias": "eth0"

        }

    },

Dynamic port breakout:

sudo config interface breakout '4x25G[10G]' Ethernet0

Turns 100G port 0 into 4 X 25G ports which you can then configure to 10G (all four of them) to make a 40-4x10 breakout work.  Extra arguments that should work, -y -v etc. seem to break it as does not having the extra MGMT_PORT blob above.

MCLAG

Because I was doing layer2 MCLAG was a requirement - the daemon for which, ICCPD, is not included in the builds by default, so then I needed a build server.  This is well documented on the Sonic site, though one step is installing Docker, which they have you do with Snap - which then doesn't work.  Whereas a proper Docker install works fine.  (Look for instructions to add the docker.com repository and gpg key to sources.list).  Code upgrades fail to persist the configuration in my experience, so archive config_db.json prior and plan to push it back on afterwards.

The Edge-Core MCLAG page is mostly right, 

https://support.edge-core.com/hc/en-us/articles/900002380706--Edgecore-SONiC-MC-LAG 

Note you can't designate which VLAN interfaces will be the unique-ip while they already have IPs on them, so designate the interface, then add the IP.

The JSON block ends up looking like this if you've done it right:

   "MCLAG_DOMAIN": {

        "1": {

            "peer_ip": "10.210.1.2",

            "peer_link": "PortChannel01",

            "source_ip": "10.210.1.1"

        }

    },

    "MCLAG_INTERFACE": {

        "1|PortChannel02": {

            "if_type": "PortChannel"

        }

    },

    "MCLAG_UNIQUE_IP": {

        "Vlan1000": {

            "unique_ip": "enable"

        }

    },

Here our IP on Vlan1000 is 10.210.1.1, with our partner .2, PortChannel01 is our peer link and our only MCLAG enabled channel PortChannel02.

Ensure IPCCD and TEAMD are running / set to start at boot.  Easiest way to to ensure those services are set top enabled in config_db.json, but may need to unmask, 'systemctl unmask iccpd' and 'systemctl start iccld' first.

There are references to various 'show mclag' commands which don't seem to exist, but 

'mclagdctl dump state'  is the key thing to show whether the partner switches can see one another and the daemons are talking.

Some of the other mclagdctl commands are implemented- though not all of them...but useful stuff does go to syslog.

VLAN assignments

So it took me hours to build out a config with 35 VLANs tagged on all ports, as far as I can see there's no efficient way to do it interactively- so if you need to do it in bulk script it to a file then dump the file into config_db.

JunOS Command cheat sheet

I'm not going to try and do an IOS - JunOS conversion guide, but just save my list of useful commands after spending a couple of months installing a ton of Juniper EX switches.  I couldn't if I tried anyhow as most of the switches I touch these days are Dell OS10, Mellanox Onyx, or NX-OS...Not a lot of IOS there.  

Every vendor has a way to configure multiple switches to be able to support MLAGs - LACP channels across two or more switches, it varies as to whether 'stacking' - where the switches then have a single conjoined control plain is required - which leads to issues when upgrade time comes.  Juniper does require Virtual Chassis for this, Dell VLT / Mellanox / Cisco VPC are slightly more distant so you still have that control plane separation which is nice.  In some environments I've managed to keep a pair of Juniper EX separate because ESXi / Cohesity / Pure could all support redundancy without LACP, which is preferable IMHO.

Juniper virtual chassis is straightforward, turn LLDP on and connect switches together (over 40 or 100G only), if they're the same type they try to do it for you.  If they're different types of switch you may need to manually configure mixed-mode which requires a reboot.  Some switches don't have the 40/100G ports set as vc-port out of the box, that's simply request virtual-chassis vc-port set ...

Upgrades may not be quite to straightforward- my preference is to update individual switches to a sensible release when they're fresh out of the box then leave well alone.  Haven't yet been trying any features new enough for this to cause an issue.  On the bench a USB key is the easiest way to get an image on, once configured putting images on a Linux VM running Ngnix is easiest - can then install the image direct from the http URL rather than fighting to copy into into /var/tmp first. Frequently this fails for no good reason. If virtual chassis pairs are not in production you can install the image on all of them then power cycle so they all come up with the new image, I've failed to do it more gracefully as the virtual-chassis doesn't re-establish once they're on different versions so then you can't reboot the others - unless you have OOB access of course.

My biggest culture shock was seeing the same physical ports listed in the configuration repeatedly as ge-0/0/1 / xe-0/0/1 / et-0/0/1 and having to ensure you put configuration under the right one according to what speed you're going to utilize the interface at.  I installed several pairs of EX4650 in a short space of time with each install smoother than the last.

Plan carefully as you can only set port speed per group of 4 ports, which can be annoying if after your neat  cabling job something doesn't come up at the speed you expect.  (or someone else patches a couple of 10g only devices to the middle of a bunch of ports set to 25 so someone has to drive to the DC and move them)

set chassis fpc 0 pic 0 port 20 speed 25g

One thing that bit me is the supported optics/DAC list is not quite the same between images/switch types, so having tested parts between one pair of switches on the bench, I was losing my mind when those same parts didn't work in the DC...(I've generally had issues with 25g that I don't with 10/40/100, lots of picky devices, worst offender being Intel 25g NICs that want Intel cables - which I couldn't find available to purchase)

'show configuration | display set'  - view the configuration as set commands rather than XML

'show | compare'    -  see what pending changes you have made in edit mode

'rollback 0'   - throw away those changes if you don't like what you see

'show interfaces diagnostics optics'  - valuable when doing the L1 hard part, laboriously chasing down why all your fiber links are not up.  Even a cheap and cheerful light meter from Techtools or Amazon is invaluable here too.

'show chassis pic pic-slot 0 fpc-slot 0' and 'show chassis hardware' are useful here to for listing out the optics/DACs present in the system.

Run has a similar function to do in Cisco land, so you can execute an operational mode command from within edit (config) mode. 

'monitor start messages' = term mon

To put an IP address on a VLAN, you create an IRB interface:

set interfaces irb unit 123 family inet address 192.168.1.1/24

set vlans newvlan vlan-id 123

set vlans newvlan l3-interface irb.123

To create an access port:

set interfaces ge-0/0/5 unit 0 family ethernet-switching vlan-member newvlan

To create a trunk port:

set interfaces ge-0/0/6 unit 0 family ethernet-switching interface-mode trunk

set interfaces ge-0/0/6 unit 0 family ethernet-switching vlan-member oldvlan

set interfaces ge-0/0/6 unit 0 family ethernet-switching vlan-member newvlan

To add a default route:

set routing-options static route 0.0.0.0/0 next-hop 172.16.1.1

LAG / MLAG / Port channels:

set chassis aggregated-devices ethernet device-count 1 (Or however many LAGs you need total)

set interfaces ae0 aggregated-ether-options lacp active

set interfaces ae0 unit 0 family ethernet-switching interface-mode trunk

set interfaces ae0 unit 0 family ethernet-switching vlan members oldvlan

set interfaces ae0 unit 0 family ethernet-switching vlan members newvlan

set interfaces ge-0/0/0 ether-options 802.3ad ae0

set interfaces ge-0/0/1 ether-options 802.3ad ae0

Note that the access class switches and datacenter class (4500+) have different defaults.  Where the access switches have RSTP on by default DC switches do not, I 'set protocols rstp interface xe-0/0/47'  just on ports that I know will be up/down links to other switches.  

Recovering an old SRX300 with an unknown password 

- that was also configured to ignore the config reset button.  

Find an image and put it on a small FAT32 USB stick (a 1G worked for me while an 8 did not (got 'cannot open package (error 22)'). Insert the stick and boot, ignore the first prompt asking if you want to interrupt the boot process, but press space on the second.  Then at (notice triple slash)

loader> install file:///junos-srxsme-10.4R3.11.tgz

Takes a long time, as it reformats the internal USB file system, but result is a new blank system, as if the reset config button still worked.  Also needed to 'delete system phone-home' and 'delete system autoinstallation' before web access works.  To permit SSH from untrusted:

set security zones security-zone untrust interfaces ge-0/0/0.0 host-inbound-traffic system-services ssh

Supermicro IPMI yet again

So I broke one of my ESXi hosts by installing 7.0U2 as a patch baseline instead of an upgrade baseline in Lifecycle Manager (formerly Update Manager).  Failed to boot getting stuck at 'loading crypto...'

Easy fix supposedly, boot from CD and upgrade install over the top of the existing install, boot right back into the cluster.

Forums abounded with other people hitting the same thing and using iDRAC, iLO etc to mount the image and recover, I tried to do the same with Supermicro IMPI.  That's how I installed 6.7 on these in first place so I knew I had the capability to mount an ISO from an SMB share.  However install was at home where I was mounting the images off a Synology.  I dimly remembered having to mess with Synology but couldn't remember just how.

After wasting a long time trying to mount the ISO of a Windows box I gave up and installed a fresh Ubuntu VM to use, figuring correctly that Samba logging would help me figure it out.  Supermicro's SMB client not only speaks only SMB 1, 'server min protocol = NT1' but also doesn't support any decent authentication methods.  So after also adding 'ntlm auth = yes' the mount worked and I could recover.  The Samba VM got 150 random SMB hits from the Internet during its brief lifespan too, though all either zero length log files or ones filled with auth failures.  (My IPMI ports are out on the internet but with ACLs to limit access to just some static IPs I have access to, I'm not completely crazy)

[global]

server min protocol = NT1

ntlm auth = yes

[shared]

path = /home/simon/shared

valid users = simon

read only = no

To get IPMI settings of local system from ESXi:

localcli hardware ipmi bmc get

esxcli hardware ipmi bmc get

What to do when VCSA 7 runs out of space

 In my case ‘var/log’ was full, it being one of the smaller 10GB virtual disks.

The beauty of vCSA having 16 disks all in separate files is the ease with which you can grow one.

Get onto the console via virtual console or SSH, run a shell, then you can 'df -h' to confirm the full mount point, then use 'lsblk' to trace that back from it’s ‘Dev/wrapper’ mountpoint to an actual device like ‘Dev/sde’.  E being the 5th letter of the alphabet correlates with it being a 10GB device here and also my disk 5 in the VM settings.

Now take a backup.  Of course you're already doing nightly backups but then check that they're actually working, mine hadn't been for six weeks without my noticing due to an NFS permissions issue.  

Gracefully shutdown vCSA taking note of which host it’s on.  Connect to that host and edit settings for the vCSA, edit that virtual disk to increase its size, feel free to expand any other disks while you're there, it's not like most virtual storage isn't thin provisioned anyhow.  I took the opportunity to increase my RAM and CPU count too as I’m not resource constrained and I figured 4 vCPUs and 24GB would make my vCenter snappier.  Power back on and get a coffee while it boots/starts services.  

If you get 'editing host resources is disabled because this host is managed by vCenter' you can workaround by SSHing to the host and restarting vpxa and hostd - this will kick you out of the GUI, but then once you have re-authenticated you can make changes.

Console or SSH in again, open a shell and run the ‘/usr/lib/applmgmt/support/scripts/autogrow.sh‘ script, it should find your extra space and grow both the partition and the file system.

Done.  

vCSA 7.0 upgrade ate my STS_INTERNAL_SSL_CERT store

So a week after upgrading to vSphere 7 I can't login to vCenter anymore, I get

"HTTP Status 400 - An error occurred while sending an authentication request to the PSC Single Sign-On server"

Google confirms this is a thing that can occur post an upgrade and there are blog posts and KB articles on fixing it.
Including this one: https://kb.vmware.com/s/article/52541
which describes how to put the cert back into the store for the machine store when it has become corrupt.  I had the additional wrinkle in that my whole store was gone, when I tried any of the commands except for backing up the certs I got,

"Error: Failed to open the store.
vecs-cli failed. Error 4312: Possible errors:
LDAP error: Unknown (extension) error
Win Error: Operation failed with error ERROR_OBJECT_NOT_FOUND (4312)"

I'm pleased to say creating a new store with:

/usr/lib/vmware-vmafd/bin/vecs-cli store create --name STS_INTERNAL_SSL_CERT

was all it took, then following the procedure in the KB, restarting all services and I'm back.  I was also finally motivated to make use of the vSCA backup process, which is super easy and supports every protocol under the sun.  My 4 node cluster creates so few DB entries that a backup is only a gigabyte, so having it backup every night to my Synology at home and retain 7 of them is something long overdue.  I'm using NFS, the only wrinkle in setting it up being remembering where in Synology to add another host to the permitted list.