Module 'CPUID' power on failed

When building my shiny new homelab I had enabled nested virtualization globally on all the hosts,

echo 'vhv.allow = "TRUE"' >> /etc/vmware/config

This seemed like a great idea and installing nested ESXi VMs went smoothly.  Then I wanted shared storage for those nested labs and had the bright idea of using NetApp OnTap virtual appliances or simulators to provide that - the OnTap appliance eval provides more storage than I need for 60 days, while the simulator is limited to 210GB but not in duration - in my lab I think the simulator is plenty and use in the lab is exactly what it is provided for.

Anyhow deploy the OVF, power on, error 'module CPUID power on failed'.  Deploy the OnTap virtual appliance, same result.

Some Googling, figure out it's to do with the vhv setting, turn that off on one host and reboot, sure enough both VMs now work fine.  Migrate a powered off ESXi VM to the modified host, that works fine too as the VMX file has the nested virtualization settings anyway.

Now for the pain - decide to remove vhv.allow from all my hosts, click maintenance mode, vmotions all fail with 'Failed to receive migration' - because the destinations have different CPU capabilities than the originating hosts.  I understand this fully for the VMs using nested virtualization but this is all VMs...such fun now going through every host powering down all the VMs in order to cold migrate them...all done now, but the lesson is DO NOT globally turn on vhv.allow anymore, it's better to turn on VT passthrough on individual VMs and not have other VMs that won't power on.  

Supermicro IPMI - Redux

The X10 IPMI support on my new servers is great - no more Java!
HTML5 virtual console for the win, that plus H5 vSphere and NSX (increasingly) means the days of needing Java or Flash are numbered.
  I did have one hiccup though; right when I thought my cluster was ready to go I tested remote access, as I wanted to secure it with an ACL in addition to a non-standard username & strong password.
  I'd used the IPMI virtual CD-ROM to install ESXi onto these so was surprised to find I couldn't access two of the four anymore.  After many reboots, trying both static and DHCP I concluded something had become wedged in the firmware, as the ports were showing as up on the switches, but though frames were being sent to them the inbound counters were all zeros.
  My theory is that in the bit of code that decides between the dedicated IPMI LAN interface and sharing LAN1 there was a bug.  This is the default 'failover' mode, where it uses the dedicated port if it is determined to be connected when power is applied, but once it's failed over to using LAN1 it never recovers without a hard reset - which is a huge pain in my new 2U boxes with shared power for the two nodes, the only way to power cycle just one is to physically pull it from the chassis, making my remote switched PDUs pointless.  Don't ever apply power to these until your switches are fully booted - which in my case is several minutes, so in the event of a power loss it would break.  I did try putting the LAN1 ports on the lights out VLAN without any change amongst many other experiments  that on my workbench at home I was happy to do for curiosities sake, where in a datacenter I'd just want the boxes back up ASAP.
  Anyhow, I built a DOS boot USB key (this is useful) and put Supermicro's IPMICFG tool as well as the latest IPMI firmware on it (already a release newer than when I started setting these servers up in December).  After upgrading from 3.77 to 3.78 and setting a static IP again I was back in business and once back in the web interface I changed their LAN mode from 'failover' to 'dedicated' which will hopefully prevent the issue from reoccurrence.

Postscript -
Managed to screw up another one by upgrading to current release, but then it never came back after rebooting.  Querying it from the Linux command line tools just gave errors. 
The AlUpdate tool was able to re-flash it - after which it worked, but be warned this needs a hard power cycle which would've been hard if the box had been off in a colo somewhere.
'AlUpdate -f REDFISH_X10_380.bin -kcs -r n'
(Update via KCS channel without preserving config)

Homelab refresh

Finally replacing my homelab, for two reasons, consisting of three hosts from 2010 it was ancient, and additionally I lost a drive and my vSAN blew up.

vSphere has finally pulled out the x86 instruction emulation code that allowed really old CPUs to work so while 6.7U1 ran on my 5630L CPUs I couldn't do a clean install (would have had to install 6.5 and upgrade) and nested virtualization was becoming limited by the same thing, which is kind of my killer app for a homelab, on the hosts themselves upgrading is OK, but not being able to instantly have a >6.5 nested host was a pain.

I didn't understand vSAN :)  I'd been running it a long time on unsupported everything (controllers, drives, NICs, you name it) and my early mistakes couldn't be easily fixed as if I tried to reconfigure anything on the fly I got error messages rather than actions.  With money I could have fixed it - by replacing the controllers and buying enough disks for an additional disk group and migrating, or doing something ugly like moving data onto a USB drive or 2 bay NAS...didn't come to that anyhow as I lost so much data there was little point in saving any.

The critical thing I hadn't understood was that erasure coding needs a minimum of four hosts, more if you want to do maintenance, so turning it on in my three host cluster was not smart.  One of my SSDs failed and about half my VMs went with it as they must have had blocks on that disk group that couldn't be pulled from elsewhere.  I daresay I could have recovered many of them but in the lab nothing was critical enough to bother, my greatest pang is for my trusty Windows 7 admin VM...I have been way too cowboy in my lab, which was fine a decade ago, when it was a fraction the size, local to me, and using NFS storage.  These days when I blow it up with a pre-release build that I then find can't be upgraded, or by turning on features for fun before I understand the consequences it's a huge effort to recover.  Nested labs make a lot of sense, where I used to almost enjoy (OK enjoy is overstating it, but I did derive some masochistic pleasure from it and revel in being an expert), the installation pains of the VMware suite, many of those have been reduced (finally) to the point there's no learning in that stage of things.

As with the old cluster I got somewhat carried away building a new one, I similarly received some cast off gear for free and supplemented from eBay and stripping my old systems (only reusing the SSDs). I wanted to grow to four nodes without taking up more space, so when I was gifted a 2015 vintage Supermicro Twin I was happy to purchase a second in order to end up with four identical hosts in 4U of space (replacing 3 X 2U boxes).  This particular model has a SAS controller onboard so I can live with the 2 PCIe slots, reusing my Intel Optane 900p NVMe drives* in one for vSAN cache layer, and installing new Intel X710 10 gig NICs in the other.  (If I'd had a third slot I would've reused my X520's in order to have NICs to pass through to VMs when playing with NSX-T etc.)
The build took a long time as I wanted the firmware on the BIOS/IPMI/SAS controller (now in HBA mode) NICs to all be current - all of which takes a lot of power cycles and messing about, I do see why people purchase vSAN ReadyNodes.  These boxes, 2028TP-DC0R, support current Xeons, I'm using E5-2630L v3, which are not very recent, but cost and power effective and importantly Haswell series so good for some time to come.

The X710's were the biggest time suck, I had two fail on me, not sure if I was unlucky with static, or upgrading their firmware bricked them after a power loss or something.  I would've put the X520s in and been done with them but I only had three and I really wanted the four nodes identical.
I also had second thoughts on RAM, having built out with 128GB per node, deciding longevity would be served better with 192 per.  VMware's stack loves RAM and once I have a pretty complete SDDC running plus a few third party integrations I'd be swapping.

I also turned back on Transparent Page Sharing, enabled nested virtualization, and though I don't thing any of my operating systems support it right now TRIM in vSAN.
I'm now a happy camper, building out a nested lab, the below shows resources consumed by my management layer:

* The Optane are awesomely fast, and have ridiculous endurance too for consumer drives, the 280GB have 336GB inside which supposedly isn't used for traditional over-provisioning, but they must use some of it to help deliver that longevity.  I figure that having the cache tier off the main controller saves the queue in that for destaging to my relatively slow consumer grade SSDs too.  (I had some Enterprise SSDs at one point but they also gave me my only SSD failures, out off warranty of course, where the Samsung Pro consumer drives have been issue free)



Bill of materials:

2 X Supermicro 2028TP-DC0R Twin systems (four nodes) (3008 SAS controller onboard)
8 X Intel Xeon E5-2650Lv3 1.8Ghz 12 core
4 X Intel Optane 900P 280GB PCIe (cache drives, not on vSAN HCL)
4 X Supermicro 64GB SATA-DOM
4 X Intel X710DA Dual 10g SFP+ dual port
4 X Intel 1.6TB S3610 SAS SSD
4 X Samsung 850Pro SATA drives (not on vSAN HCL)
64 X 16GB ECC DDR4 DIMM
2 X HPE 5900 switches (48 gigabit, 4 SFP+, 2XQSFP)

Total, about 20K, but over time and much eBay so very approximate.

P.S. Were I doing this again I'd get the 2028TP-DC0TR, which is exactly the same but with Intel X540 ten gig NICs on board, cost difference now is negligible.  

Palo Alto firewall setup

I've had to do this many times recently as I build/destroy labs containing the Palo Alto VM series but not always with the integration with NSX which does all the work for me.  Keeping this to hand saves me time as I can copy and paste into the console in order to get to a point in setup when I can continue in the GUI / hook up Panorama.

Initial login admin admin

initial IP on appliances 192.168.1.1

configure

set deviceconfig system type static

set deviceconfig system ip-address 192.168.1.9 netmask 255.255.255.0 default-gateway 192.168.1.1 dns-setting servers primary 10.23.1.10 secondary 10.23.1.11

commit

save config

Change MAC address to assigned by hypervisor - additional NICs didn't come up until I did this and rebooted.

Monitor filtering
(src eq 192.168.1.11)
(action eq deny)
(zone.src eq DMZ)
(src eq 192.168.1.11) and (dst eq 172.16.1.11)

show counter global filter value non-zero delta yes | match arpshow config diff
https://live.paloaltonetworks.com/t5/Management-Articles/Packet-Capture-Debug-Flow-basic-and-Counter-Commands/ta-p/66224
show counter global filter severity drop packet-filter yes delta yes

Palo Alto RADIUS dictionary for RSA Authentication Manager

I Googled for a pre-built RADIUS dictionary to add to RSA Auth Manager in vain, so if I can save someone else twenty minutes:
This is a mash up of the results of following, https://community.rsa.com/docs/DOC-46893, with the contents of https://live.paloaltonetworks.com/t5/Tech-Note-Articles/RADIUS-Dictionary/ta-p/53745

You need to SSH to the Authentication Manager server, cd to /opt/rsa/am/radius  
(You can also do it from file access in the Operations Console, and you need to do it on replicas too)
Then edit vendor.ini to add:

vendor-product       = Palo Alto Networks

dictionary           = paloalto

ignore-ports         = no

port-number-usage    = per-port-type

help-id              = 2000

Then to dictiona.dcm add:

@paloalto.dct

And create paloalto.dct with contents:

@radius.dct

MACRO PaloAlto(t,s) 26[vid=25461 type1=%t% len1=+2 data=%s%]

ATTRIBUTE       PaloAlto-Admin-Role           paloalto(1,string) r

# PaloAlto-Admin-Role is the name of the role for the user

# it can be the name of a custom Admin role profile configured on the

# PAN device or one of the following predefined roles

# superuser : Superuser

# superreader : Superuser (read-only)

# deviceadmin : Device administrator

# devicereader : Device administrator (read-only)

# vsysadmin : Virtual system administrator

# vsysreader : Virtual system administrator (read-only)

ATTRIBUTE       PaloAlto-Admin-Access-Domain  paloalto(2,string) r

# PaloAlto-Admin-Access-Domain is the name of the access domain object defined

# on the PAN device

ATTRIBUTE       PaloAlto-Panorama-Admin-Role  paloalto(3,string) r

# PaloAlto-Panorama-Admin-Role is the name of the role for the user

# it can be the name of a custom Admin role profile configured on the

# Panorama server or one of the following predefined roles

# superuser : Superuser

# superreader : Superuser (read-only)

# panorama-admin : Panorama administrator

ATTRIBUTE       PaloAlto-Panorama-Admin-Access-Domain   paloalto(4,string) r

# PaloAlto-Panorama-Admin-Access-Domain is the name of the access domain

# object defined on the Panorama server

ATTRIBUTE       PaloAlto-User-Group      paloalto(5,string) r

# PaloAlto-User-Group is the name of the group of users

ATTRIBUTE       PaloAlto-User-Domain        paloalto(6,string) r

# PaloAlto-User-Domain is the name of the user domain

ATTRIBUTE       PaloAlto-Client-Source-IP   paloalto(7,string) r

# PaloAlto-Client-Source-IP is the source IP address of the computer

# on which GlobalProtect client is used to log in

ATTRIBUTE       PaloAlto-Client-OS   paloalto(8,string) r

# PaloAlto-Client-OS is the operating system (OS) of the computer

# on which GlobalProtect client is used to log in

ATTRIBUTE       PaloAlto-Client-Hostname   paloalto(9,string) r

# PaloAlto-Client-Hostname is the hostname of the computer

# on which the user logs in

ATTRIBUTE       PaloAlto-GlobalProtect-Client-Version  paloalto(10,string) r

# PaloAlto-GlobalProtect-Client-Version is the version of GlobalProtect

# client which is used to log in

Then once you've restarted the RADIUS server, Palo Alto will appear in the list of RADIUS server client types you can select.  Then in the RADIUS profiles you will have a bunch of Palo Alto return attributes.
To make groups work you still need an LDAP somewhere for the firewall to pull the groups from, then you can have a return code like 'cn=admin group,ou=org groups,ou=groups,dc=orgname,dc=com'

Lessons from running a Zimbra mail server

This should maybe be titled 'Lessons from running a SMTP mail server' as it's not so much Zimbra that's been the issue, but fighting spammers.  I'm not doing anything fancy, I only have a single VM running the free edition of Zimbra with not many mailboxes, the trouble is that any presence on TCP/25 on the Internet will have the same issues - getting you mails delivered, while not getting overwhelmed by spam being received.
With only a few mailboxes the former should be easy, except that out of the box my setup wasn't secure enough to prevent tons of spam email being relayed - no I wasn't an open relay, but I hadn't rejected unlisted senders, which means someone sent a bunch of stuff 'from' my domain, which I wanted Zimbra to relay (due to system emails from vCenter / NetApp etc.)  I turned that off:

zmprov mcf zimbraMtaSmtpdRejectUnlistedRecipient yes

zmprov mcf zimbraMtaSmtpdRejectUnlistedSender yes

zmmtactl restart

zmconfigdctl restart

Receiving email should be a breeze too, except once I'd tightened the spam scoring to reduce spam to a manageable trickle, false positives crept in too.

In /opt/zimbra/data/spamassassin/ there are several configuration files, I found it useful to whitelist several domains from which I regularly receive mail, adding them to localrules.cf, though I found that they still got rejected sometimes.  I fixed that by changing the score for whitelisted domains in ../rules/50_scores.cf though these aren't supposed to be modified so be warned that an upgrade may well overwrite my changes.

After making changes you need to restart Zimbra, or at least 'zmamavisdctl restart' (as Zimbra user)

Other things that have been useful to me have been around collecting data from the command line, using zmprov to grab stuff out of the DB that I can then look at /pipe to a file:

zmprov -l gaa   - lists all accounts 

zmprov gadl     - lists all distribution lists

zmprov gdl mailinglist@example.com | grep zimbraMailForwardingAddress: | awk {'print $2'}  - lists members of a distribution list

for i in `zmprov -l gaa` ; do zmprov ga $i zimbraMailAlias ; done  - lists aliases

HP Comware and Procurve switches

Oh the joy of finding a completely new command line syntax...

I've used HP switches a good deal in the past, all manner of Procurve models, mostly modular ones as they were an inexpensive rack solution when dual power supplies were a requirement.  These days top of rack switching is so normal every vendor makes 1U datacenter switches with dual power, and when someone asked me for a recommendation I surfed a little bit and suggested HP 5900 as a cost effective option for 48 gigabit ports and 12 ten gigabit ones.  Little did I know they'd ask for my help configuring them - they are certainly very powerful, but it took me long enough to figure out how to enable SSH and basic layer 2 stuff and more features are being added to the Comware OS every few months it seems, including hardware VXLAN VTEP by the looks of it, though when/whether they will get that certified/supported is anyones guess.

On a Procurve I'd enable SSH with:
ip ssh
ip ssh filetransfer
no telnet-server

With the only caveat being that on the old switches I have in my home lab creating a self signed cert / RSA keys on the command line doesn't work, though it does in the GUI.

Back to the Comware based switch:
It expects you to have an enterprise RADIUS system to authenticate against and it took a lot of figuring out to create a self contained config.

system-view
public key local create rsa
ssh server enable
sftp server enable
ssh user simon service-type all authentication-type password
user-interface vty 0 15
authentication-mode scheme
protocol inbound ssh

There's a free ebook available from HP that may be helpful too,
https://h30590.www3.hp.com/product/HP+Networking+and+Cisco+CLI+Reference+Guide+-+Version+2-PDF-8407
now updated for version 7 of Comware.

The usual necessities:

dns domain sjhwilkes.local
dns server 10.206.3.5
dns server 10.206.3.17
ntp-service enable
ntp-service unicast-server 10.206.3.1

It took me ages to stop typing show and use display instead, and likewise no becomes undo in order to remove lines from the config.

On my (ESXi) host facing ports I have:

port link-mode bridge
 port link-type trunk
 port trunk permit vlan 1 10 to 11 15 101 to 102 150 254

Which hardcodes them to be dot1q trunks with a selection of VLANs permitted and VLAN 1 native (though I don't actually use it for anything, force of habit as it was a security recommendation many moons (years) ago)

I'm not doing LACP to my hosts, the amount of messing I do with difference versions of NSX and vSphere it's easier to stick with failover/manual.
I experimented with LACP to my old NetApp 2020, which looked like:

interface Bridge-Aggregation10
 description laxnas01
 port link-type trunk
 port trunk permit vlan 1 15
 link-aggregation mode dynamic
 lacp edge-port

Then on the constituent ports:
 port link-mode bridge
 description laxnas01-e0a
 port link-type trunk
 port trunk permit vlan 1 15
 port link-aggregation group 10

I'd still like to figure out if I can put the management interface into it's own VRF and have some sort of back door into the rack - difficult without springing for another circuit of some kind though.

To silence log messages about non-H3C transceivers (which work anyway):

transceiver phony-alarm-disable