Supermicro IPMI

Hope you have more luck with Supermicro IPMI interfaces than I, they're not old school iLO or DRAC like in reliability for me, and I guess there aren't many developers working on keeping the few models updated.  Not quite Enterprise grade is what I'm saying I guess.  Like so many products if they could just increment it so support current Java it would be less painful.

Remote power management is vital though, as is remote console and media. I really, really don't want to drive (or fly) to a colo just to reboot a purple screen or do an O/S reinstall.

The first of my recent issues was unusual as anything I touch normally is running vSphere ESXi rather than Linux, in this case I was trying to recover IPMI access on a box running Ubuntu on the bare metal.
The card was in some sort of funky state, where is didn't respond on 443, but on 22 I could see the SSH banner (with telnet), but not login with ADMIN ADMIN with SSH.

Some Googling later I downloaded and installed the Supermicro IPMIcfg utility,
ipmicfg_1.23.0_general_20151106.zip, stuck the 64 bit binaries on the affected host, ran:

modprobe ipmi_si
modprobe ipmi_msghandler
modprobe ipmi_devintf

These load the kernel modules to enable the utility to talk to the hardware.  Then you're ready to run useful commands:
ipmicfg-linux.x86_64 -m   # Which lists the IPMI IP and MAC
ipmicfg-linux.x86_64 -user list   # Which gives a list of users and privilege levels
ipmicfg-linux.x86_64 -r   # Which performs a reboot of the card

It allowed me to confirm the IP address and username was correct at least, then do a reset of the card, after which access was restored.

The second issue was with my lab, somehow I've created users that don't work and deleted ADMIN.  I can login to the web interface as me and it shows 'simon / administrator' but all the options say 'You have no permission to view this section.' with the exception of a few read only ones.
I then managed to purple screen (I presume as I couldn't get to the remote KVM) one of my hosts so really wanted to issue a reset.
IPMIcfg is no use as I'm neither running Linux nor have a running O/S, so I had a long play with another tool, SMCIPMITool.  Also downloaded as a static binary only this time it is used against a remote target card.

SMCIPMITool 10.10.10.1 ADMIN "ADMIN" ipmi sensor   # for example reads the sensor status

I found I couldn't do anything useful with the command in this form, but when I tried the shell option,

SMCIPMITool 10.10.10.1 ADMIN "ADMIN" shell

I eventually found that though 'power cycle' 'power reset' etc. returned an error (possibly permissions due to my weird account) 'power off' and 'power on' did work and I got my server back without venturing into the LA traffic.

Upon even further experimentation I also discovered that I can create new user accounts, so one quick,
user add 4 testuser testpassword 4
later, and I have an account which works properly in the GUI**.

Though getting the virtual KVM to work in modern Java on a model O/S was a huge pain - the applet is signed with a key that's both short and MD5, so even after whitelisting the IPs of my IPMIs I have to edit java.security to reenable MD5 certs and permit 256 bit certs.

RSA SecureID Authentication Manager 8.1

RSA SecureID Authentication Manager (AM), is one of those bits of software it seems I have to install once every five years or so, during which time I've lost all memory of how I did it, and anyway the product has probably evolved enough any knowledge would be out of date.
This time round OVA packaging of the appliance itself has simplified that bit of things, but the addition of a web tier for soft token distribution and user self-service added some complexity.

I don't think AM needs a lot of notes, but the complexity of licensing it and provisioning the tokens is exponentially greater than last time I did it, I'm guessing as a result of some well publicized breaches that have occurred.  Follow the docs and though tedious you end up with the required files and the application to decrypt the token seeds.

The tokens came on a CD - finding a way to read it took me a while, Celeron Linux mini system from the back of the garage pressed into service for that.  Then you use the codes printed on the CD to create a decryption file and password on the RSA site, then use the application to turn those plus the encrypted token seeds into something you can import into the app.

The AM web GUI is horribly unreliable for me, and I've tried Chrome, Mozilla, and IE, with IE being the least bad - though I still need to frequently mouse over a different tab in order to get menus to show up in the tab that I need - it took me a long time to realize this as first I thought it was a permissions issue, so I wasted time creating various different classes of administrator, logging in as them and finding still no luck on the menus.

The web tier install was complicated by RSA/EMC only supporting RHEL, which of course I don't have.  CentOS 6.5 seems to work fine but you have to change /etc/redhat-release to
'Red Hat Enterprise Linux Server release 6.5 (Santiago)'
so the RSA installer doesn't complain and exit.

I had various permissions issues, I gave up and chown rsauser / chmod 777 all the install files and their directory - I deleted them all after the install anyhow so why mess about.

Usual Linux best practices apply, NTP is vital due to the tokens etc. open-vm-tools, and the only other thing that caught me out despite my looking for it was that iptables blocked 443 out of the box, adding a rule:
'-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT'
 to /etc/sysconfig/iptables solved that.

To recover the Super Admin account, run: 

./rsautil restore-admin –u [tempadmin_name] –p [password]

from /opt/rsa/am/utils as the console user elevated to root.

VCDX

My VCDX Journey - Fourth time's a charm!

I'm proud to be able to say I'm now VCDX #197. It seems somewhat of a tradition to write one of these, so despite the count of VCDX's reaching two hundred I'll have a crack. I started writing a blog entry back in 2012 when I originally embarked on this, but never finished it or hit publish.  Through the power and elephantine memory of Google here it is:

What led me here?
I recently received my VCAP-DCA result, appropriately enough, right after I presented a session at VMworld, and unexpectedly passed - I hadn't even answered all the questions before running out of time and knew I'd not scored on a couple.  Little did I know that was normal.  I'd already passed DCD, having taken the beta November 2010, so felt with both, plus a lot of time invested in pre and post sales vSphere consulting, that going for it and trying to make the final VCDX 4 defense in Frankfurt is worthwhile.  These days I'm no longer a consultant, instead part of the VMware alliance team at F5 Networks, which is great, but means it's going to take a long time to develop the depth of both architectural and real world understanding that I think are required of the VCDX - so waiting for version 5 doesn't appeal.  VMware are also going great guns enhancing the products with every release, so there's more to cover with each too, it just gets harder.
Reminds me a lot of the CCIE program back in the day - I took it twice, in 2001 and 2002, right when it contained the kitchen sink of networking:  DLSW, IPX, Token-Ring, Appletalk, VoIP, ATM, in addition to all the IP protocols.  VCDX is very different and to some degree you can self-select your specialties in choosing what to include in your submitted design, but still has that tendancy to get broader and broader, whilst remaining just as deep.  Of course you'll get questioned in the defense on whatever you didn't include too.

Back to present:
I didn't get to Frankfurt, I was too slow at getting it all together, but did submit and defend in Burlington May 2012. I now know that I was very close to passing but didn't, so I defended again in Barcelona in October 2012 and again missed, I suspect by further than the first time.
Roll forward a couple of years, my wife and I had a daughter the following year so VCDX was pushed off the stack for a while. Joining the GCoE pre-sales team at VMware in February of '14 things changed, the team was half VCDX's already and our manager made it clear he'd like the whole team to attain it.
I'd been working on a design already as my old one was vSphere 4 so no longer eligible, and I wanted to incorporate NSX. I was probably two-thirds done with the design and had only outlined the operations and implementation guide. The suggestion was made to do a group submission, with us each playing to our strengths in the sections we write and lots of review, and the pure division of labor, targeting the PEX 2015 defense round.
Working as a group wasn't perfect - we were all in different timezones for a start, and the group decision to go for the Cloud track not ideal for me, but we got it done. We were accepted and February rolled around, I did a horrible job in the defense. I was not expert on the material and nerves ate me up, I barely touched the white board and generally did a poor job of demonstrating the skills of a VCDX. Two of us passed though so I couldn't blame the panel nor the submission.
I was determined to go again as soon as I received the result, and used the experience of the defense to improve the design - there were decisions I couldn't justify because I didn't think they were good and plenty of contradictions and typos that had escaped the ten or so reviews. I'd also labbed lots of vCD in the interim, for me no amount of reading can substitute for hands-on time and researching all the tough questions from the first defense had led me to lots of background I had been missing; expert means expert and I had some holes. The defense was much easier for it, though still nervous I could remember enough to not feel like an idiot, and white board a bunch of stuff too. Still finished feeling I'd failed again but after four times I think that's normal.
My advice after all of this? Go for it, it's worth doing for the knowledge you develop along the way if nothing else. Though I'm a bit of a certification collector (I have worked for channel partners so was paid to) I find vendor certs to be a useful training/development path. I didn't anticipate having to persevere quite so long on this, but the two year break and track change contributed, CCIE took me two attempts too, if something's worth doing it's not going to be easy.
As to the VCDX itself, you need to be an expert on the material, both on your own design and the process of getting to a logical design for a fictional customer. Every decision must be justified by the requirements and constraints! If you're anything like me you need to have enough knowledge of it all that when your performance is impaired by nerves you can still demonstrate enough of it to clear the bar.
Up next? Well as a networking guy I'd rather like to get the VCDX-NV, and every two years after renewing my CCIE (with the Data Center exam last year), I toy with idea of taking another lab...
With respect to Walmart my version of their motto would be 'Always be learning. Always'

Useful NSX CLI commands

Useful NSX CLI commands

I'm not going to repeat run of the mill install stuff, but just commands that I've found / people have pointed me to when I've hit issues.  I'll add some API stuff in another post at some point as there's a bunch of stuff not in the CLI at all yet too.

When controllers don't deploy (or deploy then get immediately deleted):
Check disk space on specified datastore
Check /var/log/netcpa.log on the ESX hosts for IP pool allocation issues on controllers
Frequently issues arise because of connectivity NSX Manager to Controllers, and the most common of all:  DNS and NTP issues.

In the Manager CLI there's a handy 'show running-config' these days, which doesn't show a whole lot but will show if you fat fingered it's own network settings.
'show manager log follow' tails the main log file, which aids with all kinds of deployment debugging as the errors can be more verbose than in the GUI.

To troubleshoot MTU issues:
‘esxcli network interface list’ ‘esxcli network nic list’ and ‘ping ++netstack=vxlan x.x.x.x -d -s 1600’  where x.x.x.x is the IP of another hosts VTEP.

vCNS commands that may work:
esxcli network vswitch dvs vmware vslan network mapping list --vds-name=myvds --vxlan-id=5001

esxcli network vswitch dvs vmware vxlan list
esxcli network vswitch dvs vmware vxlan config stats set --level 1
esxcli network vswitch dvs vmware vxlan stats list --vds-name=myvmware

esxcli network vswitch dvs vmware vxlan vmknic multicastgroup list --vds-name=myvds --vlan-id=100

esxcli network vswitch dvs vmware vxlan network stats list --vds-name=myvds --vxlan-id=5001

The dvfilter is the bit that sits between the vmnic and the vswitch and does the packet filtering (and presumably steering in the case of the PANW integration)

summarize-dvfilter - gives back a list of filters present on the host
pktcap-uw --dvfilter $filter-name can then be used to sniff traffic, with --PreDVFilter or --PostDVFilter to help figure out if a rule is not doing what is expected.

pktcap-uw -A    Broader packet capture on ESXi

vsipioctl getfwrules -f $filter_name

ESXi - Controller is TCP/1234

Equivalent of a 'sh cam dy' or sh mac-addr'
net-vdr -b –mac default+edge-1

On Edge, debug packet display interface Nic_0 host_192.168.1.1

show log follow

show service ipsec site

Rene's huge page of links:
http://vcdx133.com/2014/10/05/nsx-link-o-rama/

Logging:
When trying to introduce micro-segmentation to a running environment especially, and for debugging for ever after, not to mention for security auditing, logging is somewhat vital.
NSX has lots of logs all in different places, so redirecting them all to Log Insight / some central location is the way to go, to configure add the log host on NSX manager, and the Edges.
The distributed firewall is distributed :)  So add on every ESXi host:
esxcli system syslog config set --loghost=‘udp://192.168.110.241:514' on every ESXi host
esxcli system syslog reload
esxcli network firewall ruleset set --ruleset-id=syslog --enabled=true
esxcli network firewall refresh

Useful Linux / Virtual Appliance commands

If / when you need to recover the root password to a linux box or VMware virtual appliance:
Edit the kernel boot line and add init = /bin/bash in order to get a shell to reset the root password.

To disable IPv6 add these lines to the bottom of sysctl.conf:
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
Then run sudo sysctl -p or reboot

Plus add startup options for BIND if running, otherwise it will keep trying to use IPv6 anyhow:

OPTIONS="-4 -u bind"

To /etc/default/bind9

Always install NTP.

If running in a VM then install open-vm-tools, so much easier than installing VMware tools plus build essential, building the tools then having them break during a kernel upgrade sometime later.

LACP is provided by ifenslave:
In /etc/network/interfaces:

auto em1   # on my boxes Ubuntu changes eth0/1 to em1/2 on boot
iface em1 inet manual
bond-master bond0
auto em2
iface em2 inet manual
bond-master bond0

auto bond0
iface bond0 inet dhcp  # or more likely static
bond-mode 4     # mode 4 is 802.3ad / LACP
bond-miimon 100
bond-lacp-rate 1
bond-slaves em1 em2

To see if you're speaking LACP with the switch:
cat /proc/net/bonding/bond0

LLDP was as easy as 'apt-get install lldpd' on my hosts, output on the newer HP switches is a little funky, but my older Procurve gives the hostnames which is perfect.

'lldpcli show neighbors' shows what switch ports you're plugged into from the host side.

For cu -l /dev/ttyUSB0 -s 9600 to work ensure uucp user and group have r/w permissions on the device.  Even as root cu drops to uucp, and permission denied gives the not useful error of 'device in use'

I had no idea how many of the commands I regularly use are deprecated,
Deprecated Linux networking commands and their replacements
there's no way I'm going to stop typing the old ones yet, just as I still use wr mem on any Cisco device that will still accept it, but good to file the list of replacements me thinks.

Unattended updates are great - at least on systems where restarting services won't cause an outage (had a lot of mysterious SQL issues until I released the server was being restarted by this).  However while it cleans up downloaded files, it does not autoremove installed packages, with the issue that /boot fills up with never used kernel images.  To prevent this happening I added

0 0 * * 0 apt-get autoremove -y

to the root crontab.  If it already happened to you too, I have fixed many times with a combination of the following:




apt-get remove --purge 2.6.2x-xx-*

If /boot is full and apt-get remove or autoremove won't work, then 

rm -rf /boot/*-3.19.0-{25,56,58,59,61,65}-*

should create enough space to get apt working again.

Useful F5 TMSH commands

Useful F5 TMSH commands

These are TMSH commands I've needed repeatedly during lab work, where I've frequently had to setup a Virtual Edition F5 to test something out.  As always the idea if to get the mgmt interface reachable so you can use the GUI to license the box (physical or virtual) and complete setup.
TMSH came in with 11.4 I think but can't remember.  The management interface traditionally always came up with 192.168.1.245, but these days finally has a DHCP client available, theres plenty of times when that doesn't work or needs turning on though.

To add a static route to the mgmt interface:
tmsh
sys
create management-ip x.x.x.x/24
create management-route default gateway x.x.x.x
save config

Alternatively to activate the DHCP client:
tmsh modify sys db dhclient.mgmt value enabled

To add a default route:
tmsh create net route 0.0.0.0/0 gw x.x.x.x

save /sys config

After configuring AD, this is a handy way to test:
adtest -t auth -r "sjhwilkes.local" -u shamilton -w "password"

Installing new TMOS image from CLI:
tmsh install sys software image BIGIP-11.6.0.0.0.401.iso volume HD1.2 create-volume

When troubleshooting packet flow this is a helpful little guide to how internal routing works.  Experienced F5 folk understand the great power these boxes have, but a lot of complexity comes along with that - this is where the whole ADC/load balancer debate comes in.  Clearly I'm biased, but there's a world of things that can be done with an F5 which in my opinion makes it an Application Delivery Controller.  As to where exactly the distinguishing line falls and what other devices are where, gets into religion real fast...

1) Look for a match in a packet filter rule

2) Look for a virtual server match

3) Examine source IP for a stand alone SNAT

4) Check if the destination is a NAT

5) Check for a match with a self-IP

6) Search for an existing connection

Useful NetApp commands and links

Useful NetApp commands

I have an aging 2020 in my lab, which gradually eats drives and isn't lightning fast due to being configured with 7200RPM 500GB drives, lack of cache, etc.  That said it is rock solid apart from the disks, having proper failover, LACP to the network, snapshots and dedupe - this makes it difficult for me to make a decision to replace it with a prosumer level NAS.  One day I'll get organized and replace it with VSAN in my hosts, but it's going to be tricky to replace the controllers in my hosts with supported ones and migrate the data as I don't have the power to run both solutions at once, so it's going to need a long outage and taking a copy of all the data to a big SATA drive.

To unassign a disk:
disk assign 0b.30 0b.29 0b.28 0b.27 -s unowned -f
disk show (Show disk information)
disk show -n (Show unowned disks)
disk assign 0d.02.0 -s unowned (Changes ownership from owned to unowned or to other cluster member)
disk assign 0d.02.0 (assigns the disk to the controller you perform the command on)
options disk.auto_assign off (turns auto assign of unowned disks to controllers to off)
options disk.auto_assign on (turns auto assign of unowned disks to controllers to on)
storage show disk -p (displays primary, secondary port, shelf and bay in a metro cluster)

To remove fractional reservation:
vol options vol1 fractional_reserve 0

For lab use, change to a single spare - disables disk maintenance center:
options raid.min_spare_count 1

Display stats:
sysstat -s 5    # s is for summary
sysstat -x 1   # x is for extended

VIF
vif status (Displays the status of VIF interfaces)
vif create lacp vif0 ip e0a,e0b,e0c,e0d (Create a vif interface named VIF0 using lacp and a load balancing method of IP, bundled with interface e0a, e0b, e0c, e0d)
vif add vif0 e0a (Adds a down interface into an existing VIF)
vif destroy vif0 (deletes a VIF interface)

Network
vlan create vif0 10 (create a vlan on interface vif0. The interface will be vifo-10)
vlan delete vif0 10 (deletes the vlan interface vif0-10)
ifconfig vif0-10 x.x.x.x netmask x.x.x.x partner vif0-10 (Sets IP information on the interface named vif0-10 with a partner interface of vif0-10)
route add default 192.168.1.1 1 (Adds a default route of 192.168.1.1 with metric 1)

Nice sizing tool:
http://www.storagebyte.com/netapp.html

Growing a volume past 1TB (2020) or 2TB limit for SIS
If SIS has deduped the volume you will need to inflate the blocks again with undo sis from the privileged mode CLI.
priv set advanced
sis undo /vol/vol3

Manual snap restore:
snap restore -s .nightly.1 -r /vol/vol2/utility01/utility01-flat.vmdk /vol/vol2/utility01/utility01-flat.vmdk

The above is getting a little dated now, my hands-on with NetApp has been limited since I retired my lab 2020 (as a VMware employee VSAN is no cost to me and all flash is massively faster than the 26 7200RPM disks).  Part of the issue with the 2020 is that it only runs OnTap 7 - which works fine, but I hit an issue several times where the internal SAS drives were failing with regularity and I'd get replacements on eBay, only so find they'd been used on an OnTap 8 system.  There's no way from an OnTap 7 system to downgrade them so I'd have to send them right back...

Now OnTap 9 is current and 9.1 imminent, the simulator is probably the only way I'll get any hands on, Neil Anderson has handily written a detailed guide to building a virtual lab, http://www.flackbox.com/netapp-simulator/