FnordWiki - User contributions [en]

Mellanox Infiniband HCA and/or Ethernet adapters

2026-04-13T21:35:30Z

Adj: /* Set card mode */

I have a number of servers with ConnectX-3 cards to connect to various network infrastructure bits and bobs. Recently (today) installed on in a Cisco UCS C220 M5 server and realized that I have made no notes about things that have been done to make the cards work. Ooops.

== ConnectX-3 cards ==
Several of these are deployed around the home network. There are various models (pretty much all are ConnectX-3, though.)

=== Finding card info ===
List out all Mellanox cards in a machine. "15b3" is Mellanox's PCI vendor ID.
itops@syadasti:~$ '''lspci -d 15b3:'''
d8:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
itops@syadasti:~$
First column of output is the PCI bus ID of the card. This is used in the next bunch of commands. Running with elevated privileges (under sudo) allows reading to the capabilities info. That isn't really necessary for this step, but causes no harm.
itops@syadasti:~$ '''sudo lspci -s d8:00.0 -vv'''
d8:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
Subsystem: Mellanox Technologies Device 0003
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 83
NUMA node: 1
IOMMU group: 20
Region 0: Memory at fbe00000 (64-bit, non-prefetchable) [size=1M]
Region 2: Memory at 4ffff800000 (64-bit, prefetchable) [size=8M]
Expansion ROM at fbd00000 [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
Capabilities: [60] Express Endpoint, IntMsgNum 0
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
Capabilities: [148] Device Serial Number f4-52-14-03-00-2c-5e-00
Capabilities: [108] Single Root I/O Virtualization (SR-IOV)
Capabilities: [154] Advanced Error Reporting
Capabilities: [18c] Secondary PCI Express
Kernel driver in use: mlx4_core
Kernel modules: mlx4_core

itops@syadasti:~$
Query the card to find out its code revision, Ethernet MAC address(es), Infiniband GUID(s), and the Mellanox Parameter Set ID (PSID). This does need the elevated privileges provided by sudo:
itops@syadasti:~$ '''sudo mstflint -d d8:00.0 q full'''
Image type: FS2
FW Version: 2.36.5000
FW Release Date: 26.1.2016
MIC Version: 1.5.0
Config Sectors: 2
Product Version: 02.36.50.00
Rom Info: type=PXE version=3.4.718
Device ID: 4103
Description: Node Port1 Port2 Sys image
GUIDs: f4521403002c5e00 f4521403002c5e01 f4521403002c5e02 f4521403002c5e03
MACs: f452142c5e01 f452142c5e02
VSD:
PSID: MT_1090111019
itops@syadasti:~$

=== Where to get firmware images ===
See that PSID in the output from the <code>mstflint</code> query operation? We are running relase 2.36.5000 of the code in the ConnectX-3 chip on this card. There are also code blobs for BIOS/UEFI that can be installed.

nVidia bought Mellanox some years ago. With that purchase, firmware images for switches and adapter cards can be found at https://network.nvidia.com/support/firmware/firmware-downloads/. If it is not there in the future, try Duck-Duck-Going for something like "nVidia adapter card firmware download" until you find the right thing. Today's card is a "ConnectX-3 Pro" (as seen in the output of <code>lspci -d 15b3:</code>). These cards came in Infiniband only and Ethernet+Infiniband (and maybe just Ethernet, too?) varieties. Navigating through nVidia firmware download, I found a card with a matching PSID under the ConnectX-3 Pro Infiniband link. PSID MT_1090111019 looks to go with a Mellanox MCX354A-FCCT (dual port) or MCX353A-FCCT (single port) adapter. Download the corresponding ZIP archive from nVidia's site. As this is written, that is https://www.mellanox.com/downloads/firmware/fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin.zip.

=== Burning new firmwares ===
Unpack the just-downloaded ZIP file and find a .bin file inside. And compare what is currently on the card with what is in the .bin file:
itops@syadasti:~/Mlnx-HCA-Firmware$ '''diff -u --color <(sudo mstflint -d d8:00.0 q full) <(mstflint -i fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin q full) '''
--- /dev/fd/63 2026-04-13 19:15:20.421912090 +0000
+++ /dev/fd/62 2026-04-13 19:15:20.421912090 +0000
@@ -1,13 +1,14 @@
Image type: FS2
-FW Version: 2.36.5000
-FW Release Date: 26.1.2016
-MIC Version: 1.5.0
+FW Version: 2.42.5000
+FW Release Date: 5.9.2017
+MIC Version: 2.0.0
Config Sectors: 2
-Product Version: 02.36.50.00
-Rom Info: type=PXE version=3.4.718
+PRS Name: cx3pro_MCX354A_fdr_09v.prs
+Product Version: 02.42.50.00
+Rom Info: type=PXE version=3.4.752
Device ID: 4103
Description: Node Port1 Port2 Sys image
-GUIDs: f4521403002c5e00 f4521403002c5e01 f4521403002c5e02 f4521403002c5e03
-MACs: f452142c5e01 f452142c5e02
-VSD:
+GUIDs: 0002c9000100d050 0002c9000100d051 0002c9000100d052 0002c9000100d050
+MACs: 0002c9000001 0002c9000002
+VSD: n/a
PSID: MT_1090111019
itops@syadasti:~/Mlnx-HCA-Firmware$
Along with the firmware version numbers, we can see that the Infiniband GUIDs and Ethernet MAC addresses are different. It would probably be good to keep those even after the new firmware image is flashed. Note that the PSID stays the same. I believe it is possible to change an adapter card from a ConnectX-3 to a ConnectX-3 Pro by flashing different images into it. But we do not need to do that here.

Now, let's make a backup of what is in the card:
itops@syadasti:~/Mlnx-HCA-Firmware$ '''sudo mstflint -d d8:00.0 ri backup_$(date -Iseconds)_2.36.5000.bin'''
itops@syadasti:~/Mlnx-HCA-Firmware$ '''mstflint -i backup_2026-04-13T19\:58\:45+00\:00_2.36.5000.bin q full'''
Image type: FS2
FW Version: 2.36.5000
FW Release Date: 26.1.2016
MIC Version: 1.5.0
Config Sectors: 2
Product Version: 02.36.50.00
Rom Info: type=PXE version=3.4.718
Device ID: 4103
Description: Node Port1 Port2 Sys image
GUIDs: f4521403002c5e00 f4521403002c5e01 f4521403002c5e02 f4521403002c5e03
MACs: f452142c5e01 f452142c5e02
VSD:
PSID: MT_1090111019
itops@syadasti:~/Mlnx-HCA-Firmware$

Remove drivers from the running kernel and flash the new firmware image:
itops@syadasti:~/Mlnx-HCA-Firmware$ '''lsmod | grep mlx'''
mlx4_ib 262144 0
ib_uverbs 204800 1 mlx4_ib
mlx4_en 167936 0
mlx4_core 442368 2 mlx4_ib,mlx4_en
ib_core 524288 2 mlx4_ib,ib_uverbs
itops@syadasti:~/Mlnx-HCA-Firmware$ '''sudo modprobe -rv mlx4_ib mlx4_en mlx4_core'''
rmmod mlx4_ib
rmmod ib_uverbs
rmmod mlx4_en
rmmod mlx4_core
itops@syadasti:~/Mlnx-HCA-Firmware$ '''sudo mstflint -d d8:00.0 --guids f4521403002c5e00,f4521403002c5e01,f4521403002c5e02,f4521403002c5e03 --macs f452142c5e01,f452142c5e02 --image fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin burn'''
You are about to change the Guids/Macs/Uids on the device:

New Values Current Values
Node GUID: f4521403002c5e00 f4521403002c5e00
Port1 GUID: f4521403002c5e01 f4521403002c5e01
Port2 GUID: f4521403002c5e02 f4521403002c5e02
Sys.Image GUID: f4521403002c5e03 f4521403002c5e03
Port1 MAC: f452142c5e01 f452142c5e01
Port2 MAC: f452142c5e02 f452142c5e02

Do you want to continue ? (y/n) [n] : '''y'''

Current FW version on flash: 2.36.5000
New FW version: 2.42.5000

Burning FS2 FW image without signatures - OK
Restoring signature - OK
itops@syadasti:~/Mlnx-HCA-Firmware$
And query the card to check on the flashing:
itops@syadasti:~/Mlnx-HCA-Firmware$ '''sudo mstflint -d d8:00.0 q full'''
Image type: FS2
FW Version: 2.42.5000
FW Version(Running): 2.36.5000
FW Release Date: 5.9.2017
MIC Version: 2.0.0
Config Sectors: 2
PRS Name: cx3pro_MCX354A_fdr_09v.prs
Product Version: 02.42.50.00
Rom Info: type=PXE version=3.4.752
Device ID: 4103
Description: Node Port1 Port2 Sys image
GUIDs: f4521403002c5e00 f4521403002c5e01 f4521403002c5e02 f4521403002c5e03
MACs: f452142c5e01 f452142c5e02
VSD:
PSID: MT_1090111019
itops@syadasti:~/Mlnx-HCA-Firmware$
New code isn't running right now. But there is an "image reactivate" operation that could have been performed along with the flashing. Let's give that a go here:
itops@syadasti:~/Mlnx-HCA-Firmware$ '''sudo mstflint -d d8:00.0 ir'''
-E- Failed to execute image reactivation on device d8:00.0. Error: Operation not supported..
itops@syadasti:~/Mlnx-HCA-Firmware$
Hmmm. A different tool, maybe?
itops@syadasti:~/Mlnx-HCA-Firmware$ '''sudo mstfwreset -d d8:00.0 query'''
-E- Unsupported Device: d8:00.0 (ConnectX3Pro).
itops@syadasti:~/Mlnx-HCA-Firmware$
Fine, be that way. Halt the OS and power cycle the machine. Grrrr. Fast forward a bit (power cycle, OS start done) and we can check again:
itops@syadasti:~$ '''sudo mstflint -d d8:00.0 query full'''
Image type: FS2
FW Version: 2.42.5000
FW Release Date: 5.9.2017
MIC Version: 2.0.0
Config Sectors: 2
PRS Name: cx3pro_MCX354A_fdr_09v.prs
Product Version: 02.42.50.00
Rom Info: type=PXE version=3.4.752
Device ID: 4103
Description: Node Port1 Port2 Sys image
GUIDs: f4521403002c5e00 f4521403002c5e01 f4521403002c5e02 f4521403002c5e03
MACs: f452142c5e01 f452142c5e02
VSD:
PSID: MT_1090111019
itops@syadasti:~$
So that is looking correct. Yay!

=== Set card mode ===
This server has a 2 port card installed. And that card only has one Ethernet port right now. At last as listed by <code>ip link list</code>. This seems to be a Linux kernel driver thing more than a hardware configuration thing, though:
itops@syadasti:~$ '''sudo mstconfig -d d8:00.0 query'''

Device #1:
----------

Device type: ConnectX3Pro
Device: d8:00.0

Configurations: Next Boot
SRIOV_EN True(1)
NUM_OF_VFS 8
LINK_TYPE_P1 VPI(3)
LINK_TYPE_P2 VPI(3)
LOG_BAR_SIZE 3
BOOT_PKEY_P1 0
BOOT_PKEY_P2 0
BOOT_OPTION_ROM_EN_P1 True(1)
BOOT_VLAN_EN_P1 False(0)
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 PXE(1)
BOOT_VLAN_P1 1
BOOT_OPTION_ROM_EN_P2 True(1)
BOOT_VLAN_EN_P2 False(0)
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 PXE(1)
BOOT_VLAN_P2 1
IP_VER_P1 IPv4(0)
IP_VER_P2 IPv4(0)
CQ_TIMESTAMP True(1)
itops@syadasti:~$
Note that both LINK_TYPEs are "VPI" which is Mellanox speak for "I'm bilingual. I will speak both Infiniband '''and''' Ethernet." But there's only one. Let's set them both to Ethernet mode, reload the kernel drivers, and see what happens:
itops@syadasti:~$ '''sudo mstconfig -d d8:00.0 --enable_verbosity set LINK_TYPE_P1=2 LINK_TYPE_P2=2'''

Device #1:
----------

Device type: ConnectX3Pro
Device: d8:00.0

Configurations: Default Next Boot New
LINK_TYPE_P1 VPI(3) VPI(3) ETH(2)
LINK_TYPE_P2 VPI(3) VPI(3) ETH(2)
Device Firmware does not support reading current configurations

Apply new Configuration? (y/n) [n] : '''y'''
Applying... Done!
-I- Please reboot machine to load new configurations.
itops@syadasti:~$ '''sudo modprobe -rv mlx4_ib mlx4_en mlx4_core'''
rmmod mlx4_ib
rmmod ib_uverbs
rmmod mlx4_en
rmmod mlx4_core
itops@syadasti:~$ '''sudo modprobe -v mlx4_core'''
insmod /lib/modules/6.12.73+deb13-amd64/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko.xz
itops@syadasti:~$
And now? Well, we can see two Ethernet adapters in that PCI slot. And their MAC addresses match what was set in the firmware flashing operation above, too:
itops@syadasti:~$ '''ip link list | grep ^[0-9]'''
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
2: eno5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
3: eno6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
4: ens1f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
5: ens1f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
6: ens1f2np2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
7: ens1f3np3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
8: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
9: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
12: enp216s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
13: enp216s0d1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
itops@syadasti:~$ '''ip link list dev enp216s0; ip link list dev enp216s0d1'''
12: enp216s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether f4:52:14:2c:5e:01 brd ff:ff:ff:ff:ff:ff
altname enxf452142c5e01
13: enp216s0d1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether f4:52:14:2c:5e:02 brd ff:ff:ff:ff:ff:ff
altname enxf452142c5e02
itops@syadasti:~$
Set link up, see what's on the other end:
itops@syadasti:~$ '''sudo ip link set up enp216s0'''
itops@syadasti:~$ '''sleep 180'''
itops@syadasti:~$ '''sudo lldpcli show neighbors ports enp216s0'''
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
Interface: enp216s0, via: LLDP, RID: 2, Time: 0 day, 00:05:43
Chassis:
ChassisID: mac 00:02:c9:6d:4f:30
SysName: mellanox-sx6036-rack-2
SysDescr: SX6036,MLNX-OS,SWv3.6.8012
Capability: Bridge, on
Capability: Router, off
Port:
PortID: ifname Eth1/4
PortDescr:
TTL: 120
Unknown TLVs:
TLV: OUI: 00,80,C2, SubType: 9, Len: 21 04,00,11,22,33,19,19,19,19,00,00,00,00,02,02,02,02,00,00,00,00
TLV: OUI: 00,80,C2, SubType: 10, Len: 21 00,00,11,22,33,19,19,19,19,00,00,00,00,02,02,02,02,00,00,00,00
TLV: OUI: 00,80,C2, SubType: 12, Len: 1 00
-------------------------------------------------------------------------------
itops@syadasti:~$
== Now what? ==
Now, it is an Ethernet card. Treat it like any other. Make bond interfaces. And VLANed sub-interfaces. Copy bits across it. Have fun!

Mellanox Infiniband HCA and/or Ethernet adapters

2026-04-13T21:32:50Z

Adj: /* Set card mode */

Mellanox Infiniband HCA and/or Ethernet adapters

2026-04-13T21:23:49Z

Adj: /* Set card mode */

Mellanox Infiniband HCA and/or Ethernet adapters

2026-04-13T20:50:53Z

Adj: /* Burning new firmwares */

Mellanox Infiniband HCA and/or Ethernet adapters

2026-04-13T19:13:18Z

Adj: /* Where to get firmware images */

Mellanox Infiniband HCA and/or Ethernet adapters

2026-04-13T18:42:18Z

Adj: /* ConnectX-3 cards */

Mellanox Infiniband HCA and/or Ethernet adapters

2026-04-13T03:15:41Z

Adj: Created page with "I have a number of servers with ConnectX-3 cards to connect to various network infrastructure bits and bobs. Recently (today) installed on in a Cisco UCS C220 M5 server and realized that I have made no notes about things that have been done to make the cards work. Ooops. == ConnectX-3 cards == === Finding card info === === Where to get firmware images === === Burning new firmwares === === Set card mode ==="

Main Page

2026-04-13T03:04:28Z

Adj: /* FnordNet infrastructure */

Consult the [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.

== Ceph education events ==
[https://sofree.us// Software Freedom School] is keen to learn more about [https://ceph.com Ceph]. As such, I've been encouraged to cook up a group education session on the topic. See the [[work in progress sofree.us 2025Q3 Ceph handson]] thoughts.

== Other Ceph things ==
* [[Rotating applications' Ceph S3 credentials]]
* [[Ceph performance metrics]]

== Recreational Shooting Events ==
* [[ Dec 2022 Vedauwoo ]]
* [[ Mar 2023 Vedauwoo ]]
* [[ Jun 2023 Birthday and Bullets ]]

== Vehicular Conveyances ==
* [[ 1997 Lexus LX 450 176849 ]]
* [[ 2002 Honda CR-V ]]
* [[ 2012 Honda Pilot Touring ]]

== Pictures with Shadow ==
[[ Pictures with Shadow ]]

== Getting started ==
* [http://www.mediawiki.org/wiki/Manual:Configuration_settings Configuration settings list]
* [http://www.mediawiki.org/wiki/Manual:FAQ MediaWiki FAQ]
* [https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]

== House Stuff ==
* [[The big ass list]]

== IPv6 stuff ==
* [[IPv4_and_IPv6_compared | IPv4 and IPv6 compared]]
* [[IPv6_address_classes | IPv6 address classes]]
* [[where_to_get_a_tunnel | Where to get an IPv6 address ]]

== DockStar stuff ==
* [[DockStar]]

== PXE booting for BIOS udpates ==
* [[PXE_booting_for_BIOS_updates | PXE booting for BIOS updates]]

== Ur-Quan masters ==
A port of Star Control II to Linux
* [[UQM | UQM]]

== Dell PowerEdge servers ==
* [[ PowerEdge 1950 Debian install ]]
* [[ An incremental strategy for large amounts of storage ]]
* [[PowerEdge BMC updates with Debian]]

== PA-RISC servers ==
* [[ Debian on an HP 9000 A500 ]]
* [[ rp2470/A500 related hardware ]]

== Sun/Oracle servers ==
* [[T5140 ILOM reset ]]
* [[T5140 memory configuration]]

== FnordNet infrastructure ==
* [[ 44th Avenue Data Center ]]
* [[ New new rack ]]
* [[ New Rack ]] (being decommissioned at present)
* [[ Rack ]] (retired)
* [[ Dell PowerConnect 2716 ]] 16-port Gigabit Ethernet switch
* [[ WRT54G ]] running OpenWRT

* [[ Force10 S50 ]] Ethernet switches
* [[ Qlogic SANbox 2 ]] 16 port Fibre Channel switch
* [[ Disk partitioning miscellanea ]]
* [[ IPSEC ]]
* [[ Computer stuff shopping list ]]
* [[ discordichunk ]]
* [[ LTO labels ]]
* [[ AP9210 ]] network attached power switch
* [[ Infiniband ]]
* [[ Backups ]]
* [[ ZFS ]]
* [[ Headless Debian Installation ]]
* [[ Postfix+uucp+ssh | UNIX email when SMTP is blocked ]]
* [[ Big Mass Storage ]]
* [[ LeftHand Networks NSM 100 | creating something useful from a LeftHand Networks NSM 100 ]]
* Going [[ Hiveless ]] (Aerohive WiFi APs without cloud based management.
* [[ Watchguard_XTM_5 ]] firewall appliance running a non-Watchguard OS
* Dealing with[[ password-locked disks ]]
* A bare metal provisioning solution using OpenStack [[ Ironic-at-home ]]
* [[R630 configuration]]
* [[OpenWRT Aruba AP-303|2025 WiFi infrastructure]] on Aruba AP-303s running OpenWRT
* [[FibreChannel Storage]]
* A few notes about [[Supermicro servers]]
* [[ Powervault MD3060e ]] big ass JOBDs
* [[ Hardware inventory ]]
* [[ Backup and Recovery ]]
* [[ Notes on VM building ]]
* [[ IPv6 connectivity ]]
* [[ In transit data security ]] using EFF's Let's Encrypt, x509 certificates, and TLS
* [[ Cisco things ]]
* [[ Mellanox Infiniband HCA and/or Ethernet adapters ]]

== VOIP stuff ==
* [[Asterisk Hardening]]
* [[Cisco 7905]]
* [[Polycom Handset]]

== Miscellaneous hardware ==
* [[ Sun StorEdge T300 ]]

== Imagination Station things ==
* [[ Photos of stuff ]]

== Kendra's wedding trip photos ==
* [[ Kendra's wedding trip photos ]]

== IT certifications and education ==
=== (ISC)2 ===
* [[ CISSP ]] Certified Information Systems Security Professional

=== VMware ===
* [[ VCP5-DV ]] VMware Certified Professional for vSphere 5 Datacenter Virtualization
=== Microsoft ===
* [[ 70-413 ]] Designing and Implementing a Server Infrastructure
=== Symantec ===
=== Oracle ===

== Other interesting computer things ==
* [[Deconstructing chatiw]]
== Things in the big blue room ==
=== Sprinkler system at the house ===
Seven zones:
# [[Sprinkler Zone 1|Back yard, west end]]
# [[Sprinkler Zone 2|Back yard, west center]]
# [[Sprinkler Zone 3|Back yard, east center]]
# [[Sprinkler Zone 4|Back yard, east]]
# [[Sprinkler Zone 5|Back yard, northeast and east of driveway]]
# [[Sprinkler Zone 6|Front yard]]
# [[Sprinkler Zone 7|west side of house and front flower bed]]
=== Bent sticks ===
* Kids' bows
** [[Bow 20140000]] Zeroth build. For Ryker.
** [[Bow 20140001]] Oneth build. For Nya.

== Work related ==
* [[CU Norlin Library Linux training]]

== OpenStack things ==
* [[ Terminating access to an OpenStack environment ]]
* [[ SNOWMAN | Do you want to build a snow man? ]]

== random stuff ==
* [[ Gwenn | Entertainment ideas for Gwenn ]]

Mellanox SX6036

2026-04-12T18:06:14Z

Adj: /* Unlocking the bootloader */

== Console access ==
=== Serial port ===
There is an 8P8C modular (RJ45) connector on the right side of the port side of the switch. This connector uses the Cisco pinout and a common Cisco console cable works great for driving it.

Serial settings are 9600 bits per second, 8 data bits, no parity bits, 1 stop bit (9600 8N1)

=== Default credentials ===
User '''<code>admin</code>'''.

Password '''<code>admin</code>'''.

=== Resetting the password ===
In the lower left corner of the port side of the switch, there is a small hole labelled "RST". Holding the button inside with a paperclip or similar tool for 15 seconds will reset the switch and erase any password set for the '''<code>admin</code>''' user.

== Firmware updates ==
Updating firmware on these switches is a protracted process. Firmware images are available on the public internet, but some searching is required. To bring a switch from a 3.2.''x'' release up to the final 3.6.8012 release, locate the following images on the web and save them to an SSHable or HTTP(S)able location near you:
* image-PPC_M460EX-3.4.3002.img
* image-PPC_M460EX-3.5.1006.img
* image-PPC_M460EX-3.6.1002.img
* image-PPC_M460EX-3.6.8012.img
* image-PPC_M460EX-SX_3.3.3400.img
* image-PPC_M460EX-SX_3.3.5006.img
* image-PPC_M460EX-SX_3.4.0012.img

=== New firmware image flashing procedure ===
This is a privileged operation, so first off gain elevated privileges...

'''<code>enable</code>'''

'''<code>image fetch http://172.17.0.17/~adj/image-PPC_M460EX-SX_3.3.3400.img</code>'''

'''<code>image install image-PPC_M460EX-SX_3.3.3400.img location 2 progress track verify require-sig</code>'''

'''<code>reload</code>'''

Wait for U-Boot to prompt about a menu, and send a Ctrl-B before the countdown reaches 0. Select option 2 (no need for a newline) to boot the newly flashed image. Iterate through each of the newer images, alternating location 1 and location 2 in the '''<code>image install ...</code>''' command.

When updates version installations are finally complete, it might be good to ensure that 3.6.8012 is flashed to both locations 1 and 2.

== Useful CLI commands ==

=== Find MAC address of 1000baseT management interface 0 ===
Good for setting up the DHCP server's config for this switch. For some reason, this command needs elevated privileges. Grrr.

'''<code>enable</code>'''

'''<code>show interfaces mgmt0 | include HW</code>'''

=== Find system serial number without reading the physical tag ===
This is useful for license key generation. And inventory purposes.

'''<code>show inventory</code>'''

=== Find system host ID ===
Also can be used for license key generation. Looks suspiciously like the mgmt0 MAC address...

'''<code>show version | include Host</code>'''

== Licensing features ==
Mellanox's SwitchX2 ASIC is capable of performing Infiniband '''and''' Ethernet switching. Mellanox the company likes to lock some of these features behind licensing keys. Which is annoying. But with enough effort, is a surmountable obstacle

=== The license generation tool ===
Inside the MLNX-OS distribution images (those files with names like '''<code>image-PPC_M460EX-3.6.8012.img</code>''') there exists a gzipped tarball containing the MLNX-OS filesystem. Inside the tarball is a directory called '''<code>/opt/tms/bin</code>'''. And inside ''that'' directory, one can find the license key generation program, '''<code>genlicense</code>'''. The MLNX-OS filesystem image is a PowerPC Linux distribution that can be run from a Qemu emulated big-endian PowerPC machine, or natively on big-endian PowerPC hardware. We can use this to generate feature licenses until the cows come home.

=== Our first license, shell access from the switch CLI ===
There is a hidden CLI command, '''<code>_shell</code>''' which will cause the CLI to '''<code>execve("/bin/bash", ...)</code>'''. This command is privileged (must be run from enable mode) and locked behind a license key. The following is a working CLI session to install a key for this feature:
Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-1 login: '''admin'''
Password: '''passwordgoesherebutisnotechoed'''
Last login: Sun Feb 19 23:25:30 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal '''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''license install LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''show licenses'''
License 1: LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1
Feature: RESTRICTED_CMDS_GEN2
Description: Access to restricted system functionality
Valid: yes
Active: yes
mellanox-sx6036-rack-1 [standalone: master] #

=== Ethernet features: Ethernet, Ethernet L2, and Ethernet L3 ===
We will generate these from the switch as we already have shell access and can use the included genlicense tool.

Running the following from the switch's shell will print a key for a non-hardware-locked, non-date-locked license to activate the switch's full Ethernet capabilities:
[admin@mellanox-sx6036-rack-0 ~]# '''cd /opt/tms/bin'''
[admin@mellanox-sx6036-rack-0 bin]# '''./genlicense 2 EFM_SX m2l0n%0x9 -o 53 true -o 51 true -o 58 true'''
LK2-EFM_SX-5M11-5K11-5T11-88A1-BBD0-JP82-X
[admin@mellanox-sx6036-rack-0 bin]#

A quick explanatory diversion is in order here. There are 2 types of keys. genlicense's first option above specifies key type 2. '''<code>EFM_SX</code>''' is the '''<code>Generic SX license</code>''' type. By itself it does nothing, but all of the '''<code>-o</code>''' flags specify further features to be activated. A complete list of these can be generated by running '''<code>/opt/tms/bin/genlicense 2</code>'''. There are 14 of these additional options, and their meaning is not well documented. The magic word '''<code>m2l0n%0x9</code>''' can be found by watching for '''<code>strlen()</code>''' calls in the process flow of '''<code>/opt/tms/bin/dumplicense</code>''' when run under '''<code>ltrace</code>'''.

And it can be installed in the switch's configuration database by leaving the shell (which will disconnect the terminal session), logging in again, running '''<code>enable</code>''' to get elevated privileges, and using the '''<code>license install</code>''' configuration command like so:
Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-1 login: '''admin'''
Password:
Last login: Sun Feb 19 23:31:51 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''license install LK2-EFM_SX-5M11-5K11-5T11-88A1-BBD0-JP82-X'''
License was installed successfully. Please wait 1 minute before further configurations.
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] #

== Switch configuration commands ==
Some of these things will need licenses first.

=== I can haz Ethernet? ===
Install the Ethernet key as shown above and confirm it is working:
mellanox-sx6036-rack-1 [standalone: master] # '''show system capabilities'''
IB: Supported, L2, Adaptive Routing
Ethernet: Supported, L2, L3
GW: Not supported
Max SM nodes: 648
IB Max licensed speed: FDR10
Ethernet Max licensed speed: 56Gb
mellanox-sx6036-rack-1 [standalone: master] #
It says we can do Ethernet. Which is cool. But it does not have the Infiniband-Ethernet gateway feature licensed. And this is required to enable the switch to run in VPI mode. VPI is "Virtual Protocol Interconnect" or Mellanox's way of saying "We can do Infiniband ''and'' Ethernet on a single switch." So another license is needed. Here's a key for that:
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''license install LK2-EFM_SX-5N21-488A-182A-UQXB-Y6'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''show system capabilities '''
IB: Supported, L2, Adaptive Routing
Ethernet: Supported, L2, L3
GW: Supported
Max SM nodes: 648
IB Max licensed speed: FDR10
Ethernet Max licensed speed: 56Gb
mellanox-sx6036-rack-1 [standalone: master] #
We will need to do this next thing to convert the switch from an Infiniband-only switch to an Infiniband-and-Ethernet switch:
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''system profile vpi-single-switch'''
Warning - confirming will cause system reboot and all configuration will be deleted
Type 'yes' to confirm profile change: '''yes'''
Allow time for the switch to reboot and log in again. When it next starts up, run the command '''<code>show system profile</code>''' to confirm that it is in VPI mode:
mellanox-sx6036-rack-1 [standalone: master] > '''show system profile'''

Profile: vpi-single-switch

mellanox-sx6036-rack-1 [standalone: master] >
And finally, let's set some ports to Ethernet mode:
mellanox-sx6036-rack-1 [standalone: master] > '''show ports type'''
InfiniBand: 1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10 1/11 1/12 1/13 1/14 1/15 1/16 1/17 1/18 1/19 1/20 1/21 1/22 1/23 1/24 1/25 1/26 1/27 1/28 1/29 1/30 1/31 1/32 1/33 1/34 1/35 1/36
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface ib 1/19-1/36'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ib 1/19-1/36) # '''shutdown'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ib 1/19-1/36) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''port 1/19-1/36 type ethernet'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface ethernet 1/19-1/36'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/19-1/36) # '''no shutdown'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/19-1/36) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show ports type'''
Ethernet: 1/19 1/20 1/21 1/22 1/23 1/24 1/25 1/26 1/27 1/28 1/29 1/30 1/31 1/32 1/33 1/34 1/35 1/36
InfiniBand: 1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10 1/11 1/12 1/13 1/14 1/15 1/16 1/17 1/18
mellanox-sx6036-rack-1 [standalone: master] #

=== Switch management items ===
==== Password for the admin user ====
Factory defaults are probably not a good idea, so let's tighten things up a bit.
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''username admin password ''the_password_goes_here'' '''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] #

==== IP addressing for the management interface ====
The following will configure the switch to retrieve its IP address, subnet mask, router IP address, name server info, and hostname from DHCP server on the management interface's local network.
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface mgmt0'''
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # '''dhcp'''
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # '''dhcp hostname'''
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] #

==== NTP ====
Let's add a couple of NTP servers to the config and discipline our poorly behaved battery backed clock:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''ntp server 172.16.10.2'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''ntp server 172.17.0.17'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show ntp'''

NTP is administratively : enabled
NTP Authentication administratively: disabled

Clock is synchronized:
Reference: 172.16.10.2
Offset : -0.006 ms

Active servers and peers:
172.16.10.2:
Conf Type : serv
Status : sys.peer(*)
Stratum : 3
Offset(msec) : -0.006
Ref clock : 45.45.184.14
Poll Interval (sec): 64
Last Response (sec): 8
Auth state : none

172.17.0.17:
Conf Type : serv
Status : pending
Stratum : 2
Offset(msec) : 0.429
Ref clock : 198.60.22.240
Poll Interval (sec): 64
Last Response (sec): 7
Auth state : none

mellanox-sx6036-rack-1 [standalone: master] #
=== Ethernet goodies ===
==== LLDP ====
Turn it on and test it out like so:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''lldp'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show lldp local'''
LLDP: enabled

Local global configuration
Chassis sub type: Mac Address
Chassis id: 00:02:c9:6d:4d:b0
System Name: mellanox-sx6036-rack-1
System Description: SX6036,MLNX-OS,SWv3.6.8012
Supported capabilities: B,R
Supported capabilities enabled: B
mellanox-sx6036-rack-1 [standalone: master] # '''show lldp interfaces ethernet 1/36'''
TLV flags
PD: port-description
SN: sys-name
SD: sys-description
SC: sys-capabilities
MA: management-address
ETS-C: ETS-Configuration
ETS-R: ETS-Recommendation
AP: Application Priority
PFC: Priority Flow Control
CEE: Converged Enhanced Ethernet DCBX version
MED-CAP: Media Capabilities
MED-NWP: MED-Network Policy

-----------------------------------------------------------------------------------
Interface Receive Transmit TLVs
-----------------------------------------------------------------------------------
Eth1/36 Enabled Enabled PD, SN, SD, SC, PFC, AP, ETS-C, ETS-R
mellanox-sx6036-rack-1 [standalone: master] # '''show lldp interfaces ethernet 1/36 remote'''

Eth1/36

Remote Index: 1
Remote chassis id: cc:4e:24:83:4b:36 ; chassis id subtype: Mac Address (4)
Remote port-id: cc:4e:24:83:4b:6c ; port id subtype: Mac Address (3)
Remote port description: 40GigabitEthernet1/2/6
Remote system name: brocade-icx6610-48p-rack-1
Remote system description: Not Advertised
Remote system capabilities supported: B,R ; enabled B,R
Remote Management Addresses:
SubType Address
------------------------------------
IPv4 172.16.10.17

No Remote PFC entry

No Remote ETS entry

No Remote Application Priority entry

mellanox-sx6036-rack-1 [standalone: master] #
And enjoy getting to know all the things about your neighbors.
==== Multiple Spanning Tree Protocol ====
Wherein we avoid broadcast loops. Because they are bad, even if they do make for lots of blinken lights.
Our Brocade switches are running MSTP and, fortunately, MLNX-OS supports that on our SX6036es as well.
Setting the switch to MSTP mode is pretty simple:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''spanning-tree mode mst'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show spanning-tree'''

Switch : ethernet-default
Spanning tree protocol mst : enabled
Spanning tree force version: 3

Root ID:
Priority : 32768
Address : 00:02:c9:6d:4c:b0
Cost : 0
Port : Eth1/36
Hello Time (sec) : 2
Max Age (sec) : 20
Forward Delay (sec): 15

MST00:
Bridge is executing the mstp compatible Spanning Tree Protocol

Bridge ID:
Priority : 32768
Address : 00:02:c9:6d:4d:b0
Hello Time (sec) : 2
Max Age (sec) : 20
Forward Delay (sec): 15

L: Loop Inconsistent
R: Root Inconsistent
G: BPDU Guard Inconsistent

--------------------------------------------------------------------------
Interface Role Sts Cost Prio Type
--------------------------------------------------------------------------
Eth1/36 Root Forwarding 500 128 normal

mellanox-sx6036-rack-1 [standalone: master] #
==== VLANs ====
Create our standard assortment of VLANs like so:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 10 name Management'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 10) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 100 name "WAN 0 (Comcast 44th Avenue)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 100) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 101 name "WAN 1 (Allo)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 101) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 102 name "WAN 2 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 102) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 103 name "WAN 3 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 103) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 104 name "WAN 4 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 104) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 105 name "WAN 5 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 105) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 106 name "WAN 6 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 106) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 107 '''name "WAN 7 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 107) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 108 name "WAN 8 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 108) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 109 name "WAN 9 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 109) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1000 name "Internal Services"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1000) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1001 name "Internal clients"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1001) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1002 name "Ceph front-end"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1002) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1003 name "Ceph back-end"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1003) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 3900 name "Guest network"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 3900) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show vlan'''
----------------------------------------------------------------------
VLAN Name Ports
----------------------------------------------------------------------
1 default Eth1/19, Eth1/20, Eth1/21, Eth1/22, Eth1/23,
Eth1/24, Eth1/25, Eth1/26, Eth1/27, Eth1/28,
Eth1/29, Eth1/30, Eth1/31, Eth1/32, Eth1/33,
Eth1/34, Eth1/35, Eth1/36
10 Management
100 WAN 0 (Comcast 44th Avenue)
101 WAN 1 (Allo)
102 WAN 2 (unused)
103 WAN 3 (unused)
104 WAN 4 (unused)
105 WAN 5 (unused)
106 WAN 6 (unused)
107 WAN 7 (unused)
108 WAN 8 (unused)
109 WAN 9 (unused)
1000 Internal Services
1001 Internal clients
1002 Ceph front-end
1003 Ceph back-end
3900 Guest network
mellanox-sx6036-rack-1 [standalone: master] #
And now, to assign all those VLANs as tagged on port 1/36:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface ethernet 1/36'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport mode trunk'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan 10'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan add 100-109'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan add 1000-1003'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan add 3900'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show vlan'''
----------------------------------------------------------------------
VLAN Name Ports
----------------------------------------------------------------------
1 default Eth1/19, Eth1/20, Eth1/21, Eth1/22, Eth1/23,
Eth1/24, Eth1/25, Eth1/26, Eth1/27, Eth1/28,
Eth1/29, Eth1/30, Eth1/31, Eth1/32, Eth1/33,
Eth1/34, Eth1/35
10 Management Eth1/36
100 WAN 0 (Comcast 44th Avenue)Eth1/36
101 WAN 1 (Allo) Eth1/36
102 WAN 2 (unused) Eth1/36
103 WAN 3 (unused) Eth1/36
104 WAN 4 (unused) Eth1/36
105 WAN 5 (unused) Eth1/36
106 WAN 6 (unused) Eth1/36
107 WAN 7 (unused) Eth1/36
108 WAN 8 (unused) Eth1/36
109 WAN 9 (unused) Eth1/36
1000 Internal Services Eth1/36
1001 Internal clients Eth1/36
1002 Ceph front-end Eth1/36
1003 Ceph back-end Eth1/36
3900 Guest network Eth1/36
mellanox-sx6036-rack-1 [standalone: master] #
== Unlocking the bootloader ==
When reset or just powered on, the SX6036's bootloader does preset the human on the console a 5 second countdown timer and an opportunity to choose some other options than loading the currently active software image:
Mellanox MLNX-OS

Default image: 'PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc'
Press Enter to boot this image, or 'Ctrl B' for boot menu

Booting default image in: 0

Mellanox MLNX-OS Boot Menu:

* 1: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc
2: PPC_M460EX 3.6.8010 2018-08-20 18:04:16 ppc
u: USB menu (if USB device connected) (password required)
c: Command prompt (password required)

Choice:
Irritatingly, I have been unable to locate the password required for command prompt access. Fortunately, some kind denizens of the internet have shared a way to change (even remove!) this password from a running MLNX-OS. I found this information at https://forums.servethehome.com/index.php?threads/solved-mellanox-sx6012-u-boot-password-removal-without-bash-access.33484/

Let's record this for posterity:
Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-2 login: '''admin'''
Password:
Last login: Sun Apr 12 16:57:02 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-2 [standalone: master] > '''enable'''
mellanox-sx6036-rack-2 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-2 [standalone: master] (config) # '''boot bootmgr password 7 ""'''
mellanox-sx6036-rack-2 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-2 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-2 [standalone: master] # '''show bootvar'''

Installed images:
Partition 1:
version: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc

Partition 2:
version: PPC_M460EX 3.6.8010 2018-08-20 18:04:16 ppc

Last boot partition : 1
Next boot partition : 1
Serve image files via HTTP/HTTPS: no

No boot manager password is set.

Image signing : trusted signature always required
Admin require signed images: yes

Settings for next boot only:
Fallback reboot on configuration failure: yes (default)

mellanox-sx6036-rack-2 [standalone: master] #
Note the line saying "No boot manager password is set." A switch that has not had this treatment will report "Boot manager password is set." instead.

First try was not successful, though. I may continue poking at this again in the future.
=== Second attempt ===
Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-2 login: '''admin'''
Password:
Last login: Sun Apr 12 17:37:09 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-2 [standalone: master] > '''_shell'''
% Unrecognized command "_shell".
Type "?" for help.
mellanox-sx6036-rack-2 [standalone: master] > '''enable'''
mellanox-sx6036-rack-2 [standalone: master] # '''_shell'''
[admin@mellanox-sx6036-rack-2 ~]# '''stty rows 25 columns 160'''
[admin@mellanox-sx6036-rack-2 ~]# '''/opt/tms/bin/mddbreq /config/db/initial set modify - /system/bootmgr/password string ""'''
[admin@mellanox-sx6036-rack-2 ~]# '''eetool -a bf -s UBPASSWD=""'''
[admin@mellanox-sx6036-rack-2 ~]# '''exit'''
logo

Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-2 login: '''admin'''
Password:
Last login: Sun Apr 12 17:56:18 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-2 [standalone: master] > '''enable'''
mellanox-sx6036-rack-2 [standalone: master] # '''show bootvar'''

Installed images:
Partition 1:
version: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc

Partition 2:
version: PPC_M460EX 3.6.8010 2018-08-20 18:04:16 ppc

Last boot partition : 1
Next boot partition : 1
Serve image files via HTTP/HTTPS: no

No boot manager password is set.

Image signing : trusted signature always required
Admin require signed images: yes

Settings for next boot only:
Fallback reboot on configuration failure: yes (default)

mellanox-sx6036-rack-2 [standalone: master] #
And this time, success!
Mellanox MLNX-OS

Default image: 'PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc'
Press Enter to boot this image, or 'Ctrl B' for boot menu

Booting default image in: 0

Mellanox MLNX-OS Boot Menu:

* 1: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc
2: PPC_M460EX 3.6.8010 2018-08-20 18:04:16 ppc
u: USB menu (if USB device connected)
c: Command prompt

Choice: '''c'''
Entering command prompt
=> '''?'''
? - alias for 'help'
askenv - get environment variables from stdin
autoscr - run script from memory
base - print or set address offset
bdinfo - print Board Info structure
boot - boot default, i.e., run 'bootcmd'
bootd - boot default, i.e., run 'bootcmd'
bootelf - Boot from an ELF image in memory
bootm - boot application image from memory
bootmenu- Run boot menu
bootp - boot image via network using BOOTP/TFTP protocol
bootstrap - program the I2C bootstrap EEPROM
bootvx - Boot vxWorks from an ELF image
cmp - memory compare
coninfo - print console devices and information
cp - memory copy
crc32 - checksum calculation
dcache - enable or disable data cache
dhcp - boot image via network using DHCP/TFTP protocol
echo - echo args to console
eeprom - EEPROM sub-system
envreset- Reset the environment to the defaults
erase - erase FLASH memory
exit - exit script
ext2load- load binary file from a Ext2 filesystem
ext2ls - list files in a directory (default /)
fatinfo - print information about filesystem
fatload - load binary file from a dos filesystem
fatls - list files in a directory (default /)
fdt - flattened device tree utility commands
flinfo - print FLASH memory information
getdcr - Get an AMCC PPC 4xx DCR's value
getidcr - Get a register value via indirect DCR addressing
go - start application at address 'addr'
help - print online help
icache - enable or disable instruction cache
icrc32 - checksum calculation
iloop - infinite loop on address range
imd - i2c memory display
iminfo - print header information for application image
imls - list all images found in flash
imm - i2c memory modify (auto-incrementing)
imw - memory write (fill)
imxtract- extract a part of a multi-image
inm - memory modify (constant address)
interrupts - enable or disable interrupts
iprobe - probe to discover valid I2C chip addresses
irqinfo - print information about IRQs
itest - return true/false on integer compare
loadb - load binary file over serial line (kermit mode)
loads - load S-Record file over serial line
loady - load binary file over serial line (ymodem mode)
loop - infinite loop on address range
loopw - infinite write loop on address range
md - memory display
mdc - memory display cyclic
mii - MII utility commands
mm - memory modify (auto-incrementing)
mtest - simple RAM test
mw - memory write (fill)
mwc - memory write cyclic
nand - NAND sub-system
nboot - boot from NAND device
nfs - boot image via network using NFS protocol
nm - memory modify (constant address)
pci - list and access PCI Configuration Space
ping - send ICMP ECHO_REQUEST to network host
printenv- print environment variables
protect - enable or disable FLASH write protection
rarpboot- boot image via network using RARP/TFTP protocol
reginfo - print register information
reset - Perform RESET of the CPU
run - run commands in an environment variable
saveenv - save environment variables to persistent storage
setdcr - Set an AMCC PPC 4xx DCR's value
setenv - set environment variables
setexpr - set environment variable as the result of eval expression
setidcr - Set a register value via indirect DCR addressing
showvar- print local hushshell variables
sleep - delay execution for some time
test - minimal test like /bin/sh
tftpboot- boot image via network using TFTP protocol
usb - USB sub-system
usbboot - boot from USB device
version - print monitor version
=> '''version'''

U-Boot 2009.01 SX_PPC_M460EX SX_3.2.0330-82 ppc (Dec 20 2012 - 17:53:54)
=> '''reset'''
Look! It runs U-Boot. This should not be a huge surprise. Enjoy poking around inside the pre-Linux environment if you like.

Mellanox SX6036

2026-04-12T17:35:16Z

Adj: /* Unlocking the bootloader */

Mellanox SX6036

2026-04-12T17:32:05Z

Adj: /* Unlocking the bootloader */

== Console access ==
=== Serial port ===
There is an 8P8C modular (RJ45) connector on the right side of the port side of the switch. This connector uses the Cisco pinout and a common Cisco console cable works great for driving it.

Serial settings are 9600 bits per second, 8 data bits, no parity bits, 1 stop bit (9600 8N1)

=== Default credentials ===
User '''<code>admin</code>'''.

Password '''<code>admin</code>'''.

=== Resetting the password ===
In the lower left corner of the port side of the switch, there is a small hole labelled "RST". Holding the button inside with a paperclip or similar tool for 15 seconds will reset the switch and erase any password set for the '''<code>admin</code>''' user.

== Firmware updates ==
Updating firmware on these switches is a protracted process. Firmware images are available on the public internet, but some searching is required. To bring a switch from a 3.2.''x'' release up to the final 3.6.8012 release, locate the following images on the web and save them to an SSHable or HTTP(S)able location near you:
* image-PPC_M460EX-3.4.3002.img
* image-PPC_M460EX-3.5.1006.img
* image-PPC_M460EX-3.6.1002.img
* image-PPC_M460EX-3.6.8012.img
* image-PPC_M460EX-SX_3.3.3400.img
* image-PPC_M460EX-SX_3.3.5006.img
* image-PPC_M460EX-SX_3.4.0012.img

=== New firmware image flashing procedure ===
This is a privileged operation, so first off gain elevated privileges...

'''<code>enable</code>'''

'''<code>image fetch http://172.17.0.17/~adj/image-PPC_M460EX-SX_3.3.3400.img</code>'''

'''<code>image install image-PPC_M460EX-SX_3.3.3400.img location 2 progress track verify require-sig</code>'''

'''<code>reload</code>'''

Wait for U-Boot to prompt about a menu, and send a Ctrl-B before the countdown reaches 0. Select option 2 (no need for a newline) to boot the newly flashed image. Iterate through each of the newer images, alternating location 1 and location 2 in the '''<code>image install ...</code>''' command.

When updates version installations are finally complete, it might be good to ensure that 3.6.8012 is flashed to both locations 1 and 2.

== Useful CLI commands ==

=== Find MAC address of 1000baseT management interface 0 ===
Good for setting up the DHCP server's config for this switch. For some reason, this command needs elevated privileges. Grrr.

'''<code>enable</code>'''

'''<code>show interfaces mgmt0 | include HW</code>'''

=== Find system serial number without reading the physical tag ===
This is useful for license key generation. And inventory purposes.

'''<code>show inventory</code>'''

=== Find system host ID ===
Also can be used for license key generation. Looks suspiciously like the mgmt0 MAC address...

'''<code>show version | include Host</code>'''

== Licensing features ==
Mellanox's SwitchX2 ASIC is capable of performing Infiniband '''and''' Ethernet switching. Mellanox the company likes to lock some of these features behind licensing keys. Which is annoying. But with enough effort, is a surmountable obstacle

=== The license generation tool ===
Inside the MLNX-OS distribution images (those files with names like '''<code>image-PPC_M460EX-3.6.8012.img</code>''') there exists a gzipped tarball containing the MLNX-OS filesystem. Inside the tarball is a directory called '''<code>/opt/tms/bin</code>'''. And inside ''that'' directory, one can find the license key generation program, '''<code>genlicense</code>'''. The MLNX-OS filesystem image is a PowerPC Linux distribution that can be run from a Qemu emulated big-endian PowerPC machine, or natively on big-endian PowerPC hardware. We can use this to generate feature licenses until the cows come home.

=== Our first license, shell access from the switch CLI ===
There is a hidden CLI command, '''<code>_shell</code>''' which will cause the CLI to '''<code>execve("/bin/bash", ...)</code>'''. This command is privileged (must be run from enable mode) and locked behind a license key. The following is a working CLI session to install a key for this feature:
Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-1 login: '''admin'''
Password: '''passwordgoesherebutisnotechoed'''
Last login: Sun Feb 19 23:25:30 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal '''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''license install LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''show licenses'''
License 1: LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1
Feature: RESTRICTED_CMDS_GEN2
Description: Access to restricted system functionality
Valid: yes
Active: yes
mellanox-sx6036-rack-1 [standalone: master] #

=== Ethernet features: Ethernet, Ethernet L2, and Ethernet L3 ===
We will generate these from the switch as we already have shell access and can use the included genlicense tool.

Running the following from the switch's shell will print a key for a non-hardware-locked, non-date-locked license to activate the switch's full Ethernet capabilities:
[admin@mellanox-sx6036-rack-0 ~]# '''cd /opt/tms/bin'''
[admin@mellanox-sx6036-rack-0 bin]# '''./genlicense 2 EFM_SX m2l0n%0x9 -o 53 true -o 51 true -o 58 true'''
LK2-EFM_SX-5M11-5K11-5T11-88A1-BBD0-JP82-X
[admin@mellanox-sx6036-rack-0 bin]#

A quick explanatory diversion is in order here. There are 2 types of keys. genlicense's first option above specifies key type 2. '''<code>EFM_SX</code>''' is the '''<code>Generic SX license</code>''' type. By itself it does nothing, but all of the '''<code>-o</code>''' flags specify further features to be activated. A complete list of these can be generated by running '''<code>/opt/tms/bin/genlicense 2</code>'''. There are 14 of these additional options, and their meaning is not well documented. The magic word '''<code>m2l0n%0x9</code>''' can be found by watching for '''<code>strlen()</code>''' calls in the process flow of '''<code>/opt/tms/bin/dumplicense</code>''' when run under '''<code>ltrace</code>'''.

And it can be installed in the switch's configuration database by leaving the shell (which will disconnect the terminal session), logging in again, running '''<code>enable</code>''' to get elevated privileges, and using the '''<code>license install</code>''' configuration command like so:
Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-1 login: '''admin'''
Password:
Last login: Sun Feb 19 23:31:51 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''license install LK2-EFM_SX-5M11-5K11-5T11-88A1-BBD0-JP82-X'''
License was installed successfully. Please wait 1 minute before further configurations.
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] #

== Switch configuration commands ==
Some of these things will need licenses first.

=== I can haz Ethernet? ===
Install the Ethernet key as shown above and confirm it is working:
mellanox-sx6036-rack-1 [standalone: master] # '''show system capabilities'''
IB: Supported, L2, Adaptive Routing
Ethernet: Supported, L2, L3
GW: Not supported
Max SM nodes: 648
IB Max licensed speed: FDR10
Ethernet Max licensed speed: 56Gb
mellanox-sx6036-rack-1 [standalone: master] #
It says we can do Ethernet. Which is cool. But it does not have the Infiniband-Ethernet gateway feature licensed. And this is required to enable the switch to run in VPI mode. VPI is "Virtual Protocol Interconnect" or Mellanox's way of saying "We can do Infiniband ''and'' Ethernet on a single switch." So another license is needed. Here's a key for that:
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''license install LK2-EFM_SX-5N21-488A-182A-UQXB-Y6'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''show system capabilities '''
IB: Supported, L2, Adaptive Routing
Ethernet: Supported, L2, L3
GW: Supported
Max SM nodes: 648
IB Max licensed speed: FDR10
Ethernet Max licensed speed: 56Gb
mellanox-sx6036-rack-1 [standalone: master] #
We will need to do this next thing to convert the switch from an Infiniband-only switch to an Infiniband-and-Ethernet switch:
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''system profile vpi-single-switch'''
Warning - confirming will cause system reboot and all configuration will be deleted
Type 'yes' to confirm profile change: '''yes'''
Allow time for the switch to reboot and log in again. When it next starts up, run the command '''<code>show system profile</code>''' to confirm that it is in VPI mode:
mellanox-sx6036-rack-1 [standalone: master] > '''show system profile'''

Profile: vpi-single-switch

mellanox-sx6036-rack-1 [standalone: master] >
And finally, let's set some ports to Ethernet mode:
mellanox-sx6036-rack-1 [standalone: master] > '''show ports type'''
InfiniBand: 1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10 1/11 1/12 1/13 1/14 1/15 1/16 1/17 1/18 1/19 1/20 1/21 1/22 1/23 1/24 1/25 1/26 1/27 1/28 1/29 1/30 1/31 1/32 1/33 1/34 1/35 1/36
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface ib 1/19-1/36'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ib 1/19-1/36) # '''shutdown'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ib 1/19-1/36) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''port 1/19-1/36 type ethernet'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface ethernet 1/19-1/36'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/19-1/36) # '''no shutdown'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/19-1/36) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show ports type'''
Ethernet: 1/19 1/20 1/21 1/22 1/23 1/24 1/25 1/26 1/27 1/28 1/29 1/30 1/31 1/32 1/33 1/34 1/35 1/36
InfiniBand: 1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10 1/11 1/12 1/13 1/14 1/15 1/16 1/17 1/18
mellanox-sx6036-rack-1 [standalone: master] #

=== Switch management items ===
==== Password for the admin user ====
Factory defaults are probably not a good idea, so let's tighten things up a bit.
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''username admin password ''the_password_goes_here'' '''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] #

==== IP addressing for the management interface ====
The following will configure the switch to retrieve its IP address, subnet mask, router IP address, name server info, and hostname from DHCP server on the management interface's local network.
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface mgmt0'''
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # '''dhcp'''
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # '''dhcp hostname'''
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] #

==== NTP ====
Let's add a couple of NTP servers to the config and discipline our poorly behaved battery backed clock:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''ntp server 172.16.10.2'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''ntp server 172.17.0.17'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show ntp'''

NTP is administratively : enabled
NTP Authentication administratively: disabled

Clock is synchronized:
Reference: 172.16.10.2
Offset : -0.006 ms

Active servers and peers:
172.16.10.2:
Conf Type : serv
Status : sys.peer(*)
Stratum : 3
Offset(msec) : -0.006
Ref clock : 45.45.184.14
Poll Interval (sec): 64
Last Response (sec): 8
Auth state : none

172.17.0.17:
Conf Type : serv
Status : pending
Stratum : 2
Offset(msec) : 0.429
Ref clock : 198.60.22.240
Poll Interval (sec): 64
Last Response (sec): 7
Auth state : none

mellanox-sx6036-rack-1 [standalone: master] #
=== Ethernet goodies ===
==== LLDP ====
Turn it on and test it out like so:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''lldp'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show lldp local'''
LLDP: enabled

Local global configuration
Chassis sub type: Mac Address
Chassis id: 00:02:c9:6d:4d:b0
System Name: mellanox-sx6036-rack-1
System Description: SX6036,MLNX-OS,SWv3.6.8012
Supported capabilities: B,R
Supported capabilities enabled: B
mellanox-sx6036-rack-1 [standalone: master] # '''show lldp interfaces ethernet 1/36'''
TLV flags
PD: port-description
SN: sys-name
SD: sys-description
SC: sys-capabilities
MA: management-address
ETS-C: ETS-Configuration
ETS-R: ETS-Recommendation
AP: Application Priority
PFC: Priority Flow Control
CEE: Converged Enhanced Ethernet DCBX version
MED-CAP: Media Capabilities
MED-NWP: MED-Network Policy

-----------------------------------------------------------------------------------
Interface Receive Transmit TLVs
-----------------------------------------------------------------------------------
Eth1/36 Enabled Enabled PD, SN, SD, SC, PFC, AP, ETS-C, ETS-R
mellanox-sx6036-rack-1 [standalone: master] # '''show lldp interfaces ethernet 1/36 remote'''

Eth1/36

Remote Index: 1
Remote chassis id: cc:4e:24:83:4b:36 ; chassis id subtype: Mac Address (4)
Remote port-id: cc:4e:24:83:4b:6c ; port id subtype: Mac Address (3)
Remote port description: 40GigabitEthernet1/2/6
Remote system name: brocade-icx6610-48p-rack-1
Remote system description: Not Advertised
Remote system capabilities supported: B,R ; enabled B,R
Remote Management Addresses:
SubType Address
------------------------------------
IPv4 172.16.10.17

No Remote PFC entry

No Remote ETS entry

No Remote Application Priority entry

mellanox-sx6036-rack-1 [standalone: master] #
And enjoy getting to know all the things about your neighbors.
==== Multiple Spanning Tree Protocol ====
Wherein we avoid broadcast loops. Because they are bad, even if they do make for lots of blinken lights.
Our Brocade switches are running MSTP and, fortunately, MLNX-OS supports that on our SX6036es as well.
Setting the switch to MSTP mode is pretty simple:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''spanning-tree mode mst'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show spanning-tree'''

Switch : ethernet-default
Spanning tree protocol mst : enabled
Spanning tree force version: 3

Root ID:
Priority : 32768
Address : 00:02:c9:6d:4c:b0
Cost : 0
Port : Eth1/36
Hello Time (sec) : 2
Max Age (sec) : 20
Forward Delay (sec): 15

MST00:
Bridge is executing the mstp compatible Spanning Tree Protocol

Bridge ID:
Priority : 32768
Address : 00:02:c9:6d:4d:b0
Hello Time (sec) : 2
Max Age (sec) : 20
Forward Delay (sec): 15

L: Loop Inconsistent
R: Root Inconsistent
G: BPDU Guard Inconsistent

--------------------------------------------------------------------------
Interface Role Sts Cost Prio Type
--------------------------------------------------------------------------
Eth1/36 Root Forwarding 500 128 normal

mellanox-sx6036-rack-1 [standalone: master] #
==== VLANs ====
Create our standard assortment of VLANs like so:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 10 name Management'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 10) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 100 name "WAN 0 (Comcast 44th Avenue)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 100) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 101 name "WAN 1 (Allo)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 101) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 102 name "WAN 2 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 102) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 103 name "WAN 3 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 103) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 104 name "WAN 4 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 104) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 105 name "WAN 5 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 105) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 106 name "WAN 6 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 106) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 107 '''name "WAN 7 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 107) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 108 name "WAN 8 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 108) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 109 name "WAN 9 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 109) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1000 name "Internal Services"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1000) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1001 name "Internal clients"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1001) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1002 name "Ceph front-end"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1002) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1003 name "Ceph back-end"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1003) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 3900 name "Guest network"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 3900) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show vlan'''
----------------------------------------------------------------------
VLAN Name Ports
----------------------------------------------------------------------
1 default Eth1/19, Eth1/20, Eth1/21, Eth1/22, Eth1/23,
Eth1/24, Eth1/25, Eth1/26, Eth1/27, Eth1/28,
Eth1/29, Eth1/30, Eth1/31, Eth1/32, Eth1/33,
Eth1/34, Eth1/35, Eth1/36
10 Management
100 WAN 0 (Comcast 44th Avenue)
101 WAN 1 (Allo)
102 WAN 2 (unused)
103 WAN 3 (unused)
104 WAN 4 (unused)
105 WAN 5 (unused)
106 WAN 6 (unused)
107 WAN 7 (unused)
108 WAN 8 (unused)
109 WAN 9 (unused)
1000 Internal Services
1001 Internal clients
1002 Ceph front-end
1003 Ceph back-end
3900 Guest network
mellanox-sx6036-rack-1 [standalone: master] #
And now, to assign all those VLANs as tagged on port 1/36:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface ethernet 1/36'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport mode trunk'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan 10'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan add 100-109'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan add 1000-1003'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan add 3900'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show vlan'''
----------------------------------------------------------------------
VLAN Name Ports
----------------------------------------------------------------------
1 default Eth1/19, Eth1/20, Eth1/21, Eth1/22, Eth1/23,
Eth1/24, Eth1/25, Eth1/26, Eth1/27, Eth1/28,
Eth1/29, Eth1/30, Eth1/31, Eth1/32, Eth1/33,
Eth1/34, Eth1/35
10 Management Eth1/36
100 WAN 0 (Comcast 44th Avenue)Eth1/36
101 WAN 1 (Allo) Eth1/36
102 WAN 2 (unused) Eth1/36
103 WAN 3 (unused) Eth1/36
104 WAN 4 (unused) Eth1/36
105 WAN 5 (unused) Eth1/36
106 WAN 6 (unused) Eth1/36
107 WAN 7 (unused) Eth1/36
108 WAN 8 (unused) Eth1/36
109 WAN 9 (unused) Eth1/36
1000 Internal Services Eth1/36
1001 Internal clients Eth1/36
1002 Ceph front-end Eth1/36
1003 Ceph back-end Eth1/36
3900 Guest network Eth1/36
mellanox-sx6036-rack-1 [standalone: master] #
== Unlocking the bootloader ==
When reset or just powered on, the SX6036's bootloader does preset the human on the console a 5 second countdown timer and an opportunity to choose some other options than loading the currently active software image:
Mellanox MLNX-OS

Default image: 'PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc'
Press Enter to boot this image, or 'Ctrl B' for boot menu

Booting default image in: 0

Mellanox MLNX-OS Boot Menu:

* 1: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc
2: PPC_M460EX 3.6.8010 2018-08-20 18:04:16 ppc
u: USB menu (if USB device connected) (password required)
c: Command prompt (password required)

Choice:
Irritatingly, I have been unable to locate the password required for command prompt access. Fortunately, some kind denizens of the internet have shared a way to change (even remove!) this password from a running MLNX-OS. I found this information at https://forums.servethehome.com/index.php?threads/solved-mellanox-sx6012-u-boot-password-removal-without-bash-access.33484/

Let's record this for posterity:
Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-2 login: admin
Password:
Last login: Sun Apr 12 16:57:02 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-2 [standalone: master] > '''enable'''
mellanox-sx6036-rack-2 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-2 [standalone: master] (config) # '''boot bootmgr password 7 ""'''
mellanox-sx6036-rack-2 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-2 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-2 [standalone: master] # '''show bootvar'''

Installed images:
Partition 1:
version: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc

Partition 2:
version: PPC_M460EX 3.6.8010 2018-08-20 18:04:16 ppc

Last boot partition : 1
Next boot partition : 1
Serve image files via HTTP/HTTPS: no

No boot manager password is set.

Image signing : trusted signature always required
Admin require signed images: yes

Settings for next boot only:
Fallback reboot on configuration failure: yes (default)

mellanox-sx6036-rack-2 [standalone: master] #
Note the line saying "No boot manager password is set." A switch that has not had this treatment will report "Boot manager password is set." instead.

First try was not successful, though. I may continue poking at this again in the future.

Mellanox SX6036

2026-04-12T17:29:10Z

Adj: /* Unlocking the bootloader */

== Console access ==
=== Serial port ===
There is an 8P8C modular (RJ45) connector on the right side of the port side of the switch. This connector uses the Cisco pinout and a common Cisco console cable works great for driving it.

Serial settings are 9600 bits per second, 8 data bits, no parity bits, 1 stop bit (9600 8N1)

=== Default credentials ===
User '''<code>admin</code>'''.

Password '''<code>admin</code>'''.

=== Resetting the password ===
In the lower left corner of the port side of the switch, there is a small hole labelled "RST". Holding the button inside with a paperclip or similar tool for 15 seconds will reset the switch and erase any password set for the '''<code>admin</code>''' user.

== Firmware updates ==
Updating firmware on these switches is a protracted process. Firmware images are available on the public internet, but some searching is required. To bring a switch from a 3.2.''x'' release up to the final 3.6.8012 release, locate the following images on the web and save them to an SSHable or HTTP(S)able location near you:
* image-PPC_M460EX-3.4.3002.img
* image-PPC_M460EX-3.5.1006.img
* image-PPC_M460EX-3.6.1002.img
* image-PPC_M460EX-3.6.8012.img
* image-PPC_M460EX-SX_3.3.3400.img
* image-PPC_M460EX-SX_3.3.5006.img
* image-PPC_M460EX-SX_3.4.0012.img

=== New firmware image flashing procedure ===
This is a privileged operation, so first off gain elevated privileges...

'''<code>enable</code>'''

'''<code>image fetch http://172.17.0.17/~adj/image-PPC_M460EX-SX_3.3.3400.img</code>'''

'''<code>image install image-PPC_M460EX-SX_3.3.3400.img location 2 progress track verify require-sig</code>'''

'''<code>reload</code>'''

Wait for U-Boot to prompt about a menu, and send a Ctrl-B before the countdown reaches 0. Select option 2 (no need for a newline) to boot the newly flashed image. Iterate through each of the newer images, alternating location 1 and location 2 in the '''<code>image install ...</code>''' command.

When updates version installations are finally complete, it might be good to ensure that 3.6.8012 is flashed to both locations 1 and 2.

== Useful CLI commands ==

=== Find MAC address of 1000baseT management interface 0 ===
Good for setting up the DHCP server's config for this switch. For some reason, this command needs elevated privileges. Grrr.

'''<code>enable</code>'''

'''<code>show interfaces mgmt0 | include HW</code>'''

=== Find system serial number without reading the physical tag ===
This is useful for license key generation. And inventory purposes.

'''<code>show inventory</code>'''

=== Find system host ID ===
Also can be used for license key generation. Looks suspiciously like the mgmt0 MAC address...

'''<code>show version | include Host</code>'''

== Licensing features ==
Mellanox's SwitchX2 ASIC is capable of performing Infiniband '''and''' Ethernet switching. Mellanox the company likes to lock some of these features behind licensing keys. Which is annoying. But with enough effort, is a surmountable obstacle

=== The license generation tool ===
Inside the MLNX-OS distribution images (those files with names like '''<code>image-PPC_M460EX-3.6.8012.img</code>''') there exists a gzipped tarball containing the MLNX-OS filesystem. Inside the tarball is a directory called '''<code>/opt/tms/bin</code>'''. And inside ''that'' directory, one can find the license key generation program, '''<code>genlicense</code>'''. The MLNX-OS filesystem image is a PowerPC Linux distribution that can be run from a Qemu emulated big-endian PowerPC machine, or natively on big-endian PowerPC hardware. We can use this to generate feature licenses until the cows come home.

=== Our first license, shell access from the switch CLI ===
There is a hidden CLI command, '''<code>_shell</code>''' which will cause the CLI to '''<code>execve("/bin/bash", ...)</code>'''. This command is privileged (must be run from enable mode) and locked behind a license key. The following is a working CLI session to install a key for this feature:
Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-1 login: '''admin'''
Password: '''passwordgoesherebutisnotechoed'''
Last login: Sun Feb 19 23:25:30 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal '''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''license install LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''show licenses'''
License 1: LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1
Feature: RESTRICTED_CMDS_GEN2
Description: Access to restricted system functionality
Valid: yes
Active: yes
mellanox-sx6036-rack-1 [standalone: master] #

=== Ethernet features: Ethernet, Ethernet L2, and Ethernet L3 ===
We will generate these from the switch as we already have shell access and can use the included genlicense tool.

Running the following from the switch's shell will print a key for a non-hardware-locked, non-date-locked license to activate the switch's full Ethernet capabilities:
[admin@mellanox-sx6036-rack-0 ~]# '''cd /opt/tms/bin'''
[admin@mellanox-sx6036-rack-0 bin]# '''./genlicense 2 EFM_SX m2l0n%0x9 -o 53 true -o 51 true -o 58 true'''
LK2-EFM_SX-5M11-5K11-5T11-88A1-BBD0-JP82-X
[admin@mellanox-sx6036-rack-0 bin]#

A quick explanatory diversion is in order here. There are 2 types of keys. genlicense's first option above specifies key type 2. '''<code>EFM_SX</code>''' is the '''<code>Generic SX license</code>''' type. By itself it does nothing, but all of the '''<code>-o</code>''' flags specify further features to be activated. A complete list of these can be generated by running '''<code>/opt/tms/bin/genlicense 2</code>'''. There are 14 of these additional options, and their meaning is not well documented. The magic word '''<code>m2l0n%0x9</code>''' can be found by watching for '''<code>strlen()</code>''' calls in the process flow of '''<code>/opt/tms/bin/dumplicense</code>''' when run under '''<code>ltrace</code>'''.

And it can be installed in the switch's configuration database by leaving the shell (which will disconnect the terminal session), logging in again, running '''<code>enable</code>''' to get elevated privileges, and using the '''<code>license install</code>''' configuration command like so:
Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-1 login: '''admin'''
Password:
Last login: Sun Feb 19 23:31:51 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''license install LK2-EFM_SX-5M11-5K11-5T11-88A1-BBD0-JP82-X'''
License was installed successfully. Please wait 1 minute before further configurations.
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] #

== Switch configuration commands ==
Some of these things will need licenses first.

=== I can haz Ethernet? ===
Install the Ethernet key as shown above and confirm it is working:
mellanox-sx6036-rack-1 [standalone: master] # '''show system capabilities'''
IB: Supported, L2, Adaptive Routing
Ethernet: Supported, L2, L3
GW: Not supported
Max SM nodes: 648
IB Max licensed speed: FDR10
Ethernet Max licensed speed: 56Gb
mellanox-sx6036-rack-1 [standalone: master] #
It says we can do Ethernet. Which is cool. But it does not have the Infiniband-Ethernet gateway feature licensed. And this is required to enable the switch to run in VPI mode. VPI is "Virtual Protocol Interconnect" or Mellanox's way of saying "We can do Infiniband ''and'' Ethernet on a single switch." So another license is needed. Here's a key for that:
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''license install LK2-EFM_SX-5N21-488A-182A-UQXB-Y6'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''show system capabilities '''
IB: Supported, L2, Adaptive Routing
Ethernet: Supported, L2, L3
GW: Supported
Max SM nodes: 648
IB Max licensed speed: FDR10
Ethernet Max licensed speed: 56Gb
mellanox-sx6036-rack-1 [standalone: master] #
We will need to do this next thing to convert the switch from an Infiniband-only switch to an Infiniband-and-Ethernet switch:
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''system profile vpi-single-switch'''
Warning - confirming will cause system reboot and all configuration will be deleted
Type 'yes' to confirm profile change: '''yes'''
Allow time for the switch to reboot and log in again. When it next starts up, run the command '''<code>show system profile</code>''' to confirm that it is in VPI mode:
mellanox-sx6036-rack-1 [standalone: master] > '''show system profile'''

Profile: vpi-single-switch

mellanox-sx6036-rack-1 [standalone: master] >
And finally, let's set some ports to Ethernet mode:
mellanox-sx6036-rack-1 [standalone: master] > '''show ports type'''
InfiniBand: 1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10 1/11 1/12 1/13 1/14 1/15 1/16 1/17 1/18 1/19 1/20 1/21 1/22 1/23 1/24 1/25 1/26 1/27 1/28 1/29 1/30 1/31 1/32 1/33 1/34 1/35 1/36
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface ib 1/19-1/36'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ib 1/19-1/36) # '''shutdown'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ib 1/19-1/36) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''port 1/19-1/36 type ethernet'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface ethernet 1/19-1/36'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/19-1/36) # '''no shutdown'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/19-1/36) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show ports type'''
Ethernet: 1/19 1/20 1/21 1/22 1/23 1/24 1/25 1/26 1/27 1/28 1/29 1/30 1/31 1/32 1/33 1/34 1/35 1/36
InfiniBand: 1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10 1/11 1/12 1/13 1/14 1/15 1/16 1/17 1/18
mellanox-sx6036-rack-1 [standalone: master] #

=== Switch management items ===
==== Password for the admin user ====
Factory defaults are probably not a good idea, so let's tighten things up a bit.
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''username admin password ''the_password_goes_here'' '''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] #

==== IP addressing for the management interface ====
The following will configure the switch to retrieve its IP address, subnet mask, router IP address, name server info, and hostname from DHCP server on the management interface's local network.
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface mgmt0'''
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # '''dhcp'''
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # '''dhcp hostname'''
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] #

==== NTP ====
Let's add a couple of NTP servers to the config and discipline our poorly behaved battery backed clock:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''ntp server 172.16.10.2'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''ntp server 172.17.0.17'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show ntp'''

NTP is administratively : enabled
NTP Authentication administratively: disabled

Clock is synchronized:
Reference: 172.16.10.2
Offset : -0.006 ms

Active servers and peers:
172.16.10.2:
Conf Type : serv
Status : sys.peer(*)
Stratum : 3
Offset(msec) : -0.006
Ref clock : 45.45.184.14
Poll Interval (sec): 64
Last Response (sec): 8
Auth state : none

172.17.0.17:
Conf Type : serv
Status : pending
Stratum : 2
Offset(msec) : 0.429
Ref clock : 198.60.22.240
Poll Interval (sec): 64
Last Response (sec): 7
Auth state : none

mellanox-sx6036-rack-1 [standalone: master] #
=== Ethernet goodies ===
==== LLDP ====
Turn it on and test it out like so:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''lldp'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show lldp local'''
LLDP: enabled

Local global configuration
Chassis sub type: Mac Address
Chassis id: 00:02:c9:6d:4d:b0
System Name: mellanox-sx6036-rack-1
System Description: SX6036,MLNX-OS,SWv3.6.8012
Supported capabilities: B,R
Supported capabilities enabled: B
mellanox-sx6036-rack-1 [standalone: master] # '''show lldp interfaces ethernet 1/36'''
TLV flags
PD: port-description
SN: sys-name
SD: sys-description
SC: sys-capabilities
MA: management-address
ETS-C: ETS-Configuration
ETS-R: ETS-Recommendation
AP: Application Priority
PFC: Priority Flow Control
CEE: Converged Enhanced Ethernet DCBX version
MED-CAP: Media Capabilities
MED-NWP: MED-Network Policy

-----------------------------------------------------------------------------------
Interface Receive Transmit TLVs
-----------------------------------------------------------------------------------
Eth1/36 Enabled Enabled PD, SN, SD, SC, PFC, AP, ETS-C, ETS-R
mellanox-sx6036-rack-1 [standalone: master] # '''show lldp interfaces ethernet 1/36 remote'''

Eth1/36

Remote Index: 1
Remote chassis id: cc:4e:24:83:4b:36 ; chassis id subtype: Mac Address (4)
Remote port-id: cc:4e:24:83:4b:6c ; port id subtype: Mac Address (3)
Remote port description: 40GigabitEthernet1/2/6
Remote system name: brocade-icx6610-48p-rack-1
Remote system description: Not Advertised
Remote system capabilities supported: B,R ; enabled B,R
Remote Management Addresses:
SubType Address
------------------------------------
IPv4 172.16.10.17

No Remote PFC entry

No Remote ETS entry

No Remote Application Priority entry

mellanox-sx6036-rack-1 [standalone: master] #
And enjoy getting to know all the things about your neighbors.
==== Multiple Spanning Tree Protocol ====
Wherein we avoid broadcast loops. Because they are bad, even if they do make for lots of blinken lights.
Our Brocade switches are running MSTP and, fortunately, MLNX-OS supports that on our SX6036es as well.
Setting the switch to MSTP mode is pretty simple:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''spanning-tree mode mst'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show spanning-tree'''

Switch : ethernet-default
Spanning tree protocol mst : enabled
Spanning tree force version: 3

Root ID:
Priority : 32768
Address : 00:02:c9:6d:4c:b0
Cost : 0
Port : Eth1/36
Hello Time (sec) : 2
Max Age (sec) : 20
Forward Delay (sec): 15

MST00:
Bridge is executing the mstp compatible Spanning Tree Protocol

Bridge ID:
Priority : 32768
Address : 00:02:c9:6d:4d:b0
Hello Time (sec) : 2
Max Age (sec) : 20
Forward Delay (sec): 15

L: Loop Inconsistent
R: Root Inconsistent
G: BPDU Guard Inconsistent

--------------------------------------------------------------------------
Interface Role Sts Cost Prio Type
--------------------------------------------------------------------------
Eth1/36 Root Forwarding 500 128 normal

mellanox-sx6036-rack-1 [standalone: master] #
==== VLANs ====
Create our standard assortment of VLANs like so:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 10 name Management'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 10) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 100 name "WAN 0 (Comcast 44th Avenue)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 100) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 101 name "WAN 1 (Allo)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 101) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 102 name "WAN 2 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 102) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 103 name "WAN 3 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 103) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 104 name "WAN 4 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 104) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 105 name "WAN 5 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 105) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 106 name "WAN 6 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 106) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 107 '''name "WAN 7 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 107) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 108 name "WAN 8 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 108) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 109 name "WAN 9 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 109) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1000 name "Internal Services"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1000) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1001 name "Internal clients"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1001) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1002 name "Ceph front-end"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1002) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1003 name "Ceph back-end"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1003) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 3900 name "Guest network"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 3900) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show vlan'''
----------------------------------------------------------------------
VLAN Name Ports
----------------------------------------------------------------------
1 default Eth1/19, Eth1/20, Eth1/21, Eth1/22, Eth1/23,
Eth1/24, Eth1/25, Eth1/26, Eth1/27, Eth1/28,
Eth1/29, Eth1/30, Eth1/31, Eth1/32, Eth1/33,
Eth1/34, Eth1/35, Eth1/36
10 Management
100 WAN 0 (Comcast 44th Avenue)
101 WAN 1 (Allo)
102 WAN 2 (unused)
103 WAN 3 (unused)
104 WAN 4 (unused)
105 WAN 5 (unused)
106 WAN 6 (unused)
107 WAN 7 (unused)
108 WAN 8 (unused)
109 WAN 9 (unused)
1000 Internal Services
1001 Internal clients
1002 Ceph front-end
1003 Ceph back-end
3900 Guest network
mellanox-sx6036-rack-1 [standalone: master] #
And now, to assign all those VLANs as tagged on port 1/36:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface ethernet 1/36'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport mode trunk'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan 10'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan add 100-109'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan add 1000-1003'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan add 3900'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show vlan'''
----------------------------------------------------------------------
VLAN Name Ports
----------------------------------------------------------------------
1 default Eth1/19, Eth1/20, Eth1/21, Eth1/22, Eth1/23,
Eth1/24, Eth1/25, Eth1/26, Eth1/27, Eth1/28,
Eth1/29, Eth1/30, Eth1/31, Eth1/32, Eth1/33,
Eth1/34, Eth1/35
10 Management Eth1/36
100 WAN 0 (Comcast 44th Avenue)Eth1/36
101 WAN 1 (Allo) Eth1/36
102 WAN 2 (unused) Eth1/36
103 WAN 3 (unused) Eth1/36
104 WAN 4 (unused) Eth1/36
105 WAN 5 (unused) Eth1/36
106 WAN 6 (unused) Eth1/36
107 WAN 7 (unused) Eth1/36
108 WAN 8 (unused) Eth1/36
109 WAN 9 (unused) Eth1/36
1000 Internal Services Eth1/36
1001 Internal clients Eth1/36
1002 Ceph front-end Eth1/36
1003 Ceph back-end Eth1/36
3900 Guest network Eth1/36
mellanox-sx6036-rack-1 [standalone: master] #
== Unlocking the bootloader ==
When reset or just powered on, the SX6036's bootloader does preset the human on the console a 5 second countdown timer and an opportunity to choose some other options than loading the currently active software image:
Mellanox MLNX-OS

Default image: 'PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc'
Press Enter to boot this image, or 'Ctrl B' for boot menu

Booting default image in: 0

Mellanox MLNX-OS Boot Menu:

* 1: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc
2: PPC_M460EX 3.6.8010 2018-08-20 18:04:16 ppc
u: USB menu (if USB device connected) (password required)
c: Command prompt (password required)

Choice:
Irritatingly, I have been unable to locate the password required for command prompt access. Fortunately, some kind denizens of the internet have shared a way to change (even remove!) this password from a running MLNX-OS. I found this information at https://forums.servethehome.com/index.php?threads/solved-mellanox-sx6012-u-boot-password-removal-without-bash-access.33484/

Let's record this for posterity:
Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-2 login: admin
Password:
Last login: Sun Apr 12 16:57:02 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-2 [standalone: master] > '''enable'''
mellanox-sx6036-rack-2 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-2 [standalone: master] (config) # '''boot bootmgr password 7 ""'''
mellanox-sx6036-rack-2 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-2 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-2 [standalone: master] # '''show bootvar'''

Installed images:
Partition 1:
version: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc

Partition 2:
version: PPC_M460EX 3.6.8010 2018-08-20 18:04:16 ppc

Last boot partition : 1
Next boot partition : 1
Serve image files via HTTP/HTTPS: no

No boot manager password is set.

Image signing : trusted signature always required
Admin require signed images: yes

Settings for next boot only:
Fallback reboot on configuration failure: yes (default)

mellanox-sx6036-rack-2 [standalone: master] #
Note the line saying "No boot manager password is set." A switch that has not had this treatment will report "Boot manager password is set." instead.

Mellanox SX6036

2026-04-12T17:27:45Z

Adj: /* VLANs */

== Console access ==
=== Serial port ===
There is an 8P8C modular (RJ45) connector on the right side of the port side of the switch. This connector uses the Cisco pinout and a common Cisco console cable works great for driving it.

Serial settings are 9600 bits per second, 8 data bits, no parity bits, 1 stop bit (9600 8N1)

=== Default credentials ===
User '''<code>admin</code>'''.

Password '''<code>admin</code>'''.

=== Resetting the password ===
In the lower left corner of the port side of the switch, there is a small hole labelled "RST". Holding the button inside with a paperclip or similar tool for 15 seconds will reset the switch and erase any password set for the '''<code>admin</code>''' user.

== Firmware updates ==
Updating firmware on these switches is a protracted process. Firmware images are available on the public internet, but some searching is required. To bring a switch from a 3.2.''x'' release up to the final 3.6.8012 release, locate the following images on the web and save them to an SSHable or HTTP(S)able location near you:
* image-PPC_M460EX-3.4.3002.img
* image-PPC_M460EX-3.5.1006.img
* image-PPC_M460EX-3.6.1002.img
* image-PPC_M460EX-3.6.8012.img
* image-PPC_M460EX-SX_3.3.3400.img
* image-PPC_M460EX-SX_3.3.5006.img
* image-PPC_M460EX-SX_3.4.0012.img

=== New firmware image flashing procedure ===
This is a privileged operation, so first off gain elevated privileges...

'''<code>enable</code>'''

'''<code>image fetch http://172.17.0.17/~adj/image-PPC_M460EX-SX_3.3.3400.img</code>'''

'''<code>image install image-PPC_M460EX-SX_3.3.3400.img location 2 progress track verify require-sig</code>'''

'''<code>reload</code>'''

Wait for U-Boot to prompt about a menu, and send a Ctrl-B before the countdown reaches 0. Select option 2 (no need for a newline) to boot the newly flashed image. Iterate through each of the newer images, alternating location 1 and location 2 in the '''<code>image install ...</code>''' command.

When updates version installations are finally complete, it might be good to ensure that 3.6.8012 is flashed to both locations 1 and 2.

== Useful CLI commands ==

=== Find MAC address of 1000baseT management interface 0 ===
Good for setting up the DHCP server's config for this switch. For some reason, this command needs elevated privileges. Grrr.

'''<code>enable</code>'''

'''<code>show interfaces mgmt0 | include HW</code>'''

=== Find system serial number without reading the physical tag ===
This is useful for license key generation. And inventory purposes.

'''<code>show inventory</code>'''

=== Find system host ID ===
Also can be used for license key generation. Looks suspiciously like the mgmt0 MAC address...

'''<code>show version | include Host</code>'''

== Licensing features ==
Mellanox's SwitchX2 ASIC is capable of performing Infiniband '''and''' Ethernet switching. Mellanox the company likes to lock some of these features behind licensing keys. Which is annoying. But with enough effort, is a surmountable obstacle

=== The license generation tool ===
Inside the MLNX-OS distribution images (those files with names like '''<code>image-PPC_M460EX-3.6.8012.img</code>''') there exists a gzipped tarball containing the MLNX-OS filesystem. Inside the tarball is a directory called '''<code>/opt/tms/bin</code>'''. And inside ''that'' directory, one can find the license key generation program, '''<code>genlicense</code>'''. The MLNX-OS filesystem image is a PowerPC Linux distribution that can be run from a Qemu emulated big-endian PowerPC machine, or natively on big-endian PowerPC hardware. We can use this to generate feature licenses until the cows come home.

=== Our first license, shell access from the switch CLI ===
There is a hidden CLI command, '''<code>_shell</code>''' which will cause the CLI to '''<code>execve("/bin/bash", ...)</code>'''. This command is privileged (must be run from enable mode) and locked behind a license key. The following is a working CLI session to install a key for this feature:
Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-1 login: '''admin'''
Password: '''passwordgoesherebutisnotechoed'''
Last login: Sun Feb 19 23:25:30 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal '''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''license install LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''show licenses'''
License 1: LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1
Feature: RESTRICTED_CMDS_GEN2
Description: Access to restricted system functionality
Valid: yes
Active: yes
mellanox-sx6036-rack-1 [standalone: master] #

=== Ethernet features: Ethernet, Ethernet L2, and Ethernet L3 ===
We will generate these from the switch as we already have shell access and can use the included genlicense tool.

Running the following from the switch's shell will print a key for a non-hardware-locked, non-date-locked license to activate the switch's full Ethernet capabilities:
[admin@mellanox-sx6036-rack-0 ~]# '''cd /opt/tms/bin'''
[admin@mellanox-sx6036-rack-0 bin]# '''./genlicense 2 EFM_SX m2l0n%0x9 -o 53 true -o 51 true -o 58 true'''
LK2-EFM_SX-5M11-5K11-5T11-88A1-BBD0-JP82-X
[admin@mellanox-sx6036-rack-0 bin]#

A quick explanatory diversion is in order here. There are 2 types of keys. genlicense's first option above specifies key type 2. '''<code>EFM_SX</code>''' is the '''<code>Generic SX license</code>''' type. By itself it does nothing, but all of the '''<code>-o</code>''' flags specify further features to be activated. A complete list of these can be generated by running '''<code>/opt/tms/bin/genlicense 2</code>'''. There are 14 of these additional options, and their meaning is not well documented. The magic word '''<code>m2l0n%0x9</code>''' can be found by watching for '''<code>strlen()</code>''' calls in the process flow of '''<code>/opt/tms/bin/dumplicense</code>''' when run under '''<code>ltrace</code>'''.

And it can be installed in the switch's configuration database by leaving the shell (which will disconnect the terminal session), logging in again, running '''<code>enable</code>''' to get elevated privileges, and using the '''<code>license install</code>''' configuration command like so:
Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-1 login: '''admin'''
Password:
Last login: Sun Feb 19 23:31:51 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''license install LK2-EFM_SX-5M11-5K11-5T11-88A1-BBD0-JP82-X'''
License was installed successfully. Please wait 1 minute before further configurations.
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] #

== Switch configuration commands ==
Some of these things will need licenses first.

=== I can haz Ethernet? ===
Install the Ethernet key as shown above and confirm it is working:
mellanox-sx6036-rack-1 [standalone: master] # '''show system capabilities'''
IB: Supported, L2, Adaptive Routing
Ethernet: Supported, L2, L3
GW: Not supported
Max SM nodes: 648
IB Max licensed speed: FDR10
Ethernet Max licensed speed: 56Gb
mellanox-sx6036-rack-1 [standalone: master] #
It says we can do Ethernet. Which is cool. But it does not have the Infiniband-Ethernet gateway feature licensed. And this is required to enable the switch to run in VPI mode. VPI is "Virtual Protocol Interconnect" or Mellanox's way of saying "We can do Infiniband ''and'' Ethernet on a single switch." So another license is needed. Here's a key for that:
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''license install LK2-EFM_SX-5N21-488A-182A-UQXB-Y6'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''show system capabilities '''
IB: Supported, L2, Adaptive Routing
Ethernet: Supported, L2, L3
GW: Supported
Max SM nodes: 648
IB Max licensed speed: FDR10
Ethernet Max licensed speed: 56Gb
mellanox-sx6036-rack-1 [standalone: master] #
We will need to do this next thing to convert the switch from an Infiniband-only switch to an Infiniband-and-Ethernet switch:
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''system profile vpi-single-switch'''
Warning - confirming will cause system reboot and all configuration will be deleted
Type 'yes' to confirm profile change: '''yes'''
Allow time for the switch to reboot and log in again. When it next starts up, run the command '''<code>show system profile</code>''' to confirm that it is in VPI mode:
mellanox-sx6036-rack-1 [standalone: master] > '''show system profile'''

Profile: vpi-single-switch

mellanox-sx6036-rack-1 [standalone: master] >
And finally, let's set some ports to Ethernet mode:
mellanox-sx6036-rack-1 [standalone: master] > '''show ports type'''
InfiniBand: 1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10 1/11 1/12 1/13 1/14 1/15 1/16 1/17 1/18 1/19 1/20 1/21 1/22 1/23 1/24 1/25 1/26 1/27 1/28 1/29 1/30 1/31 1/32 1/33 1/34 1/35 1/36
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface ib 1/19-1/36'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ib 1/19-1/36) # '''shutdown'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ib 1/19-1/36) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''port 1/19-1/36 type ethernet'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface ethernet 1/19-1/36'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/19-1/36) # '''no shutdown'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/19-1/36) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show ports type'''
Ethernet: 1/19 1/20 1/21 1/22 1/23 1/24 1/25 1/26 1/27 1/28 1/29 1/30 1/31 1/32 1/33 1/34 1/35 1/36
InfiniBand: 1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10 1/11 1/12 1/13 1/14 1/15 1/16 1/17 1/18
mellanox-sx6036-rack-1 [standalone: master] #

=== Switch management items ===
==== Password for the admin user ====
Factory defaults are probably not a good idea, so let's tighten things up a bit.
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''username admin password ''the_password_goes_here'' '''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] #

==== IP addressing for the management interface ====
The following will configure the switch to retrieve its IP address, subnet mask, router IP address, name server info, and hostname from DHCP server on the management interface's local network.
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface mgmt0'''
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # '''dhcp'''
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # '''dhcp hostname'''
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] #

==== NTP ====
Let's add a couple of NTP servers to the config and discipline our poorly behaved battery backed clock:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''ntp server 172.16.10.2'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''ntp server 172.17.0.17'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show ntp'''

NTP is administratively : enabled
NTP Authentication administratively: disabled

Clock is synchronized:
Reference: 172.16.10.2
Offset : -0.006 ms

Active servers and peers:
172.16.10.2:
Conf Type : serv
Status : sys.peer(*)
Stratum : 3
Offset(msec) : -0.006
Ref clock : 45.45.184.14
Poll Interval (sec): 64
Last Response (sec): 8
Auth state : none

172.17.0.17:
Conf Type : serv
Status : pending
Stratum : 2
Offset(msec) : 0.429
Ref clock : 198.60.22.240
Poll Interval (sec): 64
Last Response (sec): 7
Auth state : none

mellanox-sx6036-rack-1 [standalone: master] #
=== Ethernet goodies ===
==== LLDP ====
Turn it on and test it out like so:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''lldp'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show lldp local'''
LLDP: enabled

Local global configuration
Chassis sub type: Mac Address
Chassis id: 00:02:c9:6d:4d:b0
System Name: mellanox-sx6036-rack-1
System Description: SX6036,MLNX-OS,SWv3.6.8012
Supported capabilities: B,R
Supported capabilities enabled: B
mellanox-sx6036-rack-1 [standalone: master] # '''show lldp interfaces ethernet 1/36'''
TLV flags
PD: port-description
SN: sys-name
SD: sys-description
SC: sys-capabilities
MA: management-address
ETS-C: ETS-Configuration
ETS-R: ETS-Recommendation
AP: Application Priority
PFC: Priority Flow Control
CEE: Converged Enhanced Ethernet DCBX version
MED-CAP: Media Capabilities
MED-NWP: MED-Network Policy

-----------------------------------------------------------------------------------
Interface Receive Transmit TLVs
-----------------------------------------------------------------------------------
Eth1/36 Enabled Enabled PD, SN, SD, SC, PFC, AP, ETS-C, ETS-R
mellanox-sx6036-rack-1 [standalone: master] # '''show lldp interfaces ethernet 1/36 remote'''

Eth1/36

Remote Index: 1
Remote chassis id: cc:4e:24:83:4b:36 ; chassis id subtype: Mac Address (4)
Remote port-id: cc:4e:24:83:4b:6c ; port id subtype: Mac Address (3)
Remote port description: 40GigabitEthernet1/2/6
Remote system name: brocade-icx6610-48p-rack-1
Remote system description: Not Advertised
Remote system capabilities supported: B,R ; enabled B,R
Remote Management Addresses:
SubType Address
------------------------------------
IPv4 172.16.10.17

No Remote PFC entry

No Remote ETS entry

No Remote Application Priority entry

mellanox-sx6036-rack-1 [standalone: master] #
And enjoy getting to know all the things about your neighbors.
==== Multiple Spanning Tree Protocol ====
Wherein we avoid broadcast loops. Because they are bad, even if they do make for lots of blinken lights.
Our Brocade switches are running MSTP and, fortunately, MLNX-OS supports that on our SX6036es as well.
Setting the switch to MSTP mode is pretty simple:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''spanning-tree mode mst'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show spanning-tree'''

Switch : ethernet-default
Spanning tree protocol mst : enabled
Spanning tree force version: 3

Root ID:
Priority : 32768
Address : 00:02:c9:6d:4c:b0
Cost : 0
Port : Eth1/36
Hello Time (sec) : 2
Max Age (sec) : 20
Forward Delay (sec): 15

MST00:
Bridge is executing the mstp compatible Spanning Tree Protocol

Bridge ID:
Priority : 32768
Address : 00:02:c9:6d:4d:b0
Hello Time (sec) : 2
Max Age (sec) : 20
Forward Delay (sec): 15

L: Loop Inconsistent
R: Root Inconsistent
G: BPDU Guard Inconsistent

--------------------------------------------------------------------------
Interface Role Sts Cost Prio Type
--------------------------------------------------------------------------
Eth1/36 Root Forwarding 500 128 normal

mellanox-sx6036-rack-1 [standalone: master] #
==== VLANs ====
Create our standard assortment of VLANs like so:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 10 name Management'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 10) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 100 name "WAN 0 (Comcast 44th Avenue)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 100) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 101 name "WAN 1 (Allo)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 101) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 102 name "WAN 2 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 102) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 103 name "WAN 3 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 103) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 104 name "WAN 4 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 104) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 105 name "WAN 5 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 105) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 106 name "WAN 6 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 106) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 107 '''name "WAN 7 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 107) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 108 name "WAN 8 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 108) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 109 name "WAN 9 (unused)"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 109) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1000 name "Internal Services"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1000) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1001 name "Internal clients"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1001) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1002 name "Ceph front-end"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1002) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 1003 name "Ceph back-end"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1003) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''vlan 3900 name "Guest network"'''
mellanox-sx6036-rack-1 [standalone: master] (config vlan 3900) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show vlan'''
----------------------------------------------------------------------
VLAN Name Ports
----------------------------------------------------------------------
1 default Eth1/19, Eth1/20, Eth1/21, Eth1/22, Eth1/23,
Eth1/24, Eth1/25, Eth1/26, Eth1/27, Eth1/28,
Eth1/29, Eth1/30, Eth1/31, Eth1/32, Eth1/33,
Eth1/34, Eth1/35, Eth1/36
10 Management
100 WAN 0 (Comcast 44th Avenue)
101 WAN 1 (Allo)
102 WAN 2 (unused)
103 WAN 3 (unused)
104 WAN 4 (unused)
105 WAN 5 (unused)
106 WAN 6 (unused)
107 WAN 7 (unused)
108 WAN 8 (unused)
109 WAN 9 (unused)
1000 Internal Services
1001 Internal clients
1002 Ceph front-end
1003 Ceph back-end
3900 Guest network
mellanox-sx6036-rack-1 [standalone: master] #
And now, to assign all those VLANs as tagged on port 1/36:
mellanox-sx6036-rack-1 [standalone: master] > '''enable'''
mellanox-sx6036-rack-1 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''interface ethernet 1/36'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport mode trunk'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan 10'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan add 100-109'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan add 1000-1003'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''switchport trunk allowed-vlan add 3900'''
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-1 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-1 [standalone: master] # '''show vlan'''
----------------------------------------------------------------------
VLAN Name Ports
----------------------------------------------------------------------
1 default Eth1/19, Eth1/20, Eth1/21, Eth1/22, Eth1/23,
Eth1/24, Eth1/25, Eth1/26, Eth1/27, Eth1/28,
Eth1/29, Eth1/30, Eth1/31, Eth1/32, Eth1/33,
Eth1/34, Eth1/35
10 Management Eth1/36
100 WAN 0 (Comcast 44th Avenue)Eth1/36
101 WAN 1 (Allo) Eth1/36
102 WAN 2 (unused) Eth1/36
103 WAN 3 (unused) Eth1/36
104 WAN 4 (unused) Eth1/36
105 WAN 5 (unused) Eth1/36
106 WAN 6 (unused) Eth1/36
107 WAN 7 (unused) Eth1/36
108 WAN 8 (unused) Eth1/36
109 WAN 9 (unused) Eth1/36
1000 Internal Services Eth1/36
1001 Internal clients Eth1/36
1002 Ceph front-end Eth1/36
1003 Ceph back-end Eth1/36
3900 Guest network Eth1/36
mellanox-sx6036-rack-1 [standalone: master] #
== Unlocking the bootloader ==
When reset or just powered on, the SX6036's bootloader does preset the human on the console a 5 second countdown timer and an opportunity to choose some other options than loading the currently active software image:
Mellanox MLNX-OS

Default image: 'PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc'
Press Enter to boot this image, or 'Ctrl B' for boot menu

Booting default image in: 0

Mellanox MLNX-OS Boot Menu:

* 1: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc
2: PPC_M460EX 3.6.8010 2018-08-20 18:04:16 ppc
u: USB menu (if USB device connected) (password required)
c: Command prompt (password required)

Choice:
Irritatingly, I have been unable to locate the password required for command prompt access. Fortunately, some kind denizens of the internet have shared a way to change (even remove!) this password from a running MLNX-OS. I found this information at https://forums.servethehome.com/index.php?threads/solved-mellanox-sx6012-u-boot-password-removal-without-bash-access.33484/

Let's record this for posterity:
Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-2 login: admin
Password:
Last login: Sun Apr 12 16:57:02 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-2 [standalone: master] > '''enable'''
mellanox-sx6036-rack-2 [standalone: master] # '''configure terminal'''
mellanox-sx6036-rack-2 [standalone: master] (config) # '''boot bootmgr password 7 ""'''
mellanox-sx6036-rack-2 [standalone: master] (config) # '''exit'''
mellanox-sx6036-rack-2 [standalone: master] # '''write memory'''
mellanox-sx6036-rack-2 [standalone: master] # '''show bootvar'''

Installed images:
Partition 1:
version: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc

Partition 2:
version: PPC_M460EX 3.6.8010 2018-08-20 18:04:16 ppc

Last boot partition : 1
Next boot partition : 1
Serve image files via HTTP/HTTPS: no

No boot manager password is set.

Image signing : trusted signature always required
Admin require signed images: yes

Settings for next boot only:
Fallback reboot on configuration failure: yes (default)

mellanox-sx6036-rack-2 [standalone: master] #
Note the line saying "No boot manager password is set." A switch that has not had this treatment will report "Boot manager password is set." instead.

OpenWRT Aruba AP-303

2026-04-10T19:44:13Z

Adj:

The Aruba AP-303 hardware became supported by OpenWRT in its 21.02.0 release. Since I am all about free software wherever possible, and these are available at low prices on eBay, I have acquired a few.

Documentation of hardware, OpenWRT installation, and network configuration is here.

== About the Hardware and low level things ==
This mostly lifted from the [https://openwrt.org/toh/aruba/ap-303|OpenWRT page describing the AP-303] at openwrt.org. The SoC is a Qualcomm IPQ4029. Marketing data sheet can be seen at
https://www.qualcomm.com/content/dam/qcomm-martech/dm-assets/documents/ipq40x8ipq40x9-productbrief.pdf. But in short, it is a 4-core 32-bit ARMv7 system integrating 2.4GHz and 5GHz Wifi components. The AP-303 is equipped with 128MiB NAND Flash memory for persistent storage, and 512MiB of RAM for program code and data.

=== Serial Console ===
Serial console is attached to an external micro B style USB connector wired to the 3.3 Volt 2 wire serial. The HPE/Aruba cable with integrated FTDI USB to low voltage serial is sold under part number JY728A. Do not wire standard 12 Volt serial to this connector. Bad things are likely to happen.

=== Power up, boot loader, etc ===
Not having any of the Aruba power supplies, but having some Power Over Ethernet capable switches here...
# Connect console cable
# Set serial port to 9600bps, 8 data bits, no parity, 1 stop bit
# Apply power to Ethernet port
# Interrupt autoboot when the '''<code>Hit <Enter> to stop autoboot:</code>''' message appears.

=== Bootloader stuff and things ===
'''<code>mfginfo</code>''' will show hardware information including the access point's serial number:
apboot> '''mfginfo'''
Inventory:
Card 0: System
Wired MAC : 90:4c:81:cb:f1:8c
Wired MAC Count : 2
Serial : CNG9K9T1HP
Date Code : 072418
Country : CCODE-US-8392f77e514e9c048d2d1e48520555e3ed355812
Card 1: CPU
Assembly : 2010282C
Serial : MX87E0634
Date Code : 072418
Major Rev : 03
Minor Rev/Variant : 00
apboot>
'''<code>osinfo</code>''' will show information about the OS images on the flash storage:
apboot> '''osinfo'''
Partition 0:
image type: 0
machine type: 48
size: 22859904
version: 8.7.1.0-8.7.1.0
build string: ArubaOS version 8.7.1.0-8.7.1.0 for Ursa (p4build@pr-hpn-build09) (gcc version 4.6.3) #77203 SMP Wed Sep 23 00:55:22 UTC 2020
flags: Instant preserve
oem: aruba

Image is signed; verifying checksum... passed
SHA2 Signature available
Signer Cert OK
Policy Cert OK
RSA signature verified using SHA2.

Partition 1:
image type: 0
machine type: 48
size: 20639392
version: 8.8.0.0-8.8.0.0
build string: ArubaOS version 8.8.0.0-8.8.0.0 for Ursa (p4build@pr-hpn-build07) (gcc version 4.6.3) #79697 SMP Sat Mar 27 05:22:09 UTC 2021
flags: Instant preserve
oem: aruba

Image is signed; verifying checksum... passed
SHA2 Signature available
Signer Cert OK
Policy Cert OK
RSA signature verified using SHA2.
apboot>

=== Clearing ArubaOS settings ===
'''<code>factory_reset</code>''' will wipe the AP's configuration. Execute like so:
apboot> '''factory_reset'''
Clearing state...
Checking OS image and flags

Image is signed; verifying checksum... passed
Preserving image partition 0
Erasing flash sector @ 0x3a0000... OK
Erasing UBIFS ...done

Purging environment...
preserving os_partition (1)
Erasing SPI flash...Writing to SPI flash...done
Erasing SPI flash...Writing to SPI flash...done
apboot>

=== Slowing down the autoboot process a bit ===
After the '''<code>factory_reset</code>''' command has been executed, the autoboot delay is only 2 seconds. Here we set it to 20 seconds and save that value to flash for the next power cycle event:

apboot> '''setenv bootdelay 20'''
apboot> '''saveenv'''
Saving Environment to SPI Flash...
Erasing SPI flash...Writing to SPI flash...done
apboot>

=== Default admin credentials ===
ArubaOS's administrator user is '''<code>admin</code>'''. The default password is the access point's serial number. After wiping the AP's config with the factory_reset boot loader command (above), feel free to login with those on the console port and poke around. But we're not super interested in ArubaOS today, so let's move on to OpenWRT.

== OpenWRT installation ==
=== OpenWRT installation image acquisition and related setup ===
OpenWRT is a free Linux distribution primarily designed for WiFi access points and similar devices. Its web home page is https://www.openwrt.org/.

OpenWRT's wiki page for the Aruba Networks AP-303 is https://openwrt.org/toh/aruba/ap-303. See that site for the most current information. But, for our purposes here:
# Acquire the OpenWRT installation image from https://downloads.openwrt.org/releases/24.10.0/targets/ipq40xx/generic/openwrt-24.10.0-ipq40xx-generic-aruba_ap-303-initramfs-uImage.itb
# Save this image to a location on a TFTP server that will be accessible to the AP and name it '''<code>ipq40xx.ari</code>'''. (Or whatever the <code>bootfile</code> variable is in the AP's bootloader.)
# Update DHCP server config to include the new AP, too.
## set '''<code>next-server</code>''' with the TFTP server's IP address
## set '''<code>filename</code>''' to the installation image file saved above. "openwrt-24.10.0-ipq40xx-generic-aruba_ap-303-initramfs-uImage.itb" for now.
# Configure the AP's network port appropriately on the switch side
## Boot loader does not know about tagged VLANs. So set primary VLAN as appropriate. And perhaps add some tagged VLANs that will be connected to wireless SSIDs
## PoE enabled for the access point to run

=== OpenWRT installation procedure ===
# Power on the AP, interrupt the autoboot process
# Tell the boot loader to get network config from a DHCP server:
## Interrupt bootloader at the <code>Hit <Enter> to stop autoboot:</code> message.
## Get network config from DHCP server: '''<code>dhcp</code>'''
# Some boot loader environment variables get set next:
apboot> '''setenv bootargs_openwrt "setenv bootargs console=ttyMSM1,9600n8"'''
apboot> '''setenv nandboot_openwrt "run bootargs_openwrt; ubi part aos1; ubi read 0x85000000 kernel; set fdt_high 0x87000000; bootm 0x85000000"'''
apboot> '''setenv ramboot_openwrt "run bootargs_openwrt; setenv ipaddr 192.168.1.105; setenv serverip 192.168.1.75; netget; set fdt_high 0x87000000; bootm"'''
apboot> '''setenv bootcmd "run nandboot_openwrt"'''
apboot> '''saveenv'''
# And now the value of the ramboot_openwrt command is actually executed: <code>apboot> '''run ramboot_openwrt'''</code>. This will download the ipq40xx file from the TFTP server and run it. Any complaint about an invalid image format can be safely ignored.
# After the Linux kernel stops spewing messages to the console, send a newline (hit Enter), and be greeted with the astonishing ASCII art OpenWRT logo and a root shell prompt.

_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt 24.10.0, r28427-6df0e3d02a
-----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@OpenWrt:~#
# Update network to use DHCP and restart it
## Change <code>option proto 'static'</code> to <code>option proto 'dhcp'</code> for the <code>lan</code> interface in <code>/etc/config/network</code>. (There is a small implementation of vi included in the installation image.)
## Restart all networking by running <code>root@OpenWrt:~# '''service network restart'''</code>.
# Fetch the OpenWRT sysupgrade image to /tmp on the AP: <code>root@OpenWrt:~# cd /tmp; wget '''https://downloads.openwrt.org/releases/24.10.0/targets/ipq40xx/generic/openwrt-24.10.0-ipq40xx-generic-aruba_ap-303-squashfs-sysupgrade.bin'''</code>.
# Clean up the flash. No more ArubaOS when this is done.
root@OpenWrt:/tmp# '''ubidetach -p /dev/mtd1'''
[ 1416.126100] ubi0: detaching mtd1
[ 1416.126816] ubi0: mtd1 is detached
root@OpenWrt:/tmp# '''ubiformat /dev/mtd1'''
ubiformat: mtd1 (nand), size 33554432 bytes (32.0 MiB), 256 eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 2048 bytes
libscan: scanning eraseblock 255 -- 100 % complete
ubiformat: 256 eraseblocks have valid erase counter, mean value is 2
ubiformat: formatting eraseblock 255 -- 100 % complete
root@OpenWrt:/tmp#
# Apply the OpenWrt sysupgrade image:
root@OpenWrt:/tmp# '''sysupgrade -n openwrt-24.10.0-ipq40xx-generic-aruba_ap-303-squashfs-sysupgrade.bin'''
Sat Feb 22 04:12:30 UTC 2025 upgrade: Commencing upgrade. Closing all shell sessions.
Watchdog handover: fd=3
- watchdog -
Watchdog did not previously reset the system
Sat Feb 22 04:12:31 UTC 2025 upgrade: Sending TERM to remaining processes ...
Sat Feb 22 04:12:31 UTC 2025 upgrade: Sending signal TERM to netifd (3025)
Sat Feb 22 04:12:35 UTC 2025 upgrade: Sending KILL to remaining processes ...
Sat Feb 22 04:12:35 UTC 2025 upgrade: Sending signal KILL to netifd (3025)
[11388.891505] stage2 (3683): drop_caches: 3
Sat Feb 22 04:12:44 UTC 2025 upgrade: Switching to ramdisk...
Sat Feb 22 04:12:46 UTC 2025 upgrade: Performing system upgrade...
verifying sysupgrade tar file integrity
[11392.106885] ubi0: attaching mtd1
[11392.439761] ubi0: scanning is finished
[11392.453417] ubi0: attached mtd1 (name "ubi", size 32 MiB)
[11392.471153] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
[11392.536731] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[11392.617971] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
[11392.700276] ubi0: good PEBs: 256, bad PEBs: 0, corrupted PEBs: 0
[11392.781509] ubi0: user volume: 0, internal volumes: 1, max. volumes count: 128
[11392.856520] ubi0: max/mean erase counter: 8/3, WL threshold: 4096, image sequence number: 1647604229
[11392.940952] ubi0: available PEBs: 232, total reserved PEBs: 24, PEBs reserved for bad PEB handling: 20
[11393.052389] ubi0: background thread "ubi_bgt0d" started, PID 4405
UBI device number 0, total 256 LEBs (32505856 bytes, 31.0 MiB), available 232 LEBs (29458432 bytes, 28.0 MiB), LEB size 126976 bytes (124.0 KiB)[11393.308786] block ubiblock0_1: created from ubi0:1(rootfs)

Volume ID 0, size 28 LEBs (3555328 bytes, 3.3 MiB), LEB size 126976 bytes (124.0 KiB), dynamic, name "kernel", alignment 1
Volume ID 1, size 33 LEBs (4190208 bytes, 3.9 MiB), LEB size 126976 bytes (124.0 KiB), dynamic, name "rootfs", alignment 1
Set volume size to 21712896
Volume ID 2, size 171 LEBs (21712896 bytes, 20.7 MiB), LEB size 126976 bytes (124.0 KiB), dynamic, name "rootfs_data", alignment 1
sysupgrade

=== Basic OpenWRT configuration ===
==== Wired Ethernet setup ====
After the OpenWrt installation has completed, the AP is lacking all configuration. Again <code>/etc/config/network</code> will need to be edited. Set it to DHCP mode and again run <code>service network restart</code>. Then wait patiently. 45 seconds may be needed for things to fully settle out.

==== The OpenWrt way of configuration ====
root@OpenWrt:~# '''uci set network.lan.proto=dhcp'''
root@OpenWrt:~# '''service network restart'''
root@OpenWrt:~# '''sleep 60; ip addr list dev br-lan'''
5: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 90:4c:81:cb:f1:8c brd ff:ff:ff:ff:ff:ff
inet 172.16.10.40/24 brd 172.16.10.255 scope global br-lan
valid_lft forever preferred_lft forever
inet6 fe80::924c:81ff:fecb:f18c/64 scope link
valid_lft forever preferred_lft forever
root@OpenWrt:~#

==== OS updates ====
root@OpenWrt:~# '''opkg update'''
Downloading https://downloads.openwrt.org/releases/24.10.0/targets/ipq40xx/generic/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_core
Downloading https://downloads.openwrt.org/releases/24.10.0/targets/ipq40xx/generic/packages/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/base/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_base
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/base/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/24.10.0/targets/ipq40xx/generic/kmods/6.6.73-1-60aeaf7e722ca0f86e06f61157755da3/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_kmods
Downloading https://downloads.openwrt.org/releases/24.10.0/targets/ipq40xx/generic/kmods/6.6.73-1-60aeaf7e722ca0f86e06f61157755da3/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_luci
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_packages
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/packages/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/routing/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_routing
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/routing/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/telephony/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_telephony
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/telephony/Packages.sig
Signature check passed.
root@OpenWrt:~# '''opkg list-upgradable'''
luci-mod-system - 24.336.49068~ef16778 - 25.049.66344~2b8e93c
ucode-mod-uloop - 2024.07.22~b610860d-r3 - 2025.02.10~a8a11aea-r1
ucode-mod-ubus - 2024.07.22~b610860d-r3 - 2025.02.10~a8a11aea-r1
luci-theme-bootstrap - 24.357.59015~779ca14 - 25.049.66344~2b8e93c
ucode-mod-uci - 2024.07.22~b610860d-r3 - 2025.02.10~a8a11aea-r1
ucode-mod-math - 2024.07.22~b610860d-r3 - 2025.02.10~a8a11aea-r1
luci-mod-status - 25.009.65022~20bf9a4 - 25.049.66344~2b8e93c
luci-app-firewall - 25.014.06885~bd39c63 - 25.049.66344~2b8e93c
ucode-mod-fs - 2024.07.22~b610860d-r3 - 2025.02.10~a8a11aea-r1
luci-ssl - 24.328.43591~9e70c9a - 25.049.66344~2b8e93c
ucode-mod-nl80211 - 2024.07.22~b610860d-r3 - 2025.02.10~a8a11aea-r1
ucode - 2024.07.22~b610860d-r3 - 2025.02.10~a8a11aea-r1
luci-app-package-manager - 24.341.02353~276fef8 - 25.049.66344~2b8e93c
luci-proto-ppp - 25.009.71487~a88d4af - 25.049.66344~2b8e93c
luci-mod-admin-full - 19.253.48496~3f93650 - 25.049.66344~2b8e93c
luci-base - 25.014.55016~7046a1c - 25.049.66344~2b8e93c
luci-proto-ipv6 - 24.350.69236~8d13ea4 - 25.049.66344~2b8e93c
ucode-mod-rtnl - 2024.07.22~b610860d-r3 - 2025.02.10~a8a11aea-r1
luci - 24.297.79519~bcd13b9 - 25.049.66344~2b8e93c
luci-light - 23.024.33244~34dee82 - 25.049.66344~2b8e93c
luci-mod-network - 25.021.30718~36b6107 - 25.049.66344~2b8e93c
libucode20230711 - 2024.07.22~b610860d-r3 - 2025.02.10~a8a11aea-r1
root@OpenWrt:~# '''opkg upgrade $(opkg list-upgradable | sort | cut -d' ' -f1)'''
Upgrading libucode20230711 on root from 2024.07.22~b610860d-r3 to 2025.02.10~a8a11aea-r1...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/base/libucode20230711_2025.02.10~a8a11aea-r1_arm_cortex-a7_neon-vfpv4.ipk
Upgrading luci on root from 24.297.79519~bcd13b9 to 25.049.66344~2b8e93c...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/luci_25.049.66344~2b8e93c_all.ipk
Upgrading luci-app-firewall on root from 25.014.06885~bd39c63 to 25.049.66344~2b8e93c...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/luci-app-firewall_25.049.66344~2b8e93c_all.ipk
Upgrading luci-app-package-manager on root from 24.341.02353~276fef8 to 25.049.66344~2b8e93c...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/luci-app-package-manager_25.049.66344~2b8e93c_all.ipk
Upgrading luci-base on root from 25.014.55016~7046a1c to 25.049.66344~2b8e93c...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/luci-base_25.049.66344~2b8e93c_arm_cortex-a7_neon-vfpv4.ipk
Upgrading luci-light on root from 23.024.33244~34dee82 to 25.049.66344~2b8e93c...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/luci-light_25.049.66344~2b8e93c_all.ipk
Upgrading luci-mod-admin-full on root from 19.253.48496~3f93650 to 25.049.66344~2b8e93c...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/luci-mod-admin-full_25.049.66344~2b8e93c_all.ipk
Upgrading luci-mod-network on root from 25.021.30718~36b6107 to 25.049.66344~2b8e93c...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/luci-mod-network_25.049.66344~2b8e93c_all.ipk
Upgrading luci-mod-status on root from 25.009.65022~20bf9a4 to 25.049.66344~2b8e93c...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/luci-mod-status_25.049.66344~2b8e93c_arm_cortex-a7_neon-vfpv4.ipk
Upgrading luci-mod-system on root from 24.336.49068~ef16778 to 25.049.66344~2b8e93c...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/luci-mod-system_25.049.66344~2b8e93c_all.ipk
Upgrading luci-proto-ipv6 on root from 24.350.69236~8d13ea4 to 25.049.66344~2b8e93c...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/luci-proto-ipv6_25.049.66344~2b8e93c_all.ipk
Upgrading luci-proto-ppp on root from 25.009.71487~a88d4af to 25.049.66344~2b8e93c...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/luci-proto-ppp_25.049.66344~2b8e93c_all.ipk
Upgrading luci-ssl on root from 24.328.43591~9e70c9a to 25.049.66344~2b8e93c...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/luci-ssl_25.049.66344~2b8e93c_all.ipk
Upgrading luci-theme-bootstrap on root from 24.357.59015~779ca14 to 25.049.66344~2b8e93c...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/luci/luci-theme-bootstrap_25.049.66344~2b8e93c_all.ipk
Upgrading ucode on root from 2024.07.22~b610860d-r3 to 2025.02.10~a8a11aea-r1...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/base/ucode_2025.02.10~a8a11aea-r1_arm_cortex-a7_neon-vfpv4.ipk
Upgrading ucode-mod-fs on root from 2024.07.22~b610860d-r3 to 2025.02.10~a8a11aea-r1...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/base/ucode-mod-fs_2025.02.10~a8a11aea-r1_arm_cortex-a7_neon-vfpv4.ipk
Upgrading ucode-mod-math on root from 2024.07.22~b610860d-r3 to 2025.02.10~a8a11aea-r1...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/base/ucode-mod-math_2025.02.10~a8a11aea-r1_arm_cortex-a7_neon-vfpv4.ipk
Upgrading ucode-mod-nl80211 on root from 2024.07.22~b610860d-r3 to 2025.02.10~a8a11aea-r1...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/base/ucode-mod-nl80211_2025.02.10~a8a11aea-r1_arm_cortex-a7_neon-vfpv4.ipk
Upgrading ucode-mod-rtnl on root from 2024.07.22~b610860d-r3 to 2025.02.10~a8a11aea-r1...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/base/ucode-mod-rtnl_2025.02.10~a8a11aea-r1_arm_cortex-a7_neon-vfpv4.ipk
Upgrading ucode-mod-ubus on root from 2024.07.22~b610860d-r3 to 2025.02.10~a8a11aea-r1...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/base/ucode-mod-ubus_2025.02.10~a8a11aea-r1_arm_cortex-a7_neon-vfpv4.ipk
Upgrading ucode-mod-uci on root from 2024.07.22~b610860d-r3 to 2025.02.10~a8a11aea-r1...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/base/ucode-mod-uci_2025.02.10~a8a11aea-r1_arm_cortex-a7_neon-vfpv4.ipk
Upgrading ucode-mod-uloop on root from 2024.07.22~b610860d-r3 to 2025.02.10~a8a11aea-r1...
Downloading https://downloads.openwrt.org/releases/24.10.0/packages/arm_cortex-a7_neon-vfpv4/base/ucode-mod-uloop_2025.02.10~a8a11aea-r1_arm_cortex-a7_neon-vfpv4.ipk
Configuring libucode20230711.
Configuring ucode.
Configuring ucode-mod-fs.
Configuring ucode-mod-nl80211.
Configuring ucode-mod-rtnl.
Configuring ucode-mod-ubus.
Configuring ucode-mod-uloop.
Configuring ucode-mod-uci.
Configuring ucode-mod-math.
Configuring luci-base.
Configuring luci-mod-system.
Configuring luci-theme-bootstrap.
Configuring luci-mod-status.
Configuring luci-app-firewall.
Configuring luci-proto-ipv6.
Configuring luci-mod-network.
Configuring luci-mod-admin-full.
Configuring luci-proto-ppp.
Configuring luci-light.
Configuring luci-app-package-manager.
Configuring luci-ssl.
Configuring luci.
Collected errors:
* resolve_conffiles: Existing conffile /etc/config/luci is different from the conffile in the new package. The new conffile will be placed at /etc/config/luci-opkg.
root@OpenWrt:~#
=== Reboot and test functionality ===
root@OpenWrt:~# '''reboot'''
== Configure our first WiFi networks ==
=== Out of the box contents of /etc/config/wireless ===
root@OpenWrt:~# '''cat /etc/config/wireless'''

config wifi-device 'radio0'
option type 'mac80211'
option path 'platform/soc/a000000.wifi'
option band '2g'
option channel '1'
option htmode 'VHT20'
option disabled '1'

config wifi-iface 'default_radio0'
option device 'radio0'
option network 'lan'
option mode 'ap'
option ssid 'OpenWrt'
option encryption 'none'

config wifi-device 'radio1'
option type 'mac80211'
option path 'platform/soc/a800000.wifi'
option band '5g'
option channel '36'
option htmode 'VHT80'
option disabled '1'

config wifi-iface 'default_radio1'
option device 'radio1'
option network 'lan'
option mode 'ap'
option ssid 'OpenWrt'
option encryption 'none'
Here we see 2 radios, both disabled. On each of the radios, a WiFi SSID called <code>OpenWrt</code> is defined and it is bridged to the network interface known as <code>lan</code>. Additionally. the radios are assigned to default channels.

=== Restoring default WiFi radio setup ===
root@OpenWrt:/etc/config# '''mv wireless wireless~'''
root@OpenWrt:/etc/config# '''wifi config'''
root@OpenWrt:/etc/config# '''ls -l wireless'''
-rw-r--r-- 1 root root 652 Feb 25 17:25 wireless
root@OpenWrt:/etc/config#

=== Desired end state ===
* 2 SSIDs defined. One for authenticated devices with access to internal network resources (VLAN 1001). And another open network attached to the guest subnet (VLAN 3900).
* Both SSIDs available on both the 2.4GHz and 5GHz spectrum.
* APs able to do some band steering, hopefully encouraging clients to connect on the 5GHz radio.
* APs doing some channel selection on their own, not just a static one from the config file
* Fast roaming between APs

=== Getting to end state: packages ===
<code>wpad-basic-mbedtls</code> is installed out of the box on this platform. It should be sufficient for our purposes, only has the WiFi mesh features turned off. (WiFi mesh is inter-AP backhaul over the air, one AP is a client of another and still serving clients on the non-backhaul radio.)
root@OpenWrt:/etc/config# '''opkg list-installed | grep wpad'''
wpad-basic-mbedtls - 2024.09.15~5ace39b0-r2
root@OpenWrt:/etc/config# '''opkg list | grep wpad-basic-mbedtls'''
wpad-basic-mbedtls - 2024.09.15~5ace39b0-r2 - This package contains a basic IEEE 802.1x/WPA Authenticator and Supplicant with WPA-PSK, SAE (WPA3-Personal), 802.11r and 802.11w support.
root@OpenWrt:/etc/config#

== Oh, look, it is 2026! ==
And things are mostly the same. New images to install listed at https://downloads.openwrt.org/releases/25.12.2/targets/ipq40xx/generic/ :
* https://downloads.openwrt.org/releases/25.12.2/targets/ipq40xx/generic/openwrt-25.12.2-ipq40xx-generic-aruba_ap-303-initramfs-uImage.itb
* https://downloads.openwrt.org/releases/25.12.2/targets/ipq40xx/generic/openwrt-25.12.2-ipq40xx-generic-aruba_ap-303-squashfs-sysupgrade.bin
And the package manager, <code>opkg</code>, has been replaced by a tool called <code>apk</code>. This seems to have nothing to do with Android's APK package files, though. An administrator's <code>apk</code> cheatsheet can be found at https://openwrt.org/docs/guide-user/additional-software/opkg-to-apk-cheatsheet.
So far, it is a bit more like Debian's <code>apt-get</code> tool. Getting all outstanding package updates is simply <code>apk update; apk upgrade</code> now.

=== Packages we want on our OpenWrt 25.12.x APs ===
* diffutils -- GNU text file delta generator, useful for comparing two versions of a config file
* usteer -- this steers WiFi clients toward the best frequency band and AP depending on signal strength, number of stations connected, and other factors.

=== And some basic system setup ===
OpenWrt has most (all?) of its config files stored under /etc/config. There is a CLI tool, <code>uci</code>, to manage them. Let's walk through setting up a new AP's hostname, and getting a juicy kernel random seed value set:
root@OpenWrt:~# '''uci show system'''
system.@system[0]=system
system.@system[0].hostname='OpenWrt'
system.@system[0].timezone='GMT0'
system.@system[0].zonename='UTC'
system.@system[0].ttylogin='0'
system.@system[0].log_size='128'
system.@system[0].urandom_seed='0'
system.@system[0].compat_version='1.1'
system.ntp=timeserver
system.ntp.enabled='1'
system.ntp.enable_server='0'
system.ntp.server='0.openwrt.pool.ntp.org' '1.openwrt.pool.ntp.org' '2.openwrt.pool.ntp.org' '3.openwrt.pool.ntp.org'
root@OpenWrt:~# '''uci set system.@system[0].hostname=fnord-ap303-4'''
root@OpenWrt:~# '''uci commit system'''
root@OpenWrt:~# '''/etc/init.d/system reload'''
root@fnord-ap303-4:~# '''uci set system.@system[0].urandom_seed=$(wget -q -O - 'https://www.random.org/cgi-bin/randbyte?nbytes=32&format=h' | tr -d ' ' | tr -d '\n')'''
root@fnord-ap303-4:~# '''uci commit system'''
root@fnord-ap303-4:~# '''/etc/init.d/system reload'''
root@fnord-ap303-4:~#
<code>paste</code> might be the better standard UNIX tool for the <code>tr -d '\n'</code> in the above. But many of the things we might expect from GNU coreutils are not installed by default on OpenWrt. So we have <code>tr</code> remove spaces from the output from random.org's super awesome random bits delivery service and then have it remove the newlines, too. End result is the same.

=== Wired network config ===
Feed this block of text to <code>uci import network</code> and finish things off with an ASCII end-of-transmission marker (<code>Ctrl-D</code>) or whatever your termios end-of-file (EOF) character is.
package network

config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'

config globals 'globals'
option ula_prefix 'fd1f:ed39:f7d::/48'
option packet_steering '1'

config device
option name 'br-lan'
option type 'bridge'
list ports 'lan'

config interface 'lan'
option device 'br-lan.10'
option proto 'dhcp'

config device
option type '8021q'
option ifname 'br-lan'
option vid '1001'
option name 'br-lan.1001'

config device
option type '8021q'
option ifname 'br-lan'
option vid '3900'
option name 'br-lan.3900'

config interface 'lan_1001'
option device 'br-lan.1001'

config interface 'lan_3900'
option device 'br-lan.3900'

config bridge-vlan
option device 'br-lan'
list ports 'lan:u*'
option vlan '10'

config bridge-vlan
option device 'br-lan'
option vlan '1001'
list ports 'lan:t'

config bridge-vlan
option device 'br-lan'
option vlan '3900'
list ports 'lan:t'
This was pieced together experimentally over time. Not guided by the reference documentation. Mostly with the LuCI web UI. The <code>ula_prefix</code> global option value probably does not matter for anything I'm doing here.

=== Wireless network config ===
This is like the wired networking config previously. Except it has SSID security passwords in it. So I am not sharing the actual contents here. Essentially, though, feed a config to <code>uci import wireless; uci commit wireless; sync; sleep 3; sync; reboot</code> and things should work out OK.

OpenWRT Aruba AP-303

2026-04-10T19:36:42Z

Adj:

OpenWRT Aruba AP-303

2026-04-10T18:58:45Z

Adj: /* Wired network config */

== Oh, look, it is 2026! ==
And things are mostly the same. New images to install listed at https://downloads.openwrt.org/releases/25.12.2/targets/ipq40xx/generic/ :
* https://downloads.openwrt.org/releases/25.12.2/targets/ipq40xx/generic/openwrt-25.12.2-ipq40xx-generic-aruba_ap-303-initramfs-uImage.itb
* https://downloads.openwrt.org/releases/25.12.2/targets/ipq40xx/generic/openwrt-25.12.2-ipq40xx-generic-aruba_ap-303-squashfs-sysupgrade.bin
And the package manager, <code>opkg</code>, has been replaced by a tool called <code>apk</code>. This seems to have nothing to do with Android's APK package files, though. An administrator's <code>apk</code> cheatsheet can be found at https://openwrt.org/docs/guide-user/additional-software/opkg-to-apk-cheatsheet.
So far, it is a bit more like Debian's <code>apt-get</code> tool. Getting all outstanding package updates is simply <code>apk update; apk upgrade</code> now.

=== Packages we want on our OpenWrt 25.12.x APs ===
* diffutils -- GNU text file delta generator, useful for comparing two versions of a config file
* usteer -- this steers WiFi clients toward the best frequency band and AP depending on signal strength, number of stations connected, and other factors.

=== And some basic system setup ===
OpenWrt has most (all?) of its config files stored under /etc/config. There is a CLI tool, <code>uci</code>, to manage them. Let's walk through setting up a new AP's hostname, and getting a juicy kernel random seed value set:
root@OpenWrt:~# '''uci show system'''
system.@system[0]=system
system.@system[0].hostname='OpenWrt'
system.@system[0].timezone='GMT0'
system.@system[0].zonename='UTC'
system.@system[0].ttylogin='0'
system.@system[0].log_size='128'
system.@system[0].urandom_seed='0'
system.@system[0].compat_version='1.1'
system.ntp=timeserver
system.ntp.enabled='1'
system.ntp.enable_server='0'
system.ntp.server='0.openwrt.pool.ntp.org' '1.openwrt.pool.ntp.org' '2.openwrt.pool.ntp.org' '3.openwrt.pool.ntp.org'
root@OpenWrt:~# '''uci set system.@system[0].hostname=fnord-ap303-4'''
root@OpenWrt:~# '''uci commit system'''
root@OpenWrt:~# '''/etc/init.d/system reload'''
root@fnord-ap303-4:~# '''uci set system.@system[0].urandom_seed=$(wget -q -O - 'https://www.random.org/cgi-bin/randbyte?nbytes=32&format=h' | tr -d ' ' | tr -d '\n')'''
root@fnord-ap303-4:~# '''uci commit system'''
root@fnord-ap303-4:~# '''/etc/init.d/system reload'''
root@fnord-ap303-4:~#
<code>paste</code> might be the better standard UNIX tool for the <code>tr -d '\n'</code> in the above. But many of the things we might expect from GNU coreutils are not installed by default on OpenWrt. So we have <code>tr</code> remove spaces from the output from random.org's super awesome random bits delivery service and then have it remove the newlines, too. End result is the same.

=== Wired network config ===
Feed this block of text to <code>uci import network</code> and finish things off with an ASCII end-of-transmission marker (<code>Ctrl-D</code>) or whatever your termios end-of-file (EOF) character is.
package network

config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'

config globals 'globals'
option ula_prefix 'fd1f:ed39:f7d::/48'
option packet_steering '1'

config device
option name 'br-lan'
option type 'bridge'
list ports 'lan'

config interface 'lan'
option device 'br-lan.10'
option proto 'dhcp'

config device
option type '8021q'
option ifname 'br-lan'
option vid '1001'
option name 'br-lan.1001'

config device
option type '8021q'
option ifname 'br-lan'
option vid '3900'
option name 'br-lan.3900'

config interface 'lan_1001'
option device 'br-lan.1001'

config interface 'lan_3900'
option device 'br-lan.3900'

config bridge-vlan
option device 'br-lan'
list ports 'lan:u*'
option vlan '10'

config bridge-vlan
option device 'br-lan'
option vlan '1001'
list ports 'lan:t'

config bridge-vlan
option device 'br-lan'
option vlan '3900'
list ports 'lan:t'
This was pieced together experimentally over time. Not guided by the reference documentation. Mostly with the LuCI web UI. The <code>ula_prefix</code> global option value probably does not matter for anything I'm doing here.

=== Wireless network config ===
This is like the wired networking config previously. Except it has SSID security passwords in it. So I am not sharing the actual contents here. Essentially, though, feed a config to <code>uci import wireless; uci commit wireless; sync; sleep 3; sync; reboot</code> and things should work out OK.

OpenWRT Aruba AP-303

2026-04-08T23:20:33Z

Adj:

OpenWRT Aruba AP-303

2026-04-04T21:35:00Z

Adj: /* Getting to end state: packages */

UCS Manager

2026-04-03T05:10:26Z

Adj: /* A new step for firmware infrastructure updates */

There are a few pieces of Cisco UCS equipment in the data center:
* One UCS 5108 blade enclosure, accommodating up to 8 B200 server blades of various generations
* 2 UCS-FI-6248UP fabric interconnects, which sit between UCS servers or blade chassis and the rest of the network
* 1 UCS C220 M5 server

Cisco UCS Manager gets installed on the Fabric Interconnects and does Cisco proprietary magic things. Like powering on a single blade. This is way more complicated that it needs to be. So this document is being written to capture knowledge of installation, upgrades, and server management tasks.

== How to get ==
Much (all?) of Cisco's UCS software is available directly without a support contract. But a customer account is required to download. Registration is free as in beer. I have had one since July 2021 and have not been troubled with Cisco spam or any sort of "are you a real person at a real company?" validation.

As this is written, [https://software.cisco.com/download/home/283612660/type/283655658/release/6.0(1e) UCS Infrastructure and UCS Manager Software] is the place to get it. As I have some older, end-of-life, end-of-support fabric interconnects (62xx series) which are not supported on the current 6.0 release train, I have instead selected the 4.2(3p) version for download. This comes as a 1Gbyte-ish <code>ucs-k9-bundle-infra.4.2.3p.A.bin</code> file.

== Untested Fabric Interconnect, what do we do? ==
5ish years ago (summer 2021), I acquired three of these 6248UP FIs. And I think I got one of them configured for use on the management network and updated to then-current software. I'm not sure which of the three that was. And I am sure I did no documentation at the time. So here we are with new notes.

=== Console and power are at the front?!?! ===
So, this is weird. Console and power connections are at the front side of the chassis. All of the connectivity is at the rear as it should be. There is sufficient room on the back for a couple of C14 power connectors '''and''' 2 more 8P8C modular connectors for management LAN and serial console. But they are at the front. So cabling is kinda sorta extra fun. ''sigh''

Console serial cable is the usual Cisco modular pinout. Signal on pins 3 and 6. 9600 bps, 8-n-1 are the serial port settings. Nothing unusual there.

Mystery solved regarding the connections at front of the chassis. My FIs have 32 ports for traffic, but there is (well, was, back when this was a current product) a 16 port expansion module available. See https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-6200-series-fabric-interconnects/data_sheet_c78-675245.html for some diagrams.

=== Basic Fabric Interconnect configuration ===
I might have lucked out on the first one I received, but it booted normally and started the interactive system setup dialog. It is all pretty basic. Set a password for the <code>admin</code> user and an IP address, netmask, IPv4 gateway address, and DNS server address for the management Ethernet interface (again, on the front). That's pretty much it.

After this is done, make sure the management Ethernet interface is connected to a switch on the correct VLAN and subnet. Then do all the things over an SSH connection. At least for version 2.1, the SSH server only knows about RSA host keys, so a modern OpenSSH client will need something like <code>-o HostKeyAlgorithms=+ssh-rsa</code> listed as a command line option.

== Clock discipline is good ==
To set the time in the FI. Or just get it close. NTP will come. Eventually.

ucs-fi-6248up-0-A# '''scope system'''
ucs-fi-6248up-0-A /system # '''scope services'''
ucs-fi-6248up-0-A /system/services # '''set clock Mar 22 2026 02 53 35'''
Sun Mar 22 02:53:35 UTC 2026
ucs-fi-6248up-0-A /system/services #

And now for the NTP:
ucs-fi-6248up-0-A# '''scope system'''
ucs-fi-6248up-0-A /system # '''scope services'''
ucs-fi-6248up-0-A /system/services # '''create ntp-server 172.16.10.2'''
ucs-fi-6248up-0-A /system/services* # '''commit-buffer'''
ucs-fi-6248up-0-A /system/services # '''create ntp-server 172.16.10.3'''
ucs-fi-6248up-0-A /system/services* # '''commit-buffer'''
ucs-fi-6248up-0-A /system/services # '''show ntp'''
NTP Servers:
Name: 172.16.10.2
Name: 172.16.10.3
ucs-fi-6248up-0-A /system/services #

== Software upgrade time! ==
Again, on the first one I have tried, let's log in as <code>admin</code> and see what's running:

Cisco UCS 6200 Series Fabric Interconnect
ucs-fi-6248up-0-A login: '''admin'''
Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

ucs-fi-6248up-0-A# '''show version'''
System version: 2.2(1c)
ucs-fi-6248up-0-A#
And, more verbosely (for reasons that are beyond me, "brief" mode shows more output?):
ucs-fi-6248up-0-A# '''show version brief'''
System version: 2.2(1c)
Package-Vers: 2.2(1c)A
ucs-fi-6248up-0-A#

So that's really very super dooper old. I downloaded a copy of the 4.1(3i) code in August 2022. Off do so some reading to see if I can skip all the in-between versions.

Did not find anything authoritative in the release notes (but I only looked very briefly.) So, we can do it stepwise. Hopefully. Process looks something like this:
ucs-fi-6248up-0-A# '''scope firmware '''
ucs-fi-6248up-0-A /firmware # '''download image tftp://172.17.0.18/ucs-k9-bundle-infra.2.2.8m.A.bin'''
ucs-fi-6248up-0-A /firmware # '''show download-task'''

Download task:
File Name Protocol Server Userid State
--------- -------- --------------- --------------- -----
ucs-k9-bundle-infra.2.2.8m.A.bin
Tftp 172.17.0.18 Downloaded
ucs-fi-6248up-0-A /firmware #

Yes, you read that right. I delivered the software image to the FI over TFTP. SSH (scp and sftp) are supported protocols, but could not connect to an OpenSSH server on Debian 13. This might be better with newer FI firmware, but for now, I already have a TFTP server handy for netbooting other machines in the environment.

And from here, the firmware bundle can actually be installed on the Fabric Interconnect like so:
ucs-fi-6248up-0-A# '''scope firmware'''
ucs-fi-6248up-0-A /firmware # '''scope auto-install'''
ucs-fi-6248up-0-A /firmware/auto-install # '''install infra infra-vers 2.2(8m)A'''
This operation upgrades firmware on UCS Infrastructure Components
(UCS manager, Fabric Interconnects and IOMs). Do you want to proceed?
(yes/no):'''yes'''

Triggering Install-Infra with:
Infrastructure Pack Version: 2.2(8m)A
ucs-fi-6248up-0-A /firmware/auto-install # '''acknowledge primary fabric-interconnect reboot'''
ucs-fi-6248up-0-A /firmware/auto-install* # '''commit-buffer'''
ucs-fi-6248up-0-A /firmware/auto-install # '''show fsm status expand'''

FSM Status:

Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: In Progress
Completion Time:
Progress (%): 87

FSM Stage:

Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployActivateUCSM Success 0
6 DeployPollActivateOfUCSM In Progress 1
7 DeployUpdateIOM Pending 0
8 DeployPollUpdateOfIOM Pending 0
9 DeployActivateIOM Pending 0
10 DeployPollActivateOfIOM Pending 0
11 DeployActivateRemoteFI Pending 0
12 DeployPollActivateOfRemoteFI Pending 0
13 DeployWaitForUserAck Pending 0
14 DeployActivateLocalFI Pending 0
15 DeployPollActivateOfLocalFI Pending 0
ucs-fi-6248up-0-A /firmware/auto-install # Connection to 172.16.10.176 closed.
Somewhere along the way, the SSH server hung up on me. But that seems like the sort of thing that '''should''' happen as the FI's OS is being upgraded. Progress can also be tracked on the serial console port like so:
ucs-fi-6248up-0-A# '''scope firmware'''
ucs-fi-6248up-0-A /firmware # '''scope auto-install'''
ucs-fi-6248up-0-A /firmware/auto-install # '''show fsm status expand | no-more'''

FSM Status:

Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: In Progress
Completion Time:
Progress (%): 98

FSM Stage:

Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployDownloadImages Skip 0
6 DeployCopyAllImagesToPeer Skip 0
7 DeployInternalBackup Success 0
8 DeployPollInternalBackup Success 2
9 DeployActivateUCSM Skip 0
10 DeployPollActivateOfUCSM Success 0
11 DeployUpdateIOM Success 0
12 DeployPollUpdateOfIOM Success 0
13 DeployActivateIOM Success 0
14 DeployPollActivateOfIOM Success 0
15 DeployActivateRemoteFI Skip 0
16 DeployPollActivateOfRemoteFI Skip 0
17 DeployWaitForUserAck Skip 0
18 DeployPollWaitForUserAck Success 0
19 DeployActivateLocalFI Success 0
20 DeployPollActivateOfLocalFI In Progress 2
ucs-fi-6248up-0-A /firmware/auto-install #
At some point, the FI will reboot itself.

Confirm new code is running like so:
ucs-fi-6248up-0-A# '''show version'''
System version: 2.2(8m)
ucs-fi-6248up-0-A# '''show version brief'''
System version: 2.2(8m)
Package-Vers: 2.2(8m)A
ucs-fi-6248up-0-A#

== STOP! Don't forget the other UCS things! ==
Um, yeah. So there is this UCS 5108 blade chassis in the data center, too. No idea what its firmware versions are. But they are sure to also need firmware updates. And it would be distressing if those components were left in a state where they are unusable because the FIs are so much newer than the blade enclosure and its servers. So I guess we need to see about warming the ZIP code 5degC by attaching the blade chassis to the mains power and see what we can see.

=== Correct cabling for single FI to UCS 5108 ===
According to https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/chassis-install-guide/ucs5108_install/ucs5108_install_chapter_010.html, a single Fabric Interconnect can only be connected to a single Fabric Extender on a UCS chassis. So for now, enable 6248UP ports 1 and 3, and cable to ports 1 and 2 on the left side (as seen from the rear) Fabric Extender.

=== SFPs go into the FI and the UCS 5108 FEXes ===
Pretty self explanatory, right. Find Cisco 10GbE SR SFP+ modules, insert into port one on each of the FEXes, insert into ports 1 and 3 on the FI. Attach nice aqua colored cable. And don't panic as the lights don't come on.

=== Telling the FI to discover the blade chassis ===
This is way more complicated that it needs to be. I'm sure it's totally amazing if you have racks full of Cisco UCS hardware. But I have just the one blade chassis and the one C series server and the 2 fabric interconnects. And the friendly FI management software is all lovely out of date (completely unrunnable in 2026!) Java stuff. So fumbling around the CLI, hoping to find my way out of the darkness here.

The process for this seems to be something like:
* Configure FI Ethernet ports as server ports
* Ensure the attached chassis has been discovered and acknowledge any faults

ucs-fi-6248up-0-A# '''scope eth-server'''
ucs-fi-6248up-0-A /eth-server # '''show fabric a detail'''

Fabric:
Id: A
Current Task:

ucs-fi-6248up-0-A /eth-server # '''scope fabric a'''
ucs-fi-6248up-0-A /eth-server/fabric # '''create interface 1 1'''
ucs-fi-6248up-0-A /eth-server/fabric/interface* # '''enable'''
ucs-fi-6248up-0-A /eth-server/fabric/interface* # '''show interface detail'''
ucs-fi-6248up-0-A /eth-server/fabric/interface* # '''commit-buffer'''
ucs-fi-6248up-0-A /eth-server/fabric/interface # '''exit'''
ucs-fi-6248up-0-A /eth-server/fabric # '''create interface 1 3'''
ucs-fi-6248up-0-A /eth-server/fabric/interface* # '''enable'''
ucs-fi-6248up-0-A /eth-server/fabric/interface* # '''commit-buffer'''
ucs-fi-6248up-0-A /eth-server/fabric/interface # '''exit'''
ucs-fi-6248up-0-A /eth-server/fabric # '''exit'''
ucs-fi-6248up-0-A /eth-server # '''show interface'''

Interface:

Fabric Slot Port Admin State Oper State State Reason Chassis Lic State Grace Prd
------- ----- ----- ------------ ------------ ------------------------------- -------- -------------------- ---------
A 1 1 Enabled Up 1 License Ok 0
A 1 3 Enabled Up 1 License Ok 0
ucs-fi-6248up-0-A /eth-server # '''exit'''
ucs-fi-6248up-0-A# '''show chassis inventory'''
Chassis PID Vendor Serial (SN) HW Revision
---------- --------------- ----------------- ----------- -----------
1 N20-C6508 Cisco Systems Inc FOX1808GLJG 0
ucs-fi-6248up-0-A#
Success! Serial number reported in the UCS Manager matches the one I found on the label.

== So how do I power on a server? ==
Still fumbling through here. But: servers (blade servers for sure, not sure about rack servers) belong to UCS "Organizations." These organizations are managed by the UCS Manager, too. And since an organization owns a server (blade or otherwise) the organization can power servers on and off.

=== What orgs exist? ===
This is what we get from a factory default UCS Manager:
ucs-fi-6248up-0-A# '''show org'''

Organizations:
Name
----
/ (root)
ucs-fi-6248up-0-A# '''show org detail'''

Organizations:
Name: / (root)
ucs-fi-6248up-0-A#

=== Enter an org, see what we have for basic policies ===
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''show bladeserver-disc-policy'''
ucs-fi-6248up-0-A /org # '''show chassis-conn-policy'''

Chassis/FEX Connectivity Policy:
Chassis/FEX Id Chassis/FEX Name Fabric Id Link Aggregation Preference
-------------- ---------------- --------- ---------------------------
1 sys/chassis-1 A Global
ucs-fi-6248up-0-A /org # '''show chassis-disc-policy'''

Chassis/FEX Discovery Policy:
Description Qualifier Action Rebalance Link Aggregation Pref Multicast Hw Hash
----------- ---------- ----------------- ----------------- --------------------- -----------------
none 1 Link User Acknowledged None Disabled
ucs-fi-6248up-0-A /org # '''show chassis-disc-policy detail'''

Chassis/FEX Discovery Policy:
Description:
Qualifier: none
Action: 1 Link
Rebalance: User Acknowledged
Link Aggregation Pref: None
Multicast Hw Hash: Disabled
ucs-fi-6248up-0-A /org # '''show rackserver-disc-policy'''

Rack Server Discovery Policy:
Action Scrub Policy
----------------- ------------
Immediate
ucs-fi-6248up-0-A /org # '''show rackserver-disc-policy detail'''

Rack Server Discovery Policy:
Action: Immediate
Scrub Policy:
Description:
Current Task:
ucs-fi-6248up-0-A /org # show '''rackserver-mgmt-policy'''

Rack Server Management Policy:
Action
------
Auto Acknowledged
ucs-fi-6248up-0-A /org # '''show rackserver-mgmt-policy detail'''

Rack Server Management Policy:
Action: Auto Acknowledged
ucs-fi-6248up-0-A /org #

=== Create a profile and add a (blade) server to it ===
ucs-fi-6248up-0-A# '''show chassis inventory 1'''
Chassis PID Vendor Serial (SN) HW Revision
---------- --------------- ----------------- ----------- -----------
1 N20-C6508 Cisco Systems Inc FOX1808GLJG 0
ucs-fi-6248up-0-A# '''show chassis inventory 1 expand '''
Chassis 1:
Servers:
Server 1/1:
Equipped Product Name: Cisco UCS B200 M3
Equipped PID: UCSB-B200-M3
Equipped VID: V03
Equipped Serial (SN): FCH162871NA
Slot Status: Equipped
Acknowledged Product Name: Cisco UCS B200 M3
Acknowledged PID: UCSB-B200-M3
Acknowledged VID: V03
Acknowledged Serial (SN): FCH162871NA
Acknowledged Memory (MB): 196608
Acknowledged Effective Memory (MB): 196608
Acknowledged Cores: 0
Acknowledged Adapters: 2

Server 1/2:
Equipped Product Name: Cisco UCS B200 M3
Equipped PID: UCSB-B200-M3
Equipped VID: V06
Equipped Serial (SN): FCH1811JELC
Slot Status: Equipped
Acknowledged Product Name: Cisco UCS B200 M3
[...]
ucs-fi-6248up-0-A# '''top'''
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''create service-profile'''
ucs-fi-6248up-0-A /org # '''create service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile* # '''associate server 1/1'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile #
=== I can power on now? ===
Well, no. Maybe it should have worked. But for this blade, there was a Power On Self Test (POST) failure. Let's dig in. To power on the server included in the service profile, do this:
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''scope service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile # '''power up'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile # '''top'''
ucs-fi-6248up-0-A #
And let's have a look at its status:
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''show'''

Server:
Slot Overall Status Service Profile Availability
------- --------------------- -------------------- ------------
1 Compute Failed Unavailable
ucs-fi-6248up-0-A /chassis/server # '''show detail'''

Server:
Slot: 1
Name:
User Label:
Overall Status: Compute Failed
Oper Qualifier: Compute Post Failure
Service Profile:
Association: None
Availability: Unavailable
Discovery: Failed
Conn Path: A
Conn Status: A
Managing Instance: A
Admin Power: Policy
Oper Power: Off
Admin State: In Service
Product Name: Cisco UCS B200 M3
PID: UCSB-B200-M3
VID: V03
Vendor: Cisco Systems Inc
Serial (SN): FCH162871NA
HW Revision: 0
Mfg Date: 2012-07-26T00:00:00.000
Part Number: 73-13217-08
Memory (MB): 196608
Effective Memory (MB): 196608
Operating Memory Speed (MHz): 1333
Operating Memory Voltage: Regular Voltage
Cores: 0
Num Of Cores Enabled: 0
Adapters: 2
Eth Host Interfaces: 0
FC Host Interfaces: 0
Burned-In UUID: 00000000-0000-0000-0000-000000000000
Dynamic UUID: 00000000-0000-0000-0000-000000000000
Current Task 1: Checking hardware configuration server 1/1(FSM-STAGE:sam:dme:ComputeBladeDiscover:Sanitize)
Current Task 2:
Current Task 3:
ucs-fi-6248up-0-A /chassis/server # '''show post'''

POST:
Global ID Code Severity Affected Object Description
--------- --------- --------- -------------------------------- -----------
6215 POST-6215 Critical sys/chassis-1/blade-1 Board Programmable version not valid for Processor Type
ucs-fi-6248up-0-A /chassis/server # '''show post detail'''

POST:
Global ID: 6215
Code: POST-6215
Local ID: 1536
Severity: Critical
Affected Object: sys/chassis-1/blade-1
Description: Board Programmable version not valid for Processor Type
Type: server: Cisco Systems Inc UCSB-B200-M3
Recoverable: Non Recoverable
Recovery Action: Refer the CPU upgrade guide.
Timestamp: 2026-03-22T16:17:06.001
ucs-fi-6248up-0-A /chassis/server #
Making an inferences from "Board Programmable version not valid for Processor Type" I am thinking that it does not like the Xeon E5-2650v2 I installed. Now to find a not v2 E5-26xx CPU and see if that works any better. I found a not obviously damaged E5-2609 in storage (er, piles in the garage) and installed that in the blade. Let's see what we have now:
ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # '''scope service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile # '''power up'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile # '''top'''
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''show inventory'''
Server 1/1:
Name:
User Label:
Equipped PID: UCSB-B200-M3
Equipped VID: V03
Equipped Serial (SN): FCH162871NA
Slot Status: Equipped
Acknowledged Product Name: Cisco UCS B200 M3
Acknowledged PID: UCSB-B200-M3
Acknowledged VID: V03
Acknowledged Serial (SN): FCH162871NA
Acknowledged Memory (MB): 16384
Acknowledged Effective Memory (MB): 16384
Acknowledged Cores: 4
Acknowledged Adapters: 2
ucs-fi-6248up-0-A /chassis/server # '''top'''
ucs-fi-6248up-0-A#
Great Success! The Xeon E5-2609 is a 4 core CPU. And there are 4x 4GiByte memory DIMMs installed. So inventory looks good. Huzzah! (And what an utter pain.)

== Server and Infrastructure issues ==
So, we just saw the first one of these above. The B200 M3 blades do not like Xeon E5-26xx v2 processors. At least with the firmware installed at present. Let's figure that one out first. https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/blade-servers/B200M3.html#reference_CD3077C1064743F99C0F9A935778BC2F has a little table listing minimum versions for various components to get the Xeon E5-2600 v2 CPUs to run. These are the required versions for E5-26xx V2 CPUs:
{|
!Software or Firmware
!Minimum Version
|-
|Server CIMC
|2.1(3)
|-
|Server BIOS
|2.1(3)
|-
|Cisco UCS Manager
|2.1(3)
|-
|Board controller firmware
|8.0
|}
Let's see if we can see what we are running in one of the blades I have and the UCS Manager itself:
ucs-fi-6248up-0-A# '''scope chassis 1'''
ucs-fi-6248up-0-A /chassis # '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''show firmware cimc'''
Server Running-Vers Package-Vers Update-Status Activate-Status
------- --------------- --------------- --------------- ---------------
1/1 3.1(23c) Ready Ready
ucs-fi-6248up-0-A /chassis/server # '''show firmware bios'''
Server 1/1:
BIOS:
Running-Vers: B200M3.2.2.6d.0.062220160055
Package-Vers:
Update-Status: Ready
Activate-Status: Ready

ucs-fi-6248up-0-A /chassis/server # '''show firmware boardcontroller'''

Management Controller:

Server Running-Vers Package-Vers Activate-Status
------- --------------- --------------- ---------------
1/1 15.0 Ready
ucs-fi-6248up-0-A /chassis/server # '''top'''
ucs-fi-6248up-0-A# '''show version brief'''
System version: 2.2(8m)
Package-Vers: 2.2(8m)A
ucs-fi-6248up-0-A#
So that is odd. The requirements listed in Cisco's documentation for supporting Xeon E5-26xx v2 CPUs '''are''' met. But the blade complains about "Board Programmable version not valid for Processor Type". Maybe the blade server's CIMC could tell us more?
== CIMC networking ==
This is going to get us started with attaching the Fabric Interconnect(s) to the rest of the network. Hopefully things go mostly OK. The network at large is all Brocade and Mellanox gear. But here's the strategy statement:
* Create a Link Aggregation Group interface on the appropriate top-of-rack Brocade ICX switches. Pipe all VLANs we care about (management and generic server, 10 and 1000) to that LAG.
* Configure the Fabric Interconnect with some uplink ports to connect to the Brocade LAG interfaces.
* Configure some VLANs on the FI uplink ports ???
* Plumb the management VLAN to the blade servers' CIMC network.
=== Brocade config ===
Fairly simple, really. Add this to the switch's config (from elevated privileges prompt (enable mode) do <code>configure terminal</code> and save (<code>write memory</code>) when done.
lag rack-2-to-ucs-fi-6248up-0 dynamic id 3
ports ethernet 1/3/2 ethernet 2/3/2
primary-port 1/3/2
deploy
And to validate:
SSH@brocade-icx6610-48p-rack-2#'''show lag rack-2-to-ucs-fi-6248up-0'''
Total number of LAGs: 3
Total number of deployed LAGs: 3
Total number of trunks created:3 (117 available)
LACP System Priority / ID: 1 / 748e.f8dd.6228
LACP Long timeout: 120, default: 120
LACP Short timeout: 3, default: 3

=== LAG "rack-2-to-ucs-fi-6248up-0" ID 3 (dynamic Deployed) ===
LAG Configuration:
Ports: e 1/3/2 e 2/3/2
Port Count: 2
Primary Port: 1/3/2
Trunk Type: hash-based
LACP Key: 20003
Deployment: HW Trunk ID 3
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/3/2 Down None None None 3 Yes N/A 0 748e.f8dd.6264
2/3/2 Down None None None 3 Yes N/A 0 748e.f8dd.6264

Port [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/3/2 1 1 20003 Yes S Agg Syn No No Def No Dwn
2/3/2 1 1 20003 Yes S Agg Syn No No Def No Dwn

Partner Info and PDU Statistics
Port Partner Partner LACP LACP
System ID Key Rx Count Tx Count
1/3/2 1-0000.0000.0000 129 0 0
2/3/2 1-0000.0000.0000 385 0 0

SSH@brocade-icx6610-48p-rack-2#
So that was not terrible.
=== UCS Fabric Interconnect uplink ports ===

Need to make some. Kinda like this:
ucs-fi-6248up-0-A# '''scope eth-uplink'''
ucs-fi-6248up-0-A /eth-uplink # '''show fabric a'''

Fabric:
Id
--
A
ucs-fi-6248up-0-A /eth-uplink # '''show fabric a detail'''

Fabric:
Id: A
Current Task 1:
Current Task 2:
ucs-fi-6248up-0-A /eth-uplink # '''show fabric a expand'''

Fabric:
Id: A
ucs-fi-6248up-0-A /eth-uplink # '''scope fabric a'''
ucs-fi-6248up-0-A /eth-uplink/fabric # '''create interface 1 17'''
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # '''enable'''
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # '''commit-buffer'''
ucs-fi-6248up-0-A /eth-uplink/fabric/interface # '''up'''
ucs-fi-6248up-0-A /eth-uplink/fabric # '''create interface 1 19'''
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # '''enable'''
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # '''commit-buffer '''
ucs-fi-6248up-0-A /eth-uplink/fabric/interface # '''exit'''
ucs-fi-6248up-0-A /eth-uplink/fabric # '''show interface '''

Interface:

Slot Id Port Id Admin State Oper State Lic State Grace Period State Reason Ethernet Link Profile name Oper Ethernet Link Profile name
---------- ---------- ----------- ---------------- -------------------- --------------- ------------ -------------------------- -------------------------------
1 17 Enabled Up License Ok 0 default fabric/lan/eth-link-prof-default
1 19 Enabled Up License Ok 0 default fabric/lan/eth-link-prof-default
ucs-fi-6248up-0-A /eth-uplink/fabric #

== Another UCS Manager (and Fabric Interconnect) update ==
I have acquired a pair of B200 M4 blades. And the FI (and UCS Manager) need upgrading to support them. Hopefully nothing breaks in the process. Hopefully...

Now that we are on 2.2(8m), the SSH client is able to fetch the .bin file using scp. Or maybe I just got the syntax correct this time. So here we are:

ucs-fi-6248up-0-A# '''show version'''
System version: 2.2(8m)
ucs-fi-6248up-0-A# '''top'''
ucs-fi-6248up-0-A# '''scope firmware'''
ucs-fi-6248up-0-A /firmware # '''download image scp://adj@172.17.0.18/home/adj/Downloads/ucs-k9-bundle-infra.2.5.2a.A.bin'''
Password:
ucs-fi-6248up-0-A /firmware # '''show download-task'''

Download task:
File Name Protocol Server Userid State
--------- -------- --------------- --------------- -----
ucs-k9-bundle-infra.2.2.8m.A.bin
Tftp 172.17.0.18 Downloaded
ucs-k9-bundle-infra.2.5.2a.A.bin
Scp 172.17.0.18 adj Downloading
ucs-fi-6248up-0-A /firmware # '''show download-task'''

Download task:
File Name Protocol Server Userid State
--------- -------- --------------- --------------- -----
ucs-k9-bundle-infra.2.2.8m.A.bin
Tftp 172.17.0.18 Downloaded
ucs-k9-bundle-infra.2.5.2a.A.bin
Scp 172.17.0.18 adj Downloading
ucs-fi-6248up-0-A /firmware # '''show download-task'''

Download task:
File Name Protocol Server Userid State
--------- -------- --------------- --------------- -----
ucs-k9-bundle-infra.2.2.8m.A.bin
Tftp 172.17.0.18 Downloaded
ucs-k9-bundle-infra.2.5.2a.A.bin
Scp 172.17.0.18 adj Downloaded
ucs-fi-6248up-0-A /firmware # '''scope auto-install
ucs-fi-6248up-0-A /firmware/auto-install # '''install infra infra-vers 2.5(2a)A'''
This operation upgrades firmware on UCS Infrastructure Components
(UCS manager, Fabric Interconnects and IOMs).
Here is the checklist of things that are recommended before starting Auto-Install
(1) Review current critical/major faults
(2) Initiate a configuration backup
(3) Check if Management Interface Monitoring Policy is enabled
(4) Check if there is a pending Fabric Interconnect Reboot activitiy
(5) Ensure NTP is configured
(6) Check if any hardware (fabric interconnects, io-modules, servers or adapters) is unsupported in the target release
Do you want to proceed? (yes/no):'''yes'''

Triggering Install-Infra with:
Infrastructure Pack Version: 2.5(2a)A

ucs-fi-6248up-0-A /firmware/auto-install # '''acknowledge primary fabric-interconnect reboot'''
ucs-fi-6248up-0-A /firmware/auto-install* # '''commit-buffer'''
ucs-fi-6248up-0-A /firmware/auto-install # '''show fsm status expand'''

FSM Status:

Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: Success
Completion Time: 2009-01-01T01:59:13.146
Progress (%): 100

FSM Stage:

Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployDownloadImages Skip 0
6 DeployCopyAllImagesToPeer Skip 0
7 DeployInternalBackup Success 0
8 DeployPollInternalBackup Success 2
9 DeployActivateUCSM Skip 0
10 DeployPollActivateOfUCSM Success 0
11 DeployUpdateIOM Success 0
12 DeployPollUpdateOfIOM Success 0
13 DeployActivateIOM Success 0
14 DeployPollActivateOfIOM Success 0
15 DeployActivateRemoteFI Skip 0
16 DeployPollActivateOfRemoteFI Skip 0
17 DeployWaitForUserAck Skip 0
18 DeployPollWaitForUserAck Success 0
19 DeployActivateLocalFI Success 0
20 DeployPollActivateOfLocalFI Success 2
ucs-fi-6248up-0-A /firmware/auto-install # '''show fsm status expand | no-more'''

FSM Status:

Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: In Progress
Completion Time:
Progress (%): 95

FSM Stage:

Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployDownloadImages Skip 0
6 DeployCopyAllImagesToPeer Skip 0
7 DeployInternalBackup Success 0
8 DeployPollInternalBackup In Progress 1
9 DeployActivateUCSM Pending 0
10 DeployPollActivateOfUCSM Pending 0
11 DeployUpdateIOM Pending 0
12 DeployPollUpdateOfIOM Pending 0
13 DeployActivateIOM Pending 0
14 DeployPollActivateOfIOM Pending 0
15 DeployActivateRemoteFI Pending 0
16 DeployPollActivateOfRemoteFI Pending 0
17 DeployWaitForUserAck Pending 0
18 DeployPollWaitForUserAck Pending 0
19 DeployActivateLocalFI Pending 0
20 DeployPollActivateOfLocalFI Pending 0
ucs-fi-6248up-0-A /firmware/auto-install # '''show fsm status expand | no-more'''

FSM Status:

Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: In Progress
Completion Time:
Progress (%): 96

FSM Stage:

Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployDownloadImages Skip 0
6 DeployCopyAllImagesToPeer Skip 0
7 DeployInternalBackup Success 0
8 DeployPollInternalBackup Success 2
9 DeployActivateUCSM Success 0
10 DeployPollActivateOfUCSM In Progress 1
11 DeployUpdateIOM Pending 0
12 DeployPollUpdateOfIOM Pending 0
13 DeployActivateIOM Pending 0
14 DeployPollActivateOfIOM Pending 0
15 DeployActivateRemoteFI Pending 0
16 DeployPollActivateOfRemoteFI Pending 0
17 DeployWaitForUserAck Pending 0
18 DeployPollWaitForUserAck Pending 0
19 DeployActivateLocalFI Pending 0
20 DeployPollActivateOfLocalFI Pending 0
ucs-fi-6248up-0-A /firmware/auto-install # Connection to 172.16.10.176 closed.
$ '''ssh -o HostKeyAlgorithms=+ssh-rsa admin@172.16.10.176'''
Cisco UCS 6200 Series Fabric Interconnect
(admin@172.16.10.176) Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

ucs-fi-6248up-0-A# '''show version'''
System version: 2.5(2a)
ucs-fi-6248up-0-A# '''show version brief'''
System version: 2.5(2a)
Package-Vers: 2.5(2a)A
ucs-fi-6248up-0-A#
And after some more time and one more disconnect, we see that another OpenSSH client compatibility option is required to connect:
$ '''ssh -o HostKeyAlgorithms=+ssh-rsa admin@172.16.10.176'''
Unable to negotiate with 172.16.10.176 port 22: no matching cipher found. Their offer: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se
$ '''ssh -o HostKeyAlgorithms=+ssh-rsa -o Ciphers=+aes256-cbc admin@172.16.10.176'''
Cisco UCS 6200 Series Fabric Interconnect
(admin@172.16.10.176) Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

ucs-fi-6248up-0-A# '''show version'''
System version: 2.5(2a)
ucs-fi-6248up-0-A# '''show version brief'''
System version: 2.5(2a)
Package-Vers: 2.5(2a)A
ucs-fi-6248up-0-A#

== But I'm still trying to get the blades' CIMCs connected! ==
The struggle is real! But I did find https://www.cisco.com/c/en/us/support/docs/servers-unified-computing/integrated-management-controller/118367-configure-cimc-00.pdf which is a GUI-specific outline of the process involved in getting UCS servers' (blade and maybe also rackmount) CIMCs attached to the greater network environment.
=== In-band? Out-of-band? Huh? ===
First decision to be made here. In UCS versions before 2.2, the servers' CIMCs always had Ethernet connectivity through the Fabric Interconnects' management port. This connection amounts to a single point of failure for CIMC connectivity.

In UCS version 2.2 and newer, there are two options for CIMC connectivity: In-band using the FIs' high bandwidth network uplink ports, and out-of band using an FI's 1000baseT management port. Naturally, being me, I will opt for the more complicated alternative. And probably complain the whole way through.

=== In-band CIMC connectivity setup ===
Log in to the FI. Go to the organization (probably <code>/</code>. And create an IP pool. We are skipping the IPv6 stuff for now and the management VLAN is IPv4-only.
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''show ip-pool'''

IP Pool:
Name Size Assigned Management mode
-------------------- ---------- ---------- ---------------
ext-mgmt 0 0 Internal
iscsi-initiator-pool 0 0 Internal
ucs-fi-6248up-0-A /org # '''create ip-pool CIMC_inband_pool'''
ucs-fi-6248up-0-A /org/ip-pool* # '''create block 172.16.10.168 172.16.10.175 172.16.10.1 255.255.255.0'''
ucs-fi-6248up-0-A /org/ip-pool/block* # '''set primary-dns 172.16.10.1 secondary-dns 172.16.10.2'''
ucs-fi-6248up-0-A /org/ip-pool/block* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/ip-pool/block # '''show detail'''

Block of IP Addresses:
From: 172.16.10.168
To: 172.16.10.175
Default Gateway: 172.16.10.1
Subnet Mask: 255.255.255.0
Primary DNS: 172.16.10.1
Secondary DNS: 172.16.10.2
ucs-fi-6248up-0-A /org/ip-pool/block # '''show expand'''

Block of IP Addresses:
From To Default Gateway Subnet Mask
--------------- --------------- --------------- -----------
172.16.10.168 172.16.10.175 172.16.10.1 255.255.255.0
ucs-fi-6248up-0-A /org/ip-pool/block # '''up'''
ucs-fi-6248up-0-A /org/ip-pool # '''show'''

IP Pool:
Name Size Assigned Management mode
-------------------- ---------- ---------- ---------------
CIMC_inband_pool 8 0 Internal
ucs-fi-6248up-0-A /org/ip-pool # '''show detail'''

IP Pool:
Name: CIMC_inband_pool
Size: 8
Assigned: 0
IPv4 Size: 8
IPv4 Assigned: 0
IPv6 Size: 0
IPv6 Assigned: 0
Descr:
Assignment Order: Default
Management mode: Internal
Guid: 00000000-0000-0000-0000-000000000000
Net bios enabled or disabled: Not Active
DHCP enaled or disabled: Not Supported
ucs-fi-6248up-0-A /org/ip-pool # '''show expand'''

IP Pool:
Name: CIMC_inband_pool
Size: 8
Assigned: 0
Management mode: Internal

Block of IP Addresses:
From To Default Gateway Subnet Mask
--------------- --------------- --------------- -----------
172.16.10.168 172.16.10.175 172.16.10.1 255.255.255.0

Pooled:
Id Subnet Assigned Assigned To
--------------- --------------- -------- -----------
172.16.10.168 255.255.255.0 No
172.16.10.169 255.255.255.0 No
172.16.10.170 255.255.255.0 No
172.16.10.171 255.255.255.0 No
172.16.10.172 255.255.255.0 No
172.16.10.173 255.255.255.0 No
172.16.10.174 255.255.255.0 No
172.16.10.175 255.255.255.0 No
ucs-fi-6248up-0-A /org/ip-pool #
So we now have a pool of IPv4 addresses to hand out to CIMCs in our UCS domain. And we can also LOL at Cisco's <code>DHCP '''enaled''' or disabled</code> spelling error. But not too loud. It may well be fixed in newer versions.

=== Create VLAN and VLAN group to connect the CIMC IP pool to ===
VLAN on the Ethernet uplinks:
ucs-fi-6248up-0-A /org/ip-pool # '''top'''
ucs-fi-6248up-0-A# '''scope eth-uplink'''
ucs-fi-6248up-0-A /eth-uplink # '''create vlan Management 10'''
ucs-fi-6248up-0-A /eth-uplink/vlan* # '''set native no'''
ucs-fi-6248up-0-A /eth-uplink/vlan* # '''set vlan-id 10'''
ucs-fi-6248up-0-A /eth-uplink/vlan* # '''commit-buffer'''
VLAN group now:
ucs-fi-6248up-0-A /eth-uplink/vlan # '''up'''
ucs-fi-6248up-0-A /eth-uplink # '''show vlan-group'''
ucs-fi-6248up-0-A /eth-uplink # '''create vlan-group Management_group'''
ucs-fi-6248up-0-A /eth-uplink/vlan-group* # '''create member-vlan Management'''
ucs-fi-6248up-0-A /eth-uplink/vlan-group/member-vlan* # '''commit-buffer'''
Add IP pool, VLAN, VLAN group to in-band profile:
ucs-fi-6248up-0-A /eth-uplink/vlan-group/member-vlan # '''top'''
ucs-fi-6248up-0-A# '''scope eth-uplink'''
ucs-fi-6248up-0-A /eth-uplink # '''scope inband-profile'''
ucs-fi-6248up-0-A /eth-uplink/inband-profile # '''set default-pool-name CIMC_inband_pool'''
ucs-fi-6248up-0-A /eth-uplink/inband-profile* # '''set default-vlan-name Management'''
ucs-fi-6248up-0-A /eth-uplink/inband-profile* # '''set net-group-name Management_group'''
ucs-fi-6248up-0-A /eth-uplink/inband-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /eth-uplink/inband-profile #
Now that the in-band profile is set, update a blade's service profile to reference it:
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''show service-profile'''

Service Profile:
Service Profile Name Type Server Assignment Association
-------------------- ----------------- ------- ---------- -----------
UCS_B200_M3_FCH162871NA
Instance 1/1 Assigned Associating
ucs-fi-6248up-0-A /org # '''scope service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile # '''set ext-mgmt-ip-pool-name CIMC_inband_pool'''
ucs-fi-6248up-0-A /org/service-profile* # '''set ext-mgmt-ip-state pooled'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A# '''top'''
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''reset hard-reset-immediate'''
ucs-fi-6248up-0-A /chassis/server* # '''commit-buffer'''
ucs-fi-6248up-0-A /chassis/server # '''reset-kvm'''
ucs-fi-6248up-0-A /chassis/server* # '''commit-buffer'''
ucs-fi-6248up-0-A /chassis/server* # '''reset-ipmi'''
ucs-fi-6248up-0-A /chassis/server* # '''commit-buffer'''
ucs-fi-6248up-0-A /chassis/server # '''scope cimc'''
ucs-fi-6248up-0-A /chassis/server/cimc # '''show'''

CIMC:
PID Serial (SN) HW Revision
---------------- ---------------- -----------
UCSB-B200-M3 FCH162871NA 0
ucs-fi-6248up-0-A /chassis/server/cimc # '''show detail'''

CIMC:
Product Name: Cisco UCS B200 M3
PID: UCSB-B200-M3
VID: V01
Vendor: Cisco Systems Inc
Serial (SN): FCH162871NA
HW Revision: 0
GUID:
Current Task:
ucs-fi-6248up-0-A /chassis/server/cimc # '''show expand'''

CIMC:
PID: UCSB-B200-M3
Serial (SN): FCH162871NA
HW Revision: 0

Vmedia Mapping List:
Full Name: sys/chassis-1/blade-1/mgmt/actual-mount-list

Vmedia Mapping:
Vdisk Id Mapping Name Device Type Mount Protocol Mount Status
-------- ------------ ----------- -------------- ------------
1 Cdd Unknown Not Mounted
2 Hdd Unknown Not Mounted

External Management Interface:
Mode: In Band
Ip V4 State: Pooled
Ip V6 State: Pooled

External Management Virtual LAN:
Network Name: Management
Id: 10

External Management Pooled IP:
Name IP Address Default Gateway Subnet Primary DNS IP Secondary DNS IP
---------- --------------- --------------- --------------- --------------- ----------------
CIMC_inband_pool
172.16.10.169 172.16.10.1 255.255.255.0 172.16.10.1 172.16.10.2

External Management Pooled IPv6:
Name IP Address Default Gateway Prefix Primary DNS IP Secondary DNS IP
---------- ---------- --------------- ------ -------------- ----------------
CIMC_inband_pool
:: :: 64 :: ::

Management Interface:
Access Type: Unspecified
IP Address: 0.0.0.0
Netmask: 0.0.0.0
Gateway: 0.0.0.0
MAC Address: 30:F7:0D:BE:89:EA
Fabric ID: A
Peer Port:
Peer Port Id: 1
Peer Slot Id: 1
Peer Chassis Id: 1
Discovery: Absent

Management Endpoint Log Control:
Type ID Capacity
------ --- --------
SEL 0 Available
ucs-fi-6248up-0-A /chassis/server/cimc #
(Some) Success! The blade's CIMC is assigned an IPv4 address. And it is pingable on the local network. SSH is a no-go, though. Maybe not a thing at all on B series blades? And some further fiddling seems to be needed to get IPMI over LAN working. Something like this:
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''create ipmi-access-profile ipmi_profile'''
ucs-fi-6248up-0-A /org/ipmi-access-profile* # '''create ipmi-user admin'''
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user* # '''set password'''
Enter a password:
Confirm the password:
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user # '''set privilege admin'''
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user # '''up'''
ucs-fi-6248up-0-A /org/ipmi-access-profile # '''set ipmi-over-lan enable'''
ucs-fi-6248up-0-A /org/ipmi-access-profile # '''commit-buffer'''
ucs-fi-6248up-0-A /org/ipmi-access-profile # '''up'''
ucs-fi-6248up-0-A /org # '''scope service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile # '''set ipmi-access-profile ipmi_profile'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile # '''top'''
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''scope cimc'''
ucs-fi-6248up-0-A /chassis/server/cimc # '''reset'''
ucs-fi-6248up-0-A /chassis/server/cimc* # '''commit-buffer'''
ucs-fi-6248up-0-A /chassis/server/cimc # '''top'''
ucs-fi-6248up-0-A# '''connect cimc 1/1'''
Trying 127.5.1.1...
Connected to 127.5.1.1.
Escape character is '^]'.

CIMC Debug Firmware Utility Shell [ support ]
[ help ]# '''help'''
__________________________________________
Debug Firmware Utility
__________________________________________
Command List
__________________________________________
alarms
cores
exit
i2cstats
images
mctools
memory
messages
mrcout
network
obfl
post
power
programmables
sensors
sel
fru
tasks
top
update
users
version
mezz1fru
mezz2fru
sldp
help [COMMAND]
__________________________________________
Notes:
"enter Key" will execute last command
"COMMAND ?" will execute help for that command
__________________________________________
[ help ]# '''users'''
0. [ "admin" ] [ "" ] [ 2 ]
1. [ Empty Entry ]
2. [ Empty Entry ]
3. [ Empty Entry ]
4. [ Empty Entry ]
5. [ Empty Entry ]
6. [ Empty Entry ]
7. [ Empty Entry ]
8. [ Empty Entry ]
9. [ Empty Entry ]
10. [ Empty Entry ]
11. [ Empty Entry ]
12. [ Empty Entry ]
13. [ Empty Entry ]
14. [ Empty Entry ]
15. [ Empty Entry ]
[ users ]# '''exit'''
Connection closed by foreign host.
ucs-fi-6248up-0-A#

So that looks like we should have IPMI over LAN. Let's poke at it a bit and see what happens:
$ '''ping -c 5 172.16.10.169'''
PING 172.16.10.169 (172.16.10.169) 56(84) bytes of data.
64 bytes from 172.16.10.169: icmp_seq=1 ttl=63 time=0.930 ms
64 bytes from 172.16.10.169: icmp_seq=2 ttl=63 time=0.655 ms
64 bytes from 172.16.10.169: icmp_seq=3 ttl=63 time=0.677 ms
64 bytes from 172.16.10.169: icmp_seq=4 ttl=63 time=0.658 ms
64 bytes from 172.16.10.169: icmp_seq=5 ttl=63 time=0.732 ms

--- 172.16.10.169 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4058ms
rtt min/avg/max/mdev = 0.655/0.730/0.930/0.103 ms
$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin chassis status'''
Password:
Error: Unable to establish IPMI v2 / RMCP+ session
$
And that is a promising start with a disappointing finish. Just for grins, let's try from a machine on the same subnet. Maybe ICMP echo replies get routed but the IPMI over LAN traffic doesn't?
router$ '''ping -c 5 172.16.10.169'''
PING 172.16.10.169 (172.16.10.169) 56(84) bytes of data.
64 bytes from 172.16.10.169: icmp_seq=1 ttl=64 time=1.85 ms
64 bytes from 172.16.10.169: icmp_seq=2 ttl=64 time=0.498 ms
64 bytes from 172.16.10.169: icmp_seq=3 ttl=64 time=0.463 ms
64 bytes from 172.16.10.169: icmp_seq=4 ttl=64 time=0.452 ms
64 bytes from 172.16.10.169: icmp_seq=5 ttl=64 time=0.460 ms

--- 172.16.10.169 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4067ms
rtt min/avg/max/mdev = 0.452/0.745/1.853/0.554 ms
router$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin mc info'''
Password:
Device ID : 32
Device Revision : 0
Firmware Revision : 3.01
IPMI Version : 2.0
Manufacturer ID : 5771
Manufacturer Name : Cisco Systems, Inc.
Product ID : 9 (0x0009)
Product Name : Unknown (0x09)
Device Available : yes
Provides Device SDRs : yes
Additional Device Support :
Sensor Device
SDR Repository Device
SEL Device
FRU Inventory Device
IPMB Event Receiver
IPMB Event Generator
Aux Firmware Rev Info :
0x00
0x00
0x2a
0x8b
router$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin lan print'''
Password:
Set in Progress : Set Complete
Auth Type Support : NONE MD2 MD5 PASSWORD
Auth Type Enable : Callback :
: User :
: Operator :
: Admin :
: OEM :
IP Address Source : Static Address
IP Address : 0.0.0.0
Subnet Mask : 0.0.0.0
MAC Address : 30:f7:0d:be:89:ea
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x00 Precedence=0x00 TOS=0x10
BMC ARP Control : ARP Responses Disabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 2.0 seconds
Default Gateway IP : 0.0.0.0
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,128
Cipher Suite Priv Max : XXXaaaXXaaaXaaa
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
Bad Password Threshold : Not Available
router$
Yay! Very strange that it can be pinged from a different subnet, but <code>ipmitool</code> cannot talk to it that way. Also very strange that it says it has no IP address assigned. I am just guessing here, but perhaps the code in the FI is doing that for us and I have not found the documentation describing how it works.

=== Serial console for blades? ===
I am fine with low bit rate serial stuff. No need for a fancy bit mapped screen to manage my computers. And it saves me having to run crazy Avocent Java applications in the process. So let's see what happens when we try to connect from a machine on the same subnet:
router$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin sol activate'''
Password:
Error activating SOL payload: Invalid data field in request
router$
Disappointing. But I think I might have seem something about serial over LAN in the service profiles somewhere. Let's see if we can get it turned on:
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''show sol-policy'''
ucs-fi-6248up-0-A /org # '''create sol-policy sol_pol'''
ucs-fi-6248up-0-A /org/sol-policy* # '''enable'''
ucs-fi-6248up-0-A /org/sol-policy* # '''set speed 115200'''
ucs-fi-6248up-0-A /org/sol-policy* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/sol-policy # '''up'''
ucs-fi-6248up-0-A /org # '''show service-profile'''

Service Profile:
Service Profile Name Type Server Assignment Association
-------------------- ----------------- ------- ---------- -----------
UCS_B200_M3_FCH162871NA
Instance 1/1 Assigned Associating
ucs-fi-6248up-0-A /org # '''scope service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile # '''set sol-policy sol_pol'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile # '''top'''
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''scope cimc'''
ucs-fi-6248up-0-A /chassis/server/cimc # '''reset'''
ucs-fi-6248up-0-A /chassis/server/cimc* # '''commit-buffer'''
ucs-fi-6248up-0-A /chassis/server/cimc #

And trying again from a machine on the same subnet:
router$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin sol activate'''
Password:
Error activating SOL payload: Invalid data field in request
router$

So still no. Grrrr. Maybe it needs to be powered on first?
router$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin chassis power status'''
Password:
Chassis Power is off
router$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin chassis power on'''
Password:
Chassis Power Control: Up/On
router$ '''sleep 30'''
router$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin sol activate'''
Password:
Error activating SOL payload: Invalid data field in request
router$

More grrrr. Manuals suggest I might need a BIOS policy that enables the serial port, too?
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''create bios-policy serial_is_good'''
ucs-fi-6248up-0-A /org/bios-policy* # '''set quiet-boot-config quiet-boot disabled'''
ucs-fi-6248up-0-A /org/bios-policy* # '''set serial-port-a-config serial-port-a enabled'''
ucs-fi-6248up-0-A /org/bios-policy* # '''set console-redir-config baud-rate 115200'''
ucs-fi-6248up-0-A /org/bios-policy* # '''set console-redir-config console-redir serial-port-a'''
ucs-fi-6248up-0-A /org/bios-policy* # '''set console-redir-config flow-control none'''
ucs-fi-6248up-0-A /org/bios-policy* # '''set console-redir-config legacy-os-redir disabled'''
ucs-fi-6248up-0-A /org/bios-policy* # '''set console-redir-config terminal-type vt100-plus'''
ucs-fi-6248up-0-A /org/bios-policy* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/bios-policy # '''up'''
ucs-fi-6248up-0-A /org # '''show service-profile'''

Service Profile:
Service Profile Name Type Server Assignment Association
-------------------- ----------------- ------- ---------- -----------
UCS_B200_M3_FCH162871NA
Instance 1/1 Assigned Associating
ucs-fi-6248up-0-A /org # '''scope service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile # '''set bios-policy serial_is_good'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile # '''top'''
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''scope cimc'''
ucs-fi-6248up-0-A /chassis/server/cimc # '''reset'''
ucs-fi-6248up-0-A /chassis/server/cimc* # '''commit-buffer'''
ucs-fi-6248up-0-A /chassis/server/cimc # '''top'''
ucs-fi-6248up-0-A#
So there is still no working serial console on the blade at this point. https://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-b-series-blade-servers/200105-Utilizing-SoL-logging-for-Serial-Redirec.html says to SSH to the CIMC IP address. Just getting a TCP RST back ("connection refused") when trying that. And the IPMI over LAN serial just does not seem to be supported at all. Maybe the blade needs a firmware update, too?
=== Applying firmware update to blades ===
So far (I think) I have only done firmware updates to the fabric interconnect and the blade chassis FEXes. First up, let's see what is available for our B200 M3 blades:
ucs-fi-6248up-0-A# '''scope firmware'''
ucs-fi-6248up-0-A /firmware # '''show image | i b200-m3'''
ucs-b200-m3-bios.B200M3.2.0.4a.0.080920121557.bin
ucs-b200-m3-bios.B200M3.2.2.1a.0.111220131105.bin
ucs-b200-m3-brdprog.11.0.bin Board Controller 11.0
ucs-b200-m3-k9-cimc.2.0.4b.bin CIMC 2.0(4b)
ucs-b200-m3-k9-cimc.2.2.1c.bin CIMC 2.2(1c)
ucs-b200-m3-mrsasctlr.20.10.1-0100_4.30.00_NA.bin
ucs-b200-m3-mrsasctlr.20.12.1-0160_4.37.00_NA.bin
ucs-fi-6248up-0-A /firmware #
Those might be kinda old? Let's see what is installed on the B200 M3 that is in the chassis:
ucs-fi-6248up-0-A /firmware # '''top'''
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''show firmware'''
Server 1/1:
Adapter 1:
Running-Vers: 4.1(3a)
Package-Vers:
Update-Status: Ready
Activate-Status: Ready
Adapter 2:
Running-Vers: 4.1(3a)
Package-Vers:
Update-Status: Ready
Activate-Status: Ready
BIOS:
Running-Vers: B200M3.2.2.6d.0.062220160055
Package-Vers:
Update-Status: Ready
Activate-Status: Ready

RAID Controller 1:
Running-Vers: 20.13.1-0255
Package-Vers:
Activate-Status: Ready

BoardController:
Running-Vers: 15.0
ucs-fi-6248up-0-A /chassis/server #
From this, it would seem that the blade in the chassis is already running a newer firmware bundle than the one running on the fabric interconnect. So, let's find out what we are running on the FI, and see about some more updates to it:
ucs-fi-6248up-0-A /chassis/server # '''top'''
ucs-fi-6248up-0-A# '''show version brief'''
System version: 2.5(2a)
Package-Vers: 2.5(2a)A
ucs-fi-6248up-0-A#
It would seem that 3.0(2f) is the latest in the next-newer release train. So acquire that from Cisco's download site and install on the FI and apply following the process documented above. And there is a hiccup with the SSH in 2.5(2a) not being able to get a host key from the SSH servers running on Debian 12 or Debian 13. Here's an example Debian server log of that:
adj@zarathud:~$ '''sudo grep ssh.*172.16.10.176 /var/log/auth.log'''
2026-04-02T17:24:54.156462+00:00 zarathud sshd[3090225]: Unable to negotiate with 172.16.10.176 port 33905: no matching host key type found. Their offer: x509v3-sign-rsa,x509v3-sign-dss,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss [preauth]
2026-04-02T17:41:20.766493+00:00 zarathud sshd[3092657]: Unable to negotiate with 172.16.10.176 port 58959: no matching host key type found. Their offer: x509v3-sign-rsa,x509v3-sign-dss,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss [preauth]
2026-04-02T17:50:44.154486+00:00 zarathud sshd[3094045]: Unable to negotiate with 172.16.10.176 port 45452: no matching host key type found. Their offer: x509v3-sign-rsa,x509v3-sign-dss,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss [preauth]
2026-04-02T17:51:36.645000+00:00 zarathud sudo: adj : TTY=pts/22 ; PWD=/home/adj ; USER=root ; COMMAND=/usr/bin/grep ssh.*172.16.10.176 /var/log/auth.log
2026-04-02T17:53:33.561802+00:00 zarathud sudo: adj : TTY=pts/22 ; PWD=/home/adj ; USER=root ; COMMAND=/usr/bin/grep ssh.*172.16.10.176 /var/log/auth.log
adj@zarathud:~$
So, for expediency's sake, we will just do it over TFTP this time. Maybe the SSH client in the 3.1(3l) release will be better able to talk to my SSH servers? That last remains to be seen. But the SSH server is a bit nearer to 2026 standards:
adj@yetanotherthinclient:~$ '''ssh admin@172.16.10.176'''
Unable to negotiate with 172.16.10.176 port 22: no matching host key type found. Their offer: ssh-rsa
adj@yetanotherthinclient:~$ '''ssh -o HostKeyAlgorithms=+ssh-rsa admin@172.16.10.176'''
Cisco UCS 6200 Series Fabric Interconnect
(admin@172.16.10.176) Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

ucs-fi-6248up-0-A# '''show version'''
System version: 3.1(3l)
ucs-fi-6248up-0-A# '''show version brief'''
System version: 3.1(3l)
Package-Vers: 3.1(3l)A
Service pack version: 3.1(3)SP0(Default)
Running-Modules:
Package-Vers:
ucs-fi-6248up-0-A#
Still only does RSA host keys. But it does support more recent symmetric encryption schemes than the SSH server in 2.5(2a). But major excitement! The web UI now just needs HTML rendering. And ECMAscript. And CSS. But no more need for Java just to see what is going on with the UCS infrastructure. It does seem to still be needed for the KVM. But this is a major step forward. I very strongly recommend getting to the 3.1 UCS Manager releases.

And now for some server (blade) firmware. Available at the same place as the UCS Manager download, a section called "Related Software" has the server firmware images. Download and copy those to the UCS Manager running on the fabric interconnects, too:
ucs-fi-6248up-0-A# '''scope firmware'''
ucs-fi-6248up-0-A /firmware # '''download image scp://adj@172.17.0.18/home/adj/Downloads/ucs-k9-bundle-b-series.3.1.3l.B.bin'''
ucs-fi-6248up-0-A /firmware # '''show download-task'''

Download task:
File Name Protocol Server Userid State
--------- -------- ------------------------------------- --------------- -----
ucs-k9-bundle-b-series.3.1.3l.B.bin
Scp 172.17.0.18 adj Downloading
ucs-k9-bundle-infra.2.2.8m.A.bin
Tftp 172.17.0.18 Downloaded
ucs-k9-bundle-infra.2.5.2a.A.bin
Scp 172.17.0.18 adj Downloaded
ucs-k9-bundle-infra.3.1.3l.A.bin
Tftp 172.17.0.18 Downloaded
ucs-mini-k9-bundle-infra.3.0.2f.A.bin
Tftp 172.17.0.18 Downloaded
ucs-fi-6248up-0-A /firmware # '''show download-task ucs-k9-bundle-b-series.3.1.3l.B.bin detail'''

Download task:
File Name: ucs-k9-bundle-b-series.3.1.3l.B.bin
Protocol: Scp
Server: 172.17.0.18
Userid: adj
Path: /home/adj/Downloads
Downloaded Image Size (KB): 517633
State: Downloading
Current Task: unpacking image ucs-k9-bundle-b-series.3.1.3l.B.bin on primary(FSM-STAGE:sam:dme:FirmwareDownloaderDownload:UnpackLocal)
ucs-fi-6248up-0-A /firmware # '''show download-task ucs-k9-bundle-b-series.3.1.3l.B.bin detail'''

Download task:
File Name: ucs-k9-bundle-b-series.3.1.3l.B.bin
Protocol: Scp
Server: 172.17.0.18
Userid: adj
Path: /home/adj/Downloads
Downloaded Image Size (KB): 517633
State: Downloaded
Current Task:
ucs-fi-6248up-0-A /firmware #
Note that the UCS Manager SSH client can now talk to a Debian 12 OpenSSH server. Small wins are still wins!

=== A new step for firmware infrastructure updates ===
It seems that one must accept that a fabric interconnect reboot has happened before applying a new infrastructure firmware pack. Something like this:
ucs-fi-6248up-0-A# '''scope monitoring'''
ucs-fi-6248up-0-A /monitoring # '''show new-faults'''
Severity Code Last Transition Time ID Description
--------- -------- ------------------------ -------- -----------
Info F0440 2026-04-02T18:37:14.718 113208 Chassis discovery policy conflict: Link IOM 1/1/2 to peer port A:N/A/1/2 not configured
ucs-fi-6248up-0-A /monitoring # '''top'''
ucs-fi-6248up-0-A# '''scope firmware'''
ucs-fi-6248up-0-A /firmware # '''scope auto-install'''
ucs-fi-6248up-0-A /firmware/auto-install # '''acknowledge primary fabric-interconnect reboot '''
ucs-fi-6248up-0-A /firmware/auto-install* # '''commit-buffer'''
ucs-fi-6248up-0-A /firmware/auto-install # '''install infra infra-vers 3.2(3p)A'''
This operation upgrades firmware on UCS Infrastructure Components
(UCS manager, Fabric Interconnects and IOMs).
Here is the checklist of things that are recommended before starting Auto-Install
(1) Review current critical/major faults
(2) Initiate a configuration backup
(3) Check if Management Interface Monitoring Policy is enabled
(4) Check if there is a pending Fabric Interconnect Reboot activitiy
(5) Ensure NTP is configured
(6) Check if any hardware (fabric interconnects, io-modules, servers or adapters) is unsupported in the target release
(7) Some fabric-interconnect service-pack install/uninstall will do additional FI reboots in order to complete install/uninstall
To ensure before triggering auto-install, check manual service-pack activation/remove where warning is given if reboots are required
Do you want to proceed? (yes/no):'''yes'''

Triggering Install-Infra with:
Infrastructure Pack Version: 3.2(3p)A
Warning: Any Service Pack installed in the system will be removed

ucs-fi-6248up-0-A /firmware/auto-install # '''acknowledge primary fabric-interconnect reboot'''
Warning: Check the outstanding faults (scope monitoring <enter> show new-faults) since last FI reboot.
Please make sure the data paths are recovered before proceeding with this FI reboot to ensure there is no interruption to the data traffic.

ucs-fi-6248up-0-A /firmware/auto-install* # '''commit-buffer '''
ucs-fi-6248up-0-A /firmware/auto-install #
Broadcast message from root (Thu Apr 2 23:36:05 2026):

The system is going down for reboot NOW!
Connection to 172.16.10.176 closed by remote host.
Connection to 172.16.10.176 closed.
The UCS Manager SSH server generated itself a new, longer host RSA key as part of the 3.2(3p) upgrade. So expect host key validation failures after the FI reboots and you tell your SSH client to connect again. Maybe we can get back to updates for the blade servers' firmware now?

=== Blade server firmware. Another attempt. ===
This is going to assume that the server (blade) firmware has been copied to the UCS manager. Cisco's filename for the B series blade firmware bundles for the 3.2(3p) release is <code>ucs-k9-bundle-b-series.3.2.3p.B.bin</code>.
ucs-fi-6248up-0-A# '''scope firmware'''
ucs-fi-6248up-0-A /firmware # '''show image | i b200-m3'''
ucs-b200-m3-bios.B200M3.2.0.4a.0.080920121557.bin
ucs-b200-m3-bios.B200M3.2.2.1a.0.111220131105.bin
ucs-b200-m3-bios.B200M3.2.2.6h.0.110720191420.bin
ucs-b200-m3-brdprog.11.0.bin Board Controller 11.0
ucs-b200-m3-brdprog.15.0.bin Board Controller 15.0
ucs-b200-m3-brdprog.16.0.bin Board Controller 16.0
ucs-b200-m3-k9-cimc.2.0.4b.bin CIMC 2.0(4b)
ucs-b200-m3-k9-cimc.2.2.1c.bin CIMC 2.2(1c)
ucs-b200-m3-k9-cimc.3.1.23e.bin CIMC 3.1(23e)
ucs-b200-m3-k9-cimc.3.1.26p.bin CIMC 3.1(26p)
ucs-b200-m3-mrsasctlr.20.10.1-0100_4.30.00_NA.bin
ucs-b200-m3-mrsasctlr.20.12.1-0160_4.37.00_NA.bin
ucs-b200-m3-mrsasctlr.20.13.1-0255_4.38.02.2_NA.bin
ucs-fi-6248up-0-A /firmware # '''top'''
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''show fw-host-pack'''

Server Host Pack:
Name Mode
-------------------- ----
default Staged
test Staged
ucs-fi-6248up-0-A /org # '''create fw-host-pack b200-m3-3.2.3p'''
ucs-fi-6248up-0-A /org/fw-host-pack* # '''create pack-image "Cisco Systems Inc" UCSB-B200-M3 board-controller 16.0'''
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image* # '''set version 16.0'''
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image # '''exit'''
ucs-fi-6248up-0-A /org/fw-host-pack # '''create pack-image "Cisco Systems Inc" UCSB-B200-M3 cimc "3.1(26p)"'''
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image* # '''set version "3.1(26p)"'''
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image # '''exit'''
ucs-fi-6248up-0-A /org/fw-host-pack # '''create pack-image "Cisco Systems, Inc." UCSB-B200-M3 server-bios "B200M3.2.2.6h.0.110720191420"'''
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image* # '''set version "B200M3.2.2.6h.0.110720191420"'''
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image # '''exit'''
ucs-fi-6248up-0-A /org/fw-host-pack # '''show'''

Server Host Pack:
Name Mode
-------------------- ----
b200-m3-3.2.3p Staged
ucs-fi-6248up-0-A /org/fw-host-pack # '''up'''
ucs-fi-6248up-0-A /org # '''show fw-host-pack'''

Server Host Pack:
Name Mode
-------------------- ----
b200-m3-3.2.3p Staged
default Staged
test Staged
ucs-fi-6248up-0-A /org #
'''VERY IMPORTANT DETAIL:''' In the above, see how sometimes the vendor string is "Cisco Systems Inc" and other times it is "Cisco Systems, Inc."? This is very stupid. Very easy to miss. And very important to getting a configuration that will validate.

And it still needs to be added to a service profile:
ucs-fi-6248up-0-A /org # '''scope org /'''
ucs-fi-6248up-0-A /org # '''show service-profile'''

Service Profile:
Service Profile Name Type Server Assignment Association
-------------------- ----------------- ------- ---------- -----------
UCS_B200_M3_FCH162871NA
Instance 1/1 Assigned Associating
ucs-fi-6248up-0-A /org # '''scope service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile # '''set host-fw-policy b200-m3-3.2.3p'''
ucs-fi-6248up-0-A /org/service-profile # '''reapply-config'''
Warning: Any unapplied configuration changes will now get applied. This potentially will cause a reboot of the associated server.
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile # '''show assoc'''
Service Profile Name Association Server Server Pool
-------------------- -------------- ------- -----------
UCS_B200_M3_FCH162871NA
Associating 1/1
ucs-fi-6248up-0-A /org/service-profile #
It still says Associating. So has not completely worked just yet. Let's see if we can figure out why.
ucs-fi-6248up-0-A /org/service-profile # '''top'''
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''show assoc'''
Association Service Profile
------------ ---------------
None UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /chassis/server #
So not sure what the issue is from that. One more attempt here. We know the service profile name. Let's see if there are any faults listed that mention it:
ucs-fi-6248up-0-A# '''show fault | i UCS_B200_M3_FCH162871NA'''
Critical F1000035 2026-03-30T23:08:37.136 281911 [FSM:FAILED]: Configuring Service Profile UCS_B200_M3_FCH162871NA(FSM:sam:dme:LsServerConfigure)
Major F0327 2026-03-23T20:49:35.442 124543 Service profile UCS_B200_M3_FCH162871NA configuration failed due to insufficient-resources,mac-address-assignment
ucs-fi-6248up-0-A# '''show fault 124543 detail'''
Fault Instance

Severity: Major
Code: F0327
Last Transition Time: 2026-03-23T20:49:35.442
ID: 124543
Status: Pinned
Description: Service profile UCS_B200_M3_FCH162871NA configuration failed due to insufficient-resources,mac-address-assignment
Affected Object: org-root/ls-UCS_B200_M3_FCH162871NA
Name: Ls Server Config Failure
Cause: Configuration Failure
Type: Server
Acknowledged: No
Occurrences: 1
Creation Time: 2026-03-23T20:49:35.442
Original Severity: Major
Previous Severity: Major
Highest Severity: Major
ucs-fi-6248up-0-A#
That seems like it is likely the culprit. We can't even get a server firmware update without giving it synthetic MAC addresses? OMFG!

Let us see what we can see... I see something about a LAN connectivity policy inside our organization. Can we just make a simple one?
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''show lan-connectivity-policy'''
ucs-fi-6248up-0-A /org # '''create lan-connectivity-policy test_lancon_pol'''
ucs-fi-6248up-0-A /org/lan-connectivity-policy* # '''create vnic test_vnic eth-if vlan_1000 fabric a-b'''
ucs-fi-6248up-0-A /org/lan-connectivity-policy/vnic* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/lan-connectivity-policy # '''show expand'''

vNIC LAN connectivity policy:
Name: test_lancon_pol
Description:

vNIC:
Name: test_vnic
Fabric ID: A B
Dynamic MAC Addr: Derived
Virtualization Preference: NONE

Ethernet Interface:
Name: vlan_1000
Dynamic MAC Addr: Derived
Type: Ether
Owner: Logical
Default Network: No
Config Qual: N/A
ucs-fi-6248up-0-A /org/lan-connectivity-policy # '''up'''
ucs-fi-6248up-0-A /org # '''scope service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile # '''set lan-connectivity-policy-name test_lancon_pol'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer verify-only'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile # '''reapply-config'''
Warning: Any unapplied configuration changes will now get applied. This potentially will cause a reboot of the associated server.
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile # '''top'''
ucs-fi-6248up-0-A# '''show fault | i UCS_B200_M3_FCH162871NA'''
Minor F0169 2026-04-03T05:03:54.043 518234 Eth vNIC test_vnic, service profile UCS_B200_M3_FCH162871NA failed to apply configuration
Critical F1000035 2026-03-30T23:08:37.136 281911 [FSM:FAILED]: Configuring Service Profile UCS_B200_M3_FCH162871NA(FSM:sam:dme:LsServerConfigure)
Major F0327 2026-03-23T20:49:35.442 124543 Service profile UCS_B200_M3_FCH162871NA configuration failed due to insufficient-resources,connection-placement,insufficient-vnic-capacity
ucs-fi-6248up-0-A# '''show fault 518234'''
Severity Code Last Transition Time ID Description
--------- -------- ------------------------ -------- -----------
Minor F0169 2026-04-03T05:03:54.043 518234 Eth vNIC test_vnic, service profile UCS_B200_M3_FCH162871NA failed to apply configuration
ucs-fi-6248up-0-A# '''show fault 518234 detail'''
Fault Instance

Severity: Minor
Code: F0169
Last Transition Time: 2026-04-03T05:03:54.043
ID: 518234
Status: None
Description: Eth vNIC test_vnic, service profile UCS_B200_M3_FCH162871NA failed to apply configuration
Affected Object: org-root/ls-UCS_B200_M3_FCH162871NA/ether-test_vnic
Name: Vnic Ether Config Failed
Cause: Configuration Failed
Type: Configuration
Acknowledged: No
Occurrences: 1
Creation Time: 2026-04-03T05:03:54.043
Original Severity: Minor
Previous Severity: Minor
Highest Severity: Minor
ucs-fi-6248up-0-A#
Still not there. Inching closer. Hopefully.

UCS Manager

2026-04-02T23:37:27Z

Adj: /* Applying firmware update to blades */

UCS Manager

2026-04-02T21:48:00Z

Adj: /* Applying firmware update to blades */

UCS Manager

2026-04-02T19:37:49Z

Adj: /* Applying firmware update to blades */

UCS Manager

2026-04-02T18:00:39Z

Adj: /* Applying firmware update to blades */

There are a few pieces of Cisco UCS equipment in the data center:
* One UCS 5108 blade enclosure, accommodating up to 8 B200 server blades of various generations
* 2 UCS-FI-6248UP fabric interconnects, which sit between UCS servers or blade chassis and the rest of the network
* 1 UCS C220 M5 server

Cisco UCS Manager gets installed on the Fabric Interconnects and does Cisco proprietary magic things. Like powering on a single blade. This is way more complicated that it needs to be. So this document is being written to capture knowledge of installation, upgrades, and server management tasks.

== How to get ==
Much (all?) of Cisco's UCS software is available directly without a support contract. But a customer account is required to download. Registration is free as in beer. I have had one since July 2021 and have not been troubled with Cisco spam or any sort of "are you a real person at a real company?" validation.

As this is written, [https://software.cisco.com/download/home/283612660/type/283655658/release/6.0(1e) UCS Infrastructure and UCS Manager Software] is the place to get it. As I have some older, end-of-life, end-of-support fabric interconnects (62xx series) which are not supported on the current 6.0 release train, I have instead selected the 4.2(3p) version for download. This comes as a 1Gbyte-ish <code>ucs-k9-bundle-infra.4.2.3p.A.bin</code> file.

== Untested Fabric Interconnect, what do we do? ==
5ish years ago (summer 2021), I acquired three of these 6248UP FIs. And I think I got one of them configured for use on the management network and updated to then-current software. I'm not sure which of the three that was. And I am sure I did no documentation at the time. So here we are with new notes.

=== Console and power are at the front?!?! ===
So, this is weird. Console and power connections are at the front side of the chassis. All of the connectivity is at the rear as it should be. There is sufficient room on the back for a couple of C14 power connectors '''and''' 2 more 8P8C modular connectors for management LAN and serial console. But they are at the front. So cabling is kinda sorta extra fun. ''sigh''

Console serial cable is the usual Cisco modular pinout. Signal on pins 3 and 6. 9600 bps, 8-n-1 are the serial port settings. Nothing unusual there.

Mystery solved regarding the connections at front of the chassis. My FIs have 32 ports for traffic, but there is (well, was, back when this was a current product) a 16 port expansion module available. See https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-6200-series-fabric-interconnects/data_sheet_c78-675245.html for some diagrams.

=== Basic Fabric Interconnect configuration ===
I might have lucked out on the first one I received, but it booted normally and started the interactive system setup dialog. It is all pretty basic. Set a password for the <code>admin</code> user and an IP address, netmask, IPv4 gateway address, and DNS server address for the management Ethernet interface (again, on the front). That's pretty much it.

After this is done, make sure the management Ethernet interface is connected to a switch on the correct VLAN and subnet. Then do all the things over an SSH connection. At least for version 2.1, the SSH server only knows about RSA host keys, so a modern OpenSSH client will need something like <code>-o HostKeyAlgorithms=+ssh-rsa</code> listed as a command line option.

== Clock discipline is good ==
To set the time in the FI. Or just get it close. NTP will come. Eventually.

ucs-fi-6248up-0-A# '''scope system'''
ucs-fi-6248up-0-A /system # '''scope services'''
ucs-fi-6248up-0-A /system/services # '''set clock Mar 22 2026 02 53 35'''
Sun Mar 22 02:53:35 UTC 2026
ucs-fi-6248up-0-A /system/services #

And now for the NTP:
ucs-fi-6248up-0-A# '''scope system'''
ucs-fi-6248up-0-A /system # '''scope services'''
ucs-fi-6248up-0-A /system/services # '''create ntp-server 172.16.10.2'''
ucs-fi-6248up-0-A /system/services* # '''commit-buffer'''
ucs-fi-6248up-0-A /system/services # '''create ntp-server 172.16.10.3'''
ucs-fi-6248up-0-A /system/services* # '''commit-buffer'''
ucs-fi-6248up-0-A /system/services # '''show ntp'''
NTP Servers:
Name: 172.16.10.2
Name: 172.16.10.3
ucs-fi-6248up-0-A /system/services #

== Software upgrade time! ==
Again, on the first one I have tried, let's log in as <code>admin</code> and see what's running:

Cisco UCS 6200 Series Fabric Interconnect
ucs-fi-6248up-0-A login: '''admin'''
Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

ucs-fi-6248up-0-A# '''show version'''
System version: 2.2(1c)
ucs-fi-6248up-0-A#
And, more verbosely (for reasons that are beyond me, "brief" mode shows more output?):
ucs-fi-6248up-0-A# '''show version brief'''
System version: 2.2(1c)
Package-Vers: 2.2(1c)A
ucs-fi-6248up-0-A#

So that's really very super dooper old. I downloaded a copy of the 4.1(3i) code in August 2022. Off do so some reading to see if I can skip all the in-between versions.

Did not find anything authoritative in the release notes (but I only looked very briefly.) So, we can do it stepwise. Hopefully. Process looks something like this:
ucs-fi-6248up-0-A# '''scope firmware '''
ucs-fi-6248up-0-A /firmware # '''download image tftp://172.17.0.18/ucs-k9-bundle-infra.2.2.8m.A.bin'''
ucs-fi-6248up-0-A /firmware # '''show download-task'''

Download task:
File Name Protocol Server Userid State
--------- -------- --------------- --------------- -----
ucs-k9-bundle-infra.2.2.8m.A.bin
Tftp 172.17.0.18 Downloaded
ucs-fi-6248up-0-A /firmware #

Yes, you read that right. I delivered the software image to the FI over TFTP. SSH (scp and sftp) are supported protocols, but could not connect to an OpenSSH server on Debian 13. This might be better with newer FI firmware, but for now, I already have a TFTP server handy for netbooting other machines in the environment.

And from here, the firmware bundle can actually be installed on the Fabric Interconnect like so:
ucs-fi-6248up-0-A# '''scope firmware'''
ucs-fi-6248up-0-A /firmware # '''scope auto-install'''
ucs-fi-6248up-0-A /firmware/auto-install # '''install infra infra-vers 2.2(8m)A'''
This operation upgrades firmware on UCS Infrastructure Components
(UCS manager, Fabric Interconnects and IOMs). Do you want to proceed?
(yes/no):'''yes'''

Triggering Install-Infra with:
Infrastructure Pack Version: 2.2(8m)A
ucs-fi-6248up-0-A /firmware/auto-install # '''acknowledge primary fabric-interconnect reboot'''
ucs-fi-6248up-0-A /firmware/auto-install* # '''commit-buffer'''
ucs-fi-6248up-0-A /firmware/auto-install # '''show fsm status expand'''

FSM Status:

Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: In Progress
Completion Time:
Progress (%): 87

FSM Stage:

Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployActivateUCSM Success 0
6 DeployPollActivateOfUCSM In Progress 1
7 DeployUpdateIOM Pending 0
8 DeployPollUpdateOfIOM Pending 0
9 DeployActivateIOM Pending 0
10 DeployPollActivateOfIOM Pending 0
11 DeployActivateRemoteFI Pending 0
12 DeployPollActivateOfRemoteFI Pending 0
13 DeployWaitForUserAck Pending 0
14 DeployActivateLocalFI Pending 0
15 DeployPollActivateOfLocalFI Pending 0
ucs-fi-6248up-0-A /firmware/auto-install # Connection to 172.16.10.176 closed.
Somewhere along the way, the SSH server hung up on me. But that seems like the sort of thing that '''should''' happen as the FI's OS is being upgraded. Progress can also be tracked on the serial console port like so:
ucs-fi-6248up-0-A# '''scope firmware'''
ucs-fi-6248up-0-A /firmware # '''scope auto-install'''
ucs-fi-6248up-0-A /firmware/auto-install # '''show fsm status expand | no-more'''

FSM Status:

Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: In Progress
Completion Time:
Progress (%): 98

FSM Stage:

Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployDownloadImages Skip 0
6 DeployCopyAllImagesToPeer Skip 0
7 DeployInternalBackup Success 0
8 DeployPollInternalBackup Success 2
9 DeployActivateUCSM Skip 0
10 DeployPollActivateOfUCSM Success 0
11 DeployUpdateIOM Success 0
12 DeployPollUpdateOfIOM Success 0
13 DeployActivateIOM Success 0
14 DeployPollActivateOfIOM Success 0
15 DeployActivateRemoteFI Skip 0
16 DeployPollActivateOfRemoteFI Skip 0
17 DeployWaitForUserAck Skip 0
18 DeployPollWaitForUserAck Success 0
19 DeployActivateLocalFI Success 0
20 DeployPollActivateOfLocalFI In Progress 2
ucs-fi-6248up-0-A /firmware/auto-install #
At some point, the FI will reboot itself.

Confirm new code is running like so:
ucs-fi-6248up-0-A# '''show version'''
System version: 2.2(8m)
ucs-fi-6248up-0-A# '''show version brief'''
System version: 2.2(8m)
Package-Vers: 2.2(8m)A
ucs-fi-6248up-0-A#

== STOP! Don't forget the other UCS things! ==
Um, yeah. So there is this UCS 5108 blade chassis in the data center, too. No idea what its firmware versions are. But they are sure to also need firmware updates. And it would be distressing if those components were left in a state where they are unusable because the FIs are so much newer than the blade enclosure and its servers. So I guess we need to see about warming the ZIP code 5degC by attaching the blade chassis to the mains power and see what we can see.

=== Correct cabling for single FI to UCS 5108 ===
According to https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/chassis-install-guide/ucs5108_install/ucs5108_install_chapter_010.html, a single Fabric Interconnect can only be connected to a single Fabric Extender on a UCS chassis. So for now, enable 6248UP ports 1 and 3, and cable to ports 1 and 2 on the left side (as seen from the rear) Fabric Extender.

=== SFPs go into the FI and the UCS 5108 FEXes ===
Pretty self explanatory, right. Find Cisco 10GbE SR SFP+ modules, insert into port one on each of the FEXes, insert into ports 1 and 3 on the FI. Attach nice aqua colored cable. And don't panic as the lights don't come on.

=== Telling the FI to discover the blade chassis ===
This is way more complicated that it needs to be. I'm sure it's totally amazing if you have racks full of Cisco UCS hardware. But I have just the one blade chassis and the one C series server and the 2 fabric interconnects. And the friendly FI management software is all lovely out of date (completely unrunnable in 2026!) Java stuff. So fumbling around the CLI, hoping to find my way out of the darkness here.

The process for this seems to be something like:
* Configure FI Ethernet ports as server ports
* Ensure the attached chassis has been discovered and acknowledge any faults

ucs-fi-6248up-0-A# '''scope eth-server'''
ucs-fi-6248up-0-A /eth-server # '''show fabric a detail'''

Fabric:
Id: A
Current Task:

ucs-fi-6248up-0-A /eth-server # '''scope fabric a'''
ucs-fi-6248up-0-A /eth-server/fabric # '''create interface 1 1'''
ucs-fi-6248up-0-A /eth-server/fabric/interface* # '''enable'''
ucs-fi-6248up-0-A /eth-server/fabric/interface* # '''show interface detail'''
ucs-fi-6248up-0-A /eth-server/fabric/interface* # '''commit-buffer'''
ucs-fi-6248up-0-A /eth-server/fabric/interface # '''exit'''
ucs-fi-6248up-0-A /eth-server/fabric # '''create interface 1 3'''
ucs-fi-6248up-0-A /eth-server/fabric/interface* # '''enable'''
ucs-fi-6248up-0-A /eth-server/fabric/interface* # '''commit-buffer'''
ucs-fi-6248up-0-A /eth-server/fabric/interface # '''exit'''
ucs-fi-6248up-0-A /eth-server/fabric # '''exit'''
ucs-fi-6248up-0-A /eth-server # '''show interface'''

Interface:

Fabric Slot Port Admin State Oper State State Reason Chassis Lic State Grace Prd
------- ----- ----- ------------ ------------ ------------------------------- -------- -------------------- ---------
A 1 1 Enabled Up 1 License Ok 0
A 1 3 Enabled Up 1 License Ok 0
ucs-fi-6248up-0-A /eth-server # '''exit'''
ucs-fi-6248up-0-A# '''show chassis inventory'''
Chassis PID Vendor Serial (SN) HW Revision
---------- --------------- ----------------- ----------- -----------
1 N20-C6508 Cisco Systems Inc FOX1808GLJG 0
ucs-fi-6248up-0-A#
Success! Serial number reported in the UCS Manager matches the one I found on the label.

== So how do I power on a server? ==
Still fumbling through here. But: servers (blade servers for sure, not sure about rack servers) belong to UCS "Organizations." These organizations are managed by the UCS Manager, too. And since an organization owns a server (blade or otherwise) the organization can power servers on and off.

=== What orgs exist? ===
This is what we get from a factory default UCS Manager:
ucs-fi-6248up-0-A# '''show org'''

Organizations:
Name
----
/ (root)
ucs-fi-6248up-0-A# '''show org detail'''

Organizations:
Name: / (root)
ucs-fi-6248up-0-A#

=== Enter an org, see what we have for basic policies ===
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''show bladeserver-disc-policy'''
ucs-fi-6248up-0-A /org # '''show chassis-conn-policy'''

Chassis/FEX Connectivity Policy:
Chassis/FEX Id Chassis/FEX Name Fabric Id Link Aggregation Preference
-------------- ---------------- --------- ---------------------------
1 sys/chassis-1 A Global
ucs-fi-6248up-0-A /org # '''show chassis-disc-policy'''

Chassis/FEX Discovery Policy:
Description Qualifier Action Rebalance Link Aggregation Pref Multicast Hw Hash
----------- ---------- ----------------- ----------------- --------------------- -----------------
none 1 Link User Acknowledged None Disabled
ucs-fi-6248up-0-A /org # '''show chassis-disc-policy detail'''

Chassis/FEX Discovery Policy:
Description:
Qualifier: none
Action: 1 Link
Rebalance: User Acknowledged
Link Aggregation Pref: None
Multicast Hw Hash: Disabled
ucs-fi-6248up-0-A /org # '''show rackserver-disc-policy'''

Rack Server Discovery Policy:
Action Scrub Policy
----------------- ------------
Immediate
ucs-fi-6248up-0-A /org # '''show rackserver-disc-policy detail'''

Rack Server Discovery Policy:
Action: Immediate
Scrub Policy:
Description:
Current Task:
ucs-fi-6248up-0-A /org # show '''rackserver-mgmt-policy'''

Rack Server Management Policy:
Action
------
Auto Acknowledged
ucs-fi-6248up-0-A /org # '''show rackserver-mgmt-policy detail'''

Rack Server Management Policy:
Action: Auto Acknowledged
ucs-fi-6248up-0-A /org #

=== Create a profile and add a (blade) server to it ===
ucs-fi-6248up-0-A# '''show chassis inventory 1'''
Chassis PID Vendor Serial (SN) HW Revision
---------- --------------- ----------------- ----------- -----------
1 N20-C6508 Cisco Systems Inc FOX1808GLJG 0
ucs-fi-6248up-0-A# '''show chassis inventory 1 expand '''
Chassis 1:
Servers:
Server 1/1:
Equipped Product Name: Cisco UCS B200 M3
Equipped PID: UCSB-B200-M3
Equipped VID: V03
Equipped Serial (SN): FCH162871NA
Slot Status: Equipped
Acknowledged Product Name: Cisco UCS B200 M3
Acknowledged PID: UCSB-B200-M3
Acknowledged VID: V03
Acknowledged Serial (SN): FCH162871NA
Acknowledged Memory (MB): 196608
Acknowledged Effective Memory (MB): 196608
Acknowledged Cores: 0
Acknowledged Adapters: 2

Server 1/2:
Equipped Product Name: Cisco UCS B200 M3
Equipped PID: UCSB-B200-M3
Equipped VID: V06
Equipped Serial (SN): FCH1811JELC
Slot Status: Equipped
Acknowledged Product Name: Cisco UCS B200 M3
[...]
ucs-fi-6248up-0-A# '''top'''
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''create service-profile'''
ucs-fi-6248up-0-A /org # '''create service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile* # '''associate server 1/1'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile #
=== I can power on now? ===
Well, no. Maybe it should have worked. But for this blade, there was a Power On Self Test (POST) failure. Let's dig in. To power on the server included in the service profile, do this:
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''scope service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile # '''power up'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile # '''top'''
ucs-fi-6248up-0-A #
And let's have a look at its status:
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''show'''

Server:
Slot Overall Status Service Profile Availability
------- --------------------- -------------------- ------------
1 Compute Failed Unavailable
ucs-fi-6248up-0-A /chassis/server # '''show detail'''

Server:
Slot: 1
Name:
User Label:
Overall Status: Compute Failed
Oper Qualifier: Compute Post Failure
Service Profile:
Association: None
Availability: Unavailable
Discovery: Failed
Conn Path: A
Conn Status: A
Managing Instance: A
Admin Power: Policy
Oper Power: Off
Admin State: In Service
Product Name: Cisco UCS B200 M3
PID: UCSB-B200-M3
VID: V03
Vendor: Cisco Systems Inc
Serial (SN): FCH162871NA
HW Revision: 0
Mfg Date: 2012-07-26T00:00:00.000
Part Number: 73-13217-08
Memory (MB): 196608
Effective Memory (MB): 196608
Operating Memory Speed (MHz): 1333
Operating Memory Voltage: Regular Voltage
Cores: 0
Num Of Cores Enabled: 0
Adapters: 2
Eth Host Interfaces: 0
FC Host Interfaces: 0
Burned-In UUID: 00000000-0000-0000-0000-000000000000
Dynamic UUID: 00000000-0000-0000-0000-000000000000
Current Task 1: Checking hardware configuration server 1/1(FSM-STAGE:sam:dme:ComputeBladeDiscover:Sanitize)
Current Task 2:
Current Task 3:
ucs-fi-6248up-0-A /chassis/server # '''show post'''

POST:
Global ID Code Severity Affected Object Description
--------- --------- --------- -------------------------------- -----------
6215 POST-6215 Critical sys/chassis-1/blade-1 Board Programmable version not valid for Processor Type
ucs-fi-6248up-0-A /chassis/server # '''show post detail'''

POST:
Global ID: 6215
Code: POST-6215
Local ID: 1536
Severity: Critical
Affected Object: sys/chassis-1/blade-1
Description: Board Programmable version not valid for Processor Type
Type: server: Cisco Systems Inc UCSB-B200-M3
Recoverable: Non Recoverable
Recovery Action: Refer the CPU upgrade guide.
Timestamp: 2026-03-22T16:17:06.001
ucs-fi-6248up-0-A /chassis/server #
Making an inferences from "Board Programmable version not valid for Processor Type" I am thinking that it does not like the Xeon E5-2650v2 I installed. Now to find a not v2 E5-26xx CPU and see if that works any better. I found a not obviously damaged E5-2609 in storage (er, piles in the garage) and installed that in the blade. Let's see what we have now:
ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # '''scope service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile # '''power up'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile # '''top'''
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''show inventory'''
Server 1/1:
Name:
User Label:
Equipped PID: UCSB-B200-M3
Equipped VID: V03
Equipped Serial (SN): FCH162871NA
Slot Status: Equipped
Acknowledged Product Name: Cisco UCS B200 M3
Acknowledged PID: UCSB-B200-M3
Acknowledged VID: V03
Acknowledged Serial (SN): FCH162871NA
Acknowledged Memory (MB): 16384
Acknowledged Effective Memory (MB): 16384
Acknowledged Cores: 4
Acknowledged Adapters: 2
ucs-fi-6248up-0-A /chassis/server # '''top'''
ucs-fi-6248up-0-A#
Great Success! The Xeon E5-2609 is a 4 core CPU. And there are 4x 4GiByte memory DIMMs installed. So inventory looks good. Huzzah! (And what an utter pain.)

== Server and Infrastructure issues ==
So, we just saw the first one of these above. The B200 M3 blades do not like Xeon E5-26xx v2 processors. At least with the firmware installed at present. Let's figure that one out first. https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/blade-servers/B200M3.html#reference_CD3077C1064743F99C0F9A935778BC2F has a little table listing minimum versions for various components to get the Xeon E5-2600 v2 CPUs to run. These are the required versions for E5-26xx V2 CPUs:
{|
!Software or Firmware
!Minimum Version
|-
|Server CIMC
|2.1(3)
|-
|Server BIOS
|2.1(3)
|-
|Cisco UCS Manager
|2.1(3)
|-
|Board controller firmware
|8.0
|}
Let's see if we can see what we are running in one of the blades I have and the UCS Manager itself:
ucs-fi-6248up-0-A# '''scope chassis 1'''
ucs-fi-6248up-0-A /chassis # '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''show firmware cimc'''
Server Running-Vers Package-Vers Update-Status Activate-Status
------- --------------- --------------- --------------- ---------------
1/1 3.1(23c) Ready Ready
ucs-fi-6248up-0-A /chassis/server # '''show firmware bios'''
Server 1/1:
BIOS:
Running-Vers: B200M3.2.2.6d.0.062220160055
Package-Vers:
Update-Status: Ready
Activate-Status: Ready

ucs-fi-6248up-0-A /chassis/server # '''show firmware boardcontroller'''

Management Controller:

Server Running-Vers Package-Vers Activate-Status
------- --------------- --------------- ---------------
1/1 15.0 Ready
ucs-fi-6248up-0-A /chassis/server # '''top'''
ucs-fi-6248up-0-A# '''show version brief'''
System version: 2.2(8m)
Package-Vers: 2.2(8m)A
ucs-fi-6248up-0-A#
So that is odd. The requirements listed in Cisco's documentation for supporting Xeon E5-26xx v2 CPUs '''are''' met. But the blade complains about "Board Programmable version not valid for Processor Type". Maybe the blade server's CIMC could tell us more?
== CIMC networking ==
This is going to get us started with attaching the Fabric Interconnect(s) to the rest of the network. Hopefully things go mostly OK. The network at large is all Brocade and Mellanox gear. But here's the strategy statement:
* Create a Link Aggregation Group interface on the appropriate top-of-rack Brocade ICX switches. Pipe all VLANs we care about (management and generic server, 10 and 1000) to that LAG.
* Configure the Fabric Interconnect with some uplink ports to connect to the Brocade LAG interfaces.
* Configure some VLANs on the FI uplink ports ???
* Plumb the management VLAN to the blade servers' CIMC network.
=== Brocade config ===
Fairly simple, really. Add this to the switch's config (from elevated privileges prompt (enable mode) do <code>configure terminal</code> and save (<code>write memory</code>) when done.
lag rack-2-to-ucs-fi-6248up-0 dynamic id 3
ports ethernet 1/3/2 ethernet 2/3/2
primary-port 1/3/2
deploy
And to validate:
SSH@brocade-icx6610-48p-rack-2#'''show lag rack-2-to-ucs-fi-6248up-0'''
Total number of LAGs: 3
Total number of deployed LAGs: 3
Total number of trunks created:3 (117 available)
LACP System Priority / ID: 1 / 748e.f8dd.6228
LACP Long timeout: 120, default: 120
LACP Short timeout: 3, default: 3

=== LAG "rack-2-to-ucs-fi-6248up-0" ID 3 (dynamic Deployed) ===
LAG Configuration:
Ports: e 1/3/2 e 2/3/2
Port Count: 2
Primary Port: 1/3/2
Trunk Type: hash-based
LACP Key: 20003
Deployment: HW Trunk ID 3
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/3/2 Down None None None 3 Yes N/A 0 748e.f8dd.6264
2/3/2 Down None None None 3 Yes N/A 0 748e.f8dd.6264

Port [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/3/2 1 1 20003 Yes S Agg Syn No No Def No Dwn
2/3/2 1 1 20003 Yes S Agg Syn No No Def No Dwn

Partner Info and PDU Statistics
Port Partner Partner LACP LACP
System ID Key Rx Count Tx Count
1/3/2 1-0000.0000.0000 129 0 0
2/3/2 1-0000.0000.0000 385 0 0

SSH@brocade-icx6610-48p-rack-2#
So that was not terrible.
=== UCS Fabric Interconnect uplink ports ===

Need to make some. Kinda like this:
ucs-fi-6248up-0-A# '''scope eth-uplink'''
ucs-fi-6248up-0-A /eth-uplink # '''show fabric a'''

Fabric:
Id
--
A
ucs-fi-6248up-0-A /eth-uplink # '''show fabric a detail'''

Fabric:
Id: A
Current Task 1:
Current Task 2:
ucs-fi-6248up-0-A /eth-uplink # '''show fabric a expand'''

Fabric:
Id: A
ucs-fi-6248up-0-A /eth-uplink # '''scope fabric a'''
ucs-fi-6248up-0-A /eth-uplink/fabric # '''create interface 1 17'''
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # '''enable'''
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # '''commit-buffer'''
ucs-fi-6248up-0-A /eth-uplink/fabric/interface # '''up'''
ucs-fi-6248up-0-A /eth-uplink/fabric # '''create interface 1 19'''
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # '''enable'''
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # '''commit-buffer '''
ucs-fi-6248up-0-A /eth-uplink/fabric/interface # '''exit'''
ucs-fi-6248up-0-A /eth-uplink/fabric # '''show interface '''

Interface:

Slot Id Port Id Admin State Oper State Lic State Grace Period State Reason Ethernet Link Profile name Oper Ethernet Link Profile name
---------- ---------- ----------- ---------------- -------------------- --------------- ------------ -------------------------- -------------------------------
1 17 Enabled Up License Ok 0 default fabric/lan/eth-link-prof-default
1 19 Enabled Up License Ok 0 default fabric/lan/eth-link-prof-default
ucs-fi-6248up-0-A /eth-uplink/fabric #

== Another UCS Manager (and Fabric Interconnect) update ==
I have acquired a pair of B200 M4 blades. And the FI (and UCS Manager) need upgrading to support them. Hopefully nothing breaks in the process. Hopefully...

Now that we are on 2.2(8m), the SSH client is able to fetch the .bin file using scp. Or maybe I just got the syntax correct this time. So here we are:

ucs-fi-6248up-0-A# '''show version'''
System version: 2.2(8m)
ucs-fi-6248up-0-A# '''top'''
ucs-fi-6248up-0-A# '''scope firmware'''
ucs-fi-6248up-0-A /firmware # '''download image scp://adj@172.17.0.18/home/adj/Downloads/ucs-k9-bundle-infra.2.5.2a.A.bin'''
Password:
ucs-fi-6248up-0-A /firmware # '''show download-task'''

Download task:
File Name Protocol Server Userid State
--------- -------- --------------- --------------- -----
ucs-k9-bundle-infra.2.2.8m.A.bin
Tftp 172.17.0.18 Downloaded
ucs-k9-bundle-infra.2.5.2a.A.bin
Scp 172.17.0.18 adj Downloading
ucs-fi-6248up-0-A /firmware # '''show download-task'''

Download task:
File Name Protocol Server Userid State
--------- -------- --------------- --------------- -----
ucs-k9-bundle-infra.2.2.8m.A.bin
Tftp 172.17.0.18 Downloaded
ucs-k9-bundle-infra.2.5.2a.A.bin
Scp 172.17.0.18 adj Downloading
ucs-fi-6248up-0-A /firmware # '''show download-task'''

Download task:
File Name Protocol Server Userid State
--------- -------- --------------- --------------- -----
ucs-k9-bundle-infra.2.2.8m.A.bin
Tftp 172.17.0.18 Downloaded
ucs-k9-bundle-infra.2.5.2a.A.bin
Scp 172.17.0.18 adj Downloaded
ucs-fi-6248up-0-A /firmware # '''scope auto-install
ucs-fi-6248up-0-A /firmware/auto-install # '''install infra infra-vers 2.5(2a)A'''
This operation upgrades firmware on UCS Infrastructure Components
(UCS manager, Fabric Interconnects and IOMs).
Here is the checklist of things that are recommended before starting Auto-Install
(1) Review current critical/major faults
(2) Initiate a configuration backup
(3) Check if Management Interface Monitoring Policy is enabled
(4) Check if there is a pending Fabric Interconnect Reboot activitiy
(5) Ensure NTP is configured
(6) Check if any hardware (fabric interconnects, io-modules, servers or adapters) is unsupported in the target release
Do you want to proceed? (yes/no):'''yes'''

Triggering Install-Infra with:
Infrastructure Pack Version: 2.5(2a)A

ucs-fi-6248up-0-A /firmware/auto-install # '''acknowledge primary fabric-interconnect reboot'''
ucs-fi-6248up-0-A /firmware/auto-install* # '''commit-buffer'''
ucs-fi-6248up-0-A /firmware/auto-install # '''show fsm status expand'''

FSM Status:

Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: Success
Completion Time: 2009-01-01T01:59:13.146
Progress (%): 100

FSM Stage:

Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployDownloadImages Skip 0
6 DeployCopyAllImagesToPeer Skip 0
7 DeployInternalBackup Success 0
8 DeployPollInternalBackup Success 2
9 DeployActivateUCSM Skip 0
10 DeployPollActivateOfUCSM Success 0
11 DeployUpdateIOM Success 0
12 DeployPollUpdateOfIOM Success 0
13 DeployActivateIOM Success 0
14 DeployPollActivateOfIOM Success 0
15 DeployActivateRemoteFI Skip 0
16 DeployPollActivateOfRemoteFI Skip 0
17 DeployWaitForUserAck Skip 0
18 DeployPollWaitForUserAck Success 0
19 DeployActivateLocalFI Success 0
20 DeployPollActivateOfLocalFI Success 2
ucs-fi-6248up-0-A /firmware/auto-install # '''show fsm status expand | no-more'''

FSM Status:

Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: In Progress
Completion Time:
Progress (%): 95

FSM Stage:

Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployDownloadImages Skip 0
6 DeployCopyAllImagesToPeer Skip 0
7 DeployInternalBackup Success 0
8 DeployPollInternalBackup In Progress 1
9 DeployActivateUCSM Pending 0
10 DeployPollActivateOfUCSM Pending 0
11 DeployUpdateIOM Pending 0
12 DeployPollUpdateOfIOM Pending 0
13 DeployActivateIOM Pending 0
14 DeployPollActivateOfIOM Pending 0
15 DeployActivateRemoteFI Pending 0
16 DeployPollActivateOfRemoteFI Pending 0
17 DeployWaitForUserAck Pending 0
18 DeployPollWaitForUserAck Pending 0
19 DeployActivateLocalFI Pending 0
20 DeployPollActivateOfLocalFI Pending 0
ucs-fi-6248up-0-A /firmware/auto-install # '''show fsm status expand | no-more'''

FSM Status:

Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: In Progress
Completion Time:
Progress (%): 96

FSM Stage:

Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployDownloadImages Skip 0
6 DeployCopyAllImagesToPeer Skip 0
7 DeployInternalBackup Success 0
8 DeployPollInternalBackup Success 2
9 DeployActivateUCSM Success 0
10 DeployPollActivateOfUCSM In Progress 1
11 DeployUpdateIOM Pending 0
12 DeployPollUpdateOfIOM Pending 0
13 DeployActivateIOM Pending 0
14 DeployPollActivateOfIOM Pending 0
15 DeployActivateRemoteFI Pending 0
16 DeployPollActivateOfRemoteFI Pending 0
17 DeployWaitForUserAck Pending 0
18 DeployPollWaitForUserAck Pending 0
19 DeployActivateLocalFI Pending 0
20 DeployPollActivateOfLocalFI Pending 0
ucs-fi-6248up-0-A /firmware/auto-install # Connection to 172.16.10.176 closed.
$ '''ssh -o HostKeyAlgorithms=+ssh-rsa admin@172.16.10.176'''
Cisco UCS 6200 Series Fabric Interconnect
(admin@172.16.10.176) Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

ucs-fi-6248up-0-A# '''show version'''
System version: 2.5(2a)
ucs-fi-6248up-0-A# '''show version brief'''
System version: 2.5(2a)
Package-Vers: 2.5(2a)A
ucs-fi-6248up-0-A#
And after some more time and one more disconnect, we see that another OpenSSH client compatibility option is required to connect:
$ '''ssh -o HostKeyAlgorithms=+ssh-rsa admin@172.16.10.176'''
Unable to negotiate with 172.16.10.176 port 22: no matching cipher found. Their offer: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se
$ '''ssh -o HostKeyAlgorithms=+ssh-rsa -o Ciphers=+aes256-cbc admin@172.16.10.176'''
Cisco UCS 6200 Series Fabric Interconnect
(admin@172.16.10.176) Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

ucs-fi-6248up-0-A# '''show version'''
System version: 2.5(2a)
ucs-fi-6248up-0-A# '''show version brief'''
System version: 2.5(2a)
Package-Vers: 2.5(2a)A
ucs-fi-6248up-0-A#

== But I'm still trying to get the blades' CIMCs connected! ==
The struggle is real! But I did find https://www.cisco.com/c/en/us/support/docs/servers-unified-computing/integrated-management-controller/118367-configure-cimc-00.pdf which is a GUI-specific outline of the process involved in getting UCS servers' (blade and maybe also rackmount) CIMCs attached to the greater network environment.
=== In-band? Out-of-band? Huh? ===
First decision to be made here. In UCS versions before 2.2, the servers' CIMCs always had Ethernet connectivity through the Fabric Interconnects' management port. This connection amounts to a single point of failure for CIMC connectivity.

In UCS version 2.2 and newer, there are two options for CIMC connectivity: In-band using the FIs' high bandwidth network uplink ports, and out-of band using an FI's 1000baseT management port. Naturally, being me, I will opt for the more complicated alternative. And probably complain the whole way through.

=== In-band CIMC connectivity setup ===
Log in to the FI. Go to the organization (probably <code>/</code>. And create an IP pool. We are skipping the IPv6 stuff for now and the management VLAN is IPv4-only.
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''show ip-pool'''

IP Pool:
Name Size Assigned Management mode
-------------------- ---------- ---------- ---------------
ext-mgmt 0 0 Internal
iscsi-initiator-pool 0 0 Internal
ucs-fi-6248up-0-A /org # '''create ip-pool CIMC_inband_pool'''
ucs-fi-6248up-0-A /org/ip-pool* # '''create block 172.16.10.168 172.16.10.175 172.16.10.1 255.255.255.0'''
ucs-fi-6248up-0-A /org/ip-pool/block* # '''set primary-dns 172.16.10.1 secondary-dns 172.16.10.2'''
ucs-fi-6248up-0-A /org/ip-pool/block* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/ip-pool/block # '''show detail'''

Block of IP Addresses:
From: 172.16.10.168
To: 172.16.10.175
Default Gateway: 172.16.10.1
Subnet Mask: 255.255.255.0
Primary DNS: 172.16.10.1
Secondary DNS: 172.16.10.2
ucs-fi-6248up-0-A /org/ip-pool/block # '''show expand'''

Block of IP Addresses:
From To Default Gateway Subnet Mask
--------------- --------------- --------------- -----------
172.16.10.168 172.16.10.175 172.16.10.1 255.255.255.0
ucs-fi-6248up-0-A /org/ip-pool/block # '''up'''
ucs-fi-6248up-0-A /org/ip-pool # '''show'''

IP Pool:
Name Size Assigned Management mode
-------------------- ---------- ---------- ---------------
CIMC_inband_pool 8 0 Internal
ucs-fi-6248up-0-A /org/ip-pool # '''show detail'''

IP Pool:
Name: CIMC_inband_pool
Size: 8
Assigned: 0
IPv4 Size: 8
IPv4 Assigned: 0
IPv6 Size: 0
IPv6 Assigned: 0
Descr:
Assignment Order: Default
Management mode: Internal
Guid: 00000000-0000-0000-0000-000000000000
Net bios enabled or disabled: Not Active
DHCP enaled or disabled: Not Supported
ucs-fi-6248up-0-A /org/ip-pool # '''show expand'''

IP Pool:
Name: CIMC_inband_pool
Size: 8
Assigned: 0
Management mode: Internal

Block of IP Addresses:
From To Default Gateway Subnet Mask
--------------- --------------- --------------- -----------
172.16.10.168 172.16.10.175 172.16.10.1 255.255.255.0

Pooled:
Id Subnet Assigned Assigned To
--------------- --------------- -------- -----------
172.16.10.168 255.255.255.0 No
172.16.10.169 255.255.255.0 No
172.16.10.170 255.255.255.0 No
172.16.10.171 255.255.255.0 No
172.16.10.172 255.255.255.0 No
172.16.10.173 255.255.255.0 No
172.16.10.174 255.255.255.0 No
172.16.10.175 255.255.255.0 No
ucs-fi-6248up-0-A /org/ip-pool #
So we now have a pool of IPv4 addresses to hand out to CIMCs in our UCS domain. And we can also LOL at Cisco's <code>DHCP '''enaled''' or disabled</code> spelling error. But not too loud. It may well be fixed in newer versions.

=== Create VLAN and VLAN group to connect the CIMC IP pool to ===
VLAN on the Ethernet uplinks:
ucs-fi-6248up-0-A /org/ip-pool # '''top'''
ucs-fi-6248up-0-A# '''scope eth-uplink'''
ucs-fi-6248up-0-A /eth-uplink # '''create vlan Management 10'''
ucs-fi-6248up-0-A /eth-uplink/vlan* # '''set native no'''
ucs-fi-6248up-0-A /eth-uplink/vlan* # '''set vlan-id 10'''
ucs-fi-6248up-0-A /eth-uplink/vlan* # '''commit-buffer'''
VLAN group now:
ucs-fi-6248up-0-A /eth-uplink/vlan # '''up'''
ucs-fi-6248up-0-A /eth-uplink # '''show vlan-group'''
ucs-fi-6248up-0-A /eth-uplink # '''create vlan-group Management_group'''
ucs-fi-6248up-0-A /eth-uplink/vlan-group* # '''create member-vlan Management'''
ucs-fi-6248up-0-A /eth-uplink/vlan-group/member-vlan* # '''commit-buffer'''
Add IP pool, VLAN, VLAN group to in-band profile:
ucs-fi-6248up-0-A /eth-uplink/vlan-group/member-vlan # '''top'''
ucs-fi-6248up-0-A# '''scope eth-uplink'''
ucs-fi-6248up-0-A /eth-uplink # '''scope inband-profile'''
ucs-fi-6248up-0-A /eth-uplink/inband-profile # '''set default-pool-name CIMC_inband_pool'''
ucs-fi-6248up-0-A /eth-uplink/inband-profile* # '''set default-vlan-name Management'''
ucs-fi-6248up-0-A /eth-uplink/inband-profile* # '''set net-group-name Management_group'''
ucs-fi-6248up-0-A /eth-uplink/inband-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /eth-uplink/inband-profile #
Now that the in-band profile is set, update a blade's service profile to reference it:
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''show service-profile'''

Service Profile:
Service Profile Name Type Server Assignment Association
-------------------- ----------------- ------- ---------- -----------
UCS_B200_M3_FCH162871NA
Instance 1/1 Assigned Associating
ucs-fi-6248up-0-A /org # '''scope service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile # '''set ext-mgmt-ip-pool-name CIMC_inband_pool'''
ucs-fi-6248up-0-A /org/service-profile* # '''set ext-mgmt-ip-state pooled'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A# '''top'''
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''reset hard-reset-immediate'''
ucs-fi-6248up-0-A /chassis/server* # '''commit-buffer'''
ucs-fi-6248up-0-A /chassis/server # '''reset-kvm'''
ucs-fi-6248up-0-A /chassis/server* # '''commit-buffer'''
ucs-fi-6248up-0-A /chassis/server* # '''reset-ipmi'''
ucs-fi-6248up-0-A /chassis/server* # '''commit-buffer'''
ucs-fi-6248up-0-A /chassis/server # '''scope cimc'''
ucs-fi-6248up-0-A /chassis/server/cimc # '''show'''

CIMC:
PID Serial (SN) HW Revision
---------------- ---------------- -----------
UCSB-B200-M3 FCH162871NA 0
ucs-fi-6248up-0-A /chassis/server/cimc # '''show detail'''

CIMC:
Product Name: Cisco UCS B200 M3
PID: UCSB-B200-M3
VID: V01
Vendor: Cisco Systems Inc
Serial (SN): FCH162871NA
HW Revision: 0
GUID:
Current Task:
ucs-fi-6248up-0-A /chassis/server/cimc # '''show expand'''

CIMC:
PID: UCSB-B200-M3
Serial (SN): FCH162871NA
HW Revision: 0

Vmedia Mapping List:
Full Name: sys/chassis-1/blade-1/mgmt/actual-mount-list

Vmedia Mapping:
Vdisk Id Mapping Name Device Type Mount Protocol Mount Status
-------- ------------ ----------- -------------- ------------
1 Cdd Unknown Not Mounted
2 Hdd Unknown Not Mounted

External Management Interface:
Mode: In Band
Ip V4 State: Pooled
Ip V6 State: Pooled

External Management Virtual LAN:
Network Name: Management
Id: 10

External Management Pooled IP:
Name IP Address Default Gateway Subnet Primary DNS IP Secondary DNS IP
---------- --------------- --------------- --------------- --------------- ----------------
CIMC_inband_pool
172.16.10.169 172.16.10.1 255.255.255.0 172.16.10.1 172.16.10.2

External Management Pooled IPv6:
Name IP Address Default Gateway Prefix Primary DNS IP Secondary DNS IP
---------- ---------- --------------- ------ -------------- ----------------
CIMC_inband_pool
:: :: 64 :: ::

Management Interface:
Access Type: Unspecified
IP Address: 0.0.0.0
Netmask: 0.0.0.0
Gateway: 0.0.0.0
MAC Address: 30:F7:0D:BE:89:EA
Fabric ID: A
Peer Port:
Peer Port Id: 1
Peer Slot Id: 1
Peer Chassis Id: 1
Discovery: Absent

Management Endpoint Log Control:
Type ID Capacity
------ --- --------
SEL 0 Available
ucs-fi-6248up-0-A /chassis/server/cimc #
(Some) Success! The blade's CIMC is assigned an IPv4 address. And it is pingable on the local network. SSH is a no-go, though. Maybe not a thing at all on B series blades? And some further fiddling seems to be needed to get IPMI over LAN working. Something like this:
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''create ipmi-access-profile ipmi_profile'''
ucs-fi-6248up-0-A /org/ipmi-access-profile* # '''create ipmi-user admin'''
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user* # '''set password'''
Enter a password:
Confirm the password:
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user # '''set privilege admin'''
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user # '''up'''
ucs-fi-6248up-0-A /org/ipmi-access-profile # '''set ipmi-over-lan enable'''
ucs-fi-6248up-0-A /org/ipmi-access-profile # '''commit-buffer'''
ucs-fi-6248up-0-A /org/ipmi-access-profile # '''up'''
ucs-fi-6248up-0-A /org # '''scope service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile # '''set ipmi-access-profile ipmi_profile'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile # '''top'''
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''scope cimc'''
ucs-fi-6248up-0-A /chassis/server/cimc # '''reset'''
ucs-fi-6248up-0-A /chassis/server/cimc* # '''commit-buffer'''
ucs-fi-6248up-0-A /chassis/server/cimc # '''top'''
ucs-fi-6248up-0-A# '''connect cimc 1/1'''
Trying 127.5.1.1...
Connected to 127.5.1.1.
Escape character is '^]'.

CIMC Debug Firmware Utility Shell [ support ]
[ help ]# '''help'''
__________________________________________
Debug Firmware Utility
__________________________________________
Command List
__________________________________________
alarms
cores
exit
i2cstats
images
mctools
memory
messages
mrcout
network
obfl
post
power
programmables
sensors
sel
fru
tasks
top
update
users
version
mezz1fru
mezz2fru
sldp
help [COMMAND]
__________________________________________
Notes:
"enter Key" will execute last command
"COMMAND ?" will execute help for that command
__________________________________________
[ help ]# '''users'''
0. [ "admin" ] [ "" ] [ 2 ]
1. [ Empty Entry ]
2. [ Empty Entry ]
3. [ Empty Entry ]
4. [ Empty Entry ]
5. [ Empty Entry ]
6. [ Empty Entry ]
7. [ Empty Entry ]
8. [ Empty Entry ]
9. [ Empty Entry ]
10. [ Empty Entry ]
11. [ Empty Entry ]
12. [ Empty Entry ]
13. [ Empty Entry ]
14. [ Empty Entry ]
15. [ Empty Entry ]
[ users ]# '''exit'''
Connection closed by foreign host.
ucs-fi-6248up-0-A#

So that looks like we should have IPMI over LAN. Let's poke at it a bit and see what happens:
$ '''ping -c 5 172.16.10.169'''
PING 172.16.10.169 (172.16.10.169) 56(84) bytes of data.
64 bytes from 172.16.10.169: icmp_seq=1 ttl=63 time=0.930 ms
64 bytes from 172.16.10.169: icmp_seq=2 ttl=63 time=0.655 ms
64 bytes from 172.16.10.169: icmp_seq=3 ttl=63 time=0.677 ms
64 bytes from 172.16.10.169: icmp_seq=4 ttl=63 time=0.658 ms
64 bytes from 172.16.10.169: icmp_seq=5 ttl=63 time=0.732 ms

--- 172.16.10.169 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4058ms
rtt min/avg/max/mdev = 0.655/0.730/0.930/0.103 ms
$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin chassis status'''
Password:
Error: Unable to establish IPMI v2 / RMCP+ session
$
And that is a promising start with a disappointing finish. Just for grins, let's try from a machine on the same subnet. Maybe ICMP echo replies get routed but the IPMI over LAN traffic doesn't?
router$ '''ping -c 5 172.16.10.169'''
PING 172.16.10.169 (172.16.10.169) 56(84) bytes of data.
64 bytes from 172.16.10.169: icmp_seq=1 ttl=64 time=1.85 ms
64 bytes from 172.16.10.169: icmp_seq=2 ttl=64 time=0.498 ms
64 bytes from 172.16.10.169: icmp_seq=3 ttl=64 time=0.463 ms
64 bytes from 172.16.10.169: icmp_seq=4 ttl=64 time=0.452 ms
64 bytes from 172.16.10.169: icmp_seq=5 ttl=64 time=0.460 ms

--- 172.16.10.169 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4067ms
rtt min/avg/max/mdev = 0.452/0.745/1.853/0.554 ms
router$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin mc info'''
Password:
Device ID : 32
Device Revision : 0
Firmware Revision : 3.01
IPMI Version : 2.0
Manufacturer ID : 5771
Manufacturer Name : Cisco Systems, Inc.
Product ID : 9 (0x0009)
Product Name : Unknown (0x09)
Device Available : yes
Provides Device SDRs : yes
Additional Device Support :
Sensor Device
SDR Repository Device
SEL Device
FRU Inventory Device
IPMB Event Receiver
IPMB Event Generator
Aux Firmware Rev Info :
0x00
0x00
0x2a
0x8b
router$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin lan print'''
Password:
Set in Progress : Set Complete
Auth Type Support : NONE MD2 MD5 PASSWORD
Auth Type Enable : Callback :
: User :
: Operator :
: Admin :
: OEM :
IP Address Source : Static Address
IP Address : 0.0.0.0
Subnet Mask : 0.0.0.0
MAC Address : 30:f7:0d:be:89:ea
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x00 Precedence=0x00 TOS=0x10
BMC ARP Control : ARP Responses Disabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 2.0 seconds
Default Gateway IP : 0.0.0.0
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,128
Cipher Suite Priv Max : XXXaaaXXaaaXaaa
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
Bad Password Threshold : Not Available
router$
Yay! Very strange that it can be pinged from a different subnet, but <code>ipmitool</code> cannot talk to it that way. Also very strange that it says it has no IP address assigned. I am just guessing here, but perhaps the code in the FI is doing that for us and I have not found the documentation describing how it works.

=== Serial console for blades? ===
I am fine with low bit rate serial stuff. No need for a fancy bit mapped screen to manage my computers. And it saves me having to run crazy Avocent Java applications in the process. So let's see what happens when we try to connect from a machine on the same subnet:
router$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin sol activate'''
Password:
Error activating SOL payload: Invalid data field in request
router$
Disappointing. But I think I might have seem something about serial over LAN in the service profiles somewhere. Let's see if we can get it turned on:
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''show sol-policy'''
ucs-fi-6248up-0-A /org # '''create sol-policy sol_pol'''
ucs-fi-6248up-0-A /org/sol-policy* # '''enable'''
ucs-fi-6248up-0-A /org/sol-policy* # '''set speed 115200'''
ucs-fi-6248up-0-A /org/sol-policy* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/sol-policy # '''up'''
ucs-fi-6248up-0-A /org # '''show service-profile'''

Service Profile:
Service Profile Name Type Server Assignment Association
-------------------- ----------------- ------- ---------- -----------
UCS_B200_M3_FCH162871NA
Instance 1/1 Assigned Associating
ucs-fi-6248up-0-A /org # '''scope service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile # '''set sol-policy sol_pol'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile # '''top'''
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''scope cimc'''
ucs-fi-6248up-0-A /chassis/server/cimc # '''reset'''
ucs-fi-6248up-0-A /chassis/server/cimc* # '''commit-buffer'''
ucs-fi-6248up-0-A /chassis/server/cimc #

And trying again from a machine on the same subnet:
router$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin sol activate'''
Password:
Error activating SOL payload: Invalid data field in request
router$

So still no. Grrrr. Maybe it needs to be powered on first?
router$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin chassis power status'''
Password:
Chassis Power is off
router$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin chassis power on'''
Password:
Chassis Power Control: Up/On
router$ '''sleep 30'''
router$ '''ipmitool -I lanplus -H 172.16.10.169 -U admin sol activate'''
Password:
Error activating SOL payload: Invalid data field in request
router$

More grrrr. Manuals suggest I might need a BIOS policy that enables the serial port, too?
ucs-fi-6248up-0-A# '''scope org /'''
ucs-fi-6248up-0-A /org # '''create bios-policy serial_is_good'''
ucs-fi-6248up-0-A /org/bios-policy* # '''set quiet-boot-config quiet-boot disabled'''
ucs-fi-6248up-0-A /org/bios-policy* # '''set serial-port-a-config serial-port-a enabled'''
ucs-fi-6248up-0-A /org/bios-policy* # '''set console-redir-config baud-rate 115200'''
ucs-fi-6248up-0-A /org/bios-policy* # '''set console-redir-config console-redir serial-port-a'''
ucs-fi-6248up-0-A /org/bios-policy* # '''set console-redir-config flow-control none'''
ucs-fi-6248up-0-A /org/bios-policy* # '''set console-redir-config legacy-os-redir disabled'''
ucs-fi-6248up-0-A /org/bios-policy* # '''set console-redir-config terminal-type vt100-plus'''
ucs-fi-6248up-0-A /org/bios-policy* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/bios-policy # '''up'''
ucs-fi-6248up-0-A /org # '''show service-profile'''

Service Profile:
Service Profile Name Type Server Assignment Association
-------------------- ----------------- ------- ---------- -----------
UCS_B200_M3_FCH162871NA
Instance 1/1 Assigned Associating
ucs-fi-6248up-0-A /org # '''scope service-profile UCS_B200_M3_FCH162871NA'''
ucs-fi-6248up-0-A /org/service-profile # '''set bios-policy serial_is_good'''
ucs-fi-6248up-0-A /org/service-profile* # '''commit-buffer'''
ucs-fi-6248up-0-A /org/service-profile # '''top'''
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''scope cimc'''
ucs-fi-6248up-0-A /chassis/server/cimc # '''reset'''
ucs-fi-6248up-0-A /chassis/server/cimc* # '''commit-buffer'''
ucs-fi-6248up-0-A /chassis/server/cimc # '''top'''
ucs-fi-6248up-0-A#
So there is still no working serial console on the blade at this point. https://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-b-series-blade-servers/200105-Utilizing-SoL-logging-for-Serial-Redirec.html says to SSH to the CIMC IP address. Just getting a TCP RST back ("connection refused") when trying that. And the IPMI over LAN serial just does not seem to be supported at all. Maybe the blade needs a firmware update, too?
=== Applying firmware update to blades ===
So far (I think) I have only done firmware updates to the fabric interconnect and the blade chassis FEXes. First up, let's see what is available for our B200 M3 blades:
ucs-fi-6248up-0-A# '''scope firmware'''
ucs-fi-6248up-0-A /firmware # '''show image | i b200-m3'''
ucs-b200-m3-bios.B200M3.2.0.4a.0.080920121557.bin
ucs-b200-m3-bios.B200M3.2.2.1a.0.111220131105.bin
ucs-b200-m3-brdprog.11.0.bin Board Controller 11.0
ucs-b200-m3-k9-cimc.2.0.4b.bin CIMC 2.0(4b)
ucs-b200-m3-k9-cimc.2.2.1c.bin CIMC 2.2(1c)
ucs-b200-m3-mrsasctlr.20.10.1-0100_4.30.00_NA.bin
ucs-b200-m3-mrsasctlr.20.12.1-0160_4.37.00_NA.bin
ucs-fi-6248up-0-A /firmware #
Those might be kinda old? Let's see what is installed on the B200 M3 that is in the chassis:
ucs-fi-6248up-0-A /firmware # '''top'''
ucs-fi-6248up-0-A# '''scope server 1/1'''
ucs-fi-6248up-0-A /chassis/server # '''show firmware'''
Server 1/1:
Adapter 1:
Running-Vers: 4.1(3a)
Package-Vers:
Update-Status: Ready
Activate-Status: Ready
Adapter 2:
Running-Vers: 4.1(3a)
Package-Vers:
Update-Status: Ready
Activate-Status: Ready
BIOS:
Running-Vers: B200M3.2.2.6d.0.062220160055
Package-Vers:
Update-Status: Ready
Activate-Status: Ready

RAID Controller 1:
Running-Vers: 20.13.1-0255
Package-Vers:
Activate-Status: Ready

BoardController:
Running-Vers: 15.0
ucs-fi-6248up-0-A /chassis/server #
From this, it would seem that the blade in the chassis is already running a newer firmware bundle than the one running on the fabric interconnect. So, let's find out what we are running on the FI, and see about some more updates to it:
ucs-fi-6248up-0-A /chassis/server # '''top'''
ucs-fi-6248up-0-A# '''show version brief'''
System version: 2.5(2a)
Package-Vers: 2.5(2a)A
ucs-fi-6248up-0-A#
It would seem that 3.0(2f) is the latest in the next-newer release train. So acquire that from Cisco's download site and install on the FI and apply following the process documented above. And there is a hiccup with the SSH in 2.5(2a) not being able to get a host key from the SSH servers running on Debian 12 or Debian 13. Here's an example Debian server log of that:
adj@zarathud:~$ '''sudo grep ssh.*172.16.10.176 /var/log/auth.log'''
2026-04-02T17:24:54.156462+00:00 zarathud sshd[3090225]: Unable to negotiate with 172.16.10.176 port 33905: no matching host key type found. Their offer: x509v3-sign-rsa,x509v3-sign-dss,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss [preauth]
2026-04-02T17:41:20.766493+00:00 zarathud sshd[3092657]: Unable to negotiate with 172.16.10.176 port 58959: no matching host key type found. Their offer: x509v3-sign-rsa,x509v3-sign-dss,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss [preauth]
2026-04-02T17:50:44.154486+00:00 zarathud sshd[3094045]: Unable to negotiate with 172.16.10.176 port 45452: no matching host key type found. Their offer: x509v3-sign-rsa,x509v3-sign-dss,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss [preauth]
2026-04-02T17:51:36.645000+00:00 zarathud sudo: adj : TTY=pts/22 ; PWD=/home/adj ; USER=root ; COMMAND=/usr/bin/grep ssh.*172.16.10.176 /var/log/auth.log
2026-04-02T17:53:33.561802+00:00 zarathud sudo: adj : TTY=pts/22 ; PWD=/home/adj ; USER=root ; COMMAND=/usr/bin/grep ssh.*172.16.10.176 /var/log/auth.log
adj@zarathud:~$
So, for expediency's sake, we will just do it over TFTP this time. Maybe the SSH client in the 3.0(2f) release will be better able to talk to my SSH servers?

UCS Manager

2026-04-02T17:21:09Z

Adj: /* Serial console for blades? */

UCS Manager

2026-03-31T22:04:43Z

Adj: /* Serial console for blades? */

UCS Manager

2026-03-31T18:19:56Z

Adj: /* Create VLAN and VLAN group to connect the CIMC IP pool to */

UCS Manager

2026-03-30T22:05:08Z

Adj: /* Another UCS Manager (and Fabric Interconnect) update */

UCS Manager

2026-03-27T17:12:21Z

Adj: /* Telling the FI to discover the blade chassis */

UCS Manager

2026-03-27T02:41:04Z

Adj:

UCS Manager

2026-03-27T00:35:32Z

Adj: /* Another UCS Manager (and Fabric Interconnect) update */

UCS Manager

2026-03-27T00:05:43Z

Adj: /* UCS Fabric Interconnect uplink ports */

Cisco UCS Rack servers

2026-03-26T18:26:05Z

Adj: /* Serial over LAN */

Acquired a Cisco UCS C220 M5 on 2026-03-10. Here are some notes.

== Network based management ==
CIMC, the Cisco Integrated Management Controller. On the machine I have obtained, this was not configured to use the dedicated management Ethernet interface on the back of the server. This is easily corrected while the machine starts up. Pressing F8 from a USB keyboard during the server's power on self test sequence will bring up the CIMC configuration screen. From there, the dedicated NIC option, IPv4 option, and DHCP enabled can be selected to do the expected things. After those are selected, press the <code><F1></code> key and fix the dedicated NIC port properties if the need any adjustment.

The CIMC is accessible over https and SSH. At first login (at least by HTTPS), a new password must be set. None of Dell's just warning "Heh, you have the default credentials. Not recommended!" warning stuff.

== Firmware updates ==
As acquired, server s/n WZP22040X50 is running BIOS version C220M5.3.1.2b.0.1025170315 and CIMC version 3.1(2c). Cisco's firmware update process is heavier weight than some other vendors'. More like HP(E)'s Service Pack for Proliant than Dells "get iDRAC to download update images from an FTP or web server." An ISO image must be downloaded and booted from. (This is maybe avoidable. And if is, updates will be made to document the process here.) As we are running 3.1 release code now, download the final update ISO image, 3.1(3k). This is free-as-in-beer with an account on Cisco's support site. No $$$ required. I have not read every line of the release notes to understand whether entire release trains can be skipped in the upgrade process.

One thing that is nice about getting the account created: they have a notification system a customer can sign up for when sends an email when new firmware is released. With options for weekly, daily, or monthly notications.

After the ISO is downloaded, open the KVM on the server's CIMC and attach it to the server using the appropriate item in the "Virtual Media" menu. Reboot server and press the <code><F6></code> key at the appropriate time to invoke the boot device selection menu. From there, boot from the Cisco vKVM attached ISO image. Wait patiently. After several minutes, you might catch Linux kernel boot messages. And shortly after that, something about the Cisco UCS Host Upgrade Utility Version 3.1.3k doing its work. Then wait some more. The Host Update Utility is not very good about giving progress updates. At least not this version. After 5-10 minutes an "accept the license" screen will appear. Do the obvious. And then, the easy thing is to use the "Update All" button and get all updates applied. Wait patiently some more. But at least we get some progress reports as the actual updates are applied. After all desired updates are complete, select "Exit" The CIMC will reboot if its code was updated. And the server will as well.

If the paranoia is very strong with you, Host Upgrade Utility ISO can be booted from a second time, and all updates verified. The boot and utility loading are just as excruciatingly slow as the actual update, but the verification itself goes swiftly.

First thing of note with going from CIMC 3.1(2c) to 3.1(3k) is that the CIMC's SSH server now supports more modern SSH host keys. No more need for <code>-o HostKeyAlgorithms=+ssh-rsa</code> when pointing an SSH client at the CIMC.

Also, following the release notes, jumping straight from 3.1(2c) to the most recent 4.3(2.260007) is almost certainly not a thing. I have been doing it in increments so far: 3.1(2c) -> 3.1(3k) -> 4.0(4n) -> 4.1(3n) -> 4.2(3l)

4.1(3n) includes some sort of cloud management functionality which I probably don't want. It is called "Cisco Intersight." It will get its own section.

== Cisco Intersight Cloud Management ==
Yeah, what I said just above. I am pretty sure I don't want this. Do not need this server's BMC phoning home to the Cisco mothership. Need to do some reading about turning it off.

It turns out to not to too difficult. Kinda hidden, but here's how I did it:

# Log into CIMC web UI. You will be looking at the Chassis / Summary page.
# Click on the icon in the top left corner. It is pretty much a three bar "hamburger" menu with a right-pointing arrow over its top.
# At the bottom of the just-exposed navigation pane, select the Admin / Device Connector link.
# On the right side of the Device Connector page, find the "Settings" link and follow it.
# Toggle Device Connector off, save settings, enjoy some peace about not having your server watching you quite as much.

The above is what I did on CIMC 4.1(3n). It may be different in newer releases. It may also be re-activated after further firmware updates. So make a note to check this one after more of those.

== Doing things with the CIMC ==
=== Restarting the CIMC ===
The web UI has a "CIMC Reboot" link toward the top right of the main page. At least while running CIMC 4.3 code. From an SSH session, do the following after logging in:
C220-WZP22040X50# '''scope cimc'''
C220-WZP22040X50 /cimc # '''reboot'''
This operation will reboot the Cisco IMC.
Continue?[y|N]'''y'''
and wait for it to restart.

=== Server Power Control ===
Pretty easy:
C220-WZP22040X50#
C220-WZP22040X50# '''scope chassis'''
C220-WZP22040X50 /chassis # '''power on'''
Do you want to continue?[y|N]'''y'''
C220-WZP22040X50 /chassis # '''exit'''
C220-WZP22040X50#

It also has some other options:
C220-WZP22040X50#
C220-WZP22040X50# '''scope chassis'''
C220-WZP22040X50 /chassis # '''power'''
cycle Power off then power on
hard-reset Equivalent to pressing Front Panel reset button or IPMI reset
off Turn power off (OS given short time to shutdown)
on Turn power on
shutdown Gracefully shutdown the host OS & power off

C220-WZP22040X50 /chassis # '''power'''

=== Serial over LAN ===
This is a bit different than Dell's iDRAC or HP(E)'s iLO. It gets its own dedicated SSH port. From an SSH connection to the CIMC, the configuration can be listed out like so:
C220-WZP22040X50# '''show sol detail'''
Serial Over LAN:
Enabled: no
Baud Rate(bps): 115200
Com Port: com0
SOL SSH Port: 2400
C220-WZP22040X50#

It does need to be explicitly enabled. Again, to do that from the CIMC SSH interface (there are maybe BIOS config and CMIC web interface toggles for this, too) do this:
C220-WZP22040X50# '''scope sol'''
C220-WZP22040X50 /sol # '''set enabled yes'''
C220-WZP22040X50 /sol *# '''commit'''
C220-WZP22040X50 /sol # '''exit'''
C220-WZP22040X50#

And that is done, so connect to the SSH server on port 2400. With OpenSSH, that is done by specifying <code>-p 2400</code> on the <code>ssh</code> command line. Something like:
$ '''ssh -p 2400 admin@172.16.10.160'''
admin@172.16.10.160's password:
CISCO Serial Over LAN:
Press Ctrl+x to Exit the session

CISCO Serial Over LAN exited
Connection to 172.16.10.160 closed.
$
The Ctrl+x thing is annoying. My habit with GRUB is to kick off edited menu entries with Ctrl+x. On this one, I must hit the F10 key instead. No major complaints aside from that, though. Pasting in 200+ character disk passphrases works without issue.

== Mass storage devices, RAID, JBOD, and such ==
The ECS C220 M5 has a Broadcom/LSI MegaRAID Tri-Mode SAS3516 attached to the disk backplane in the front of the chassis.

I prefer running my disk interfaces in JBOD (just a bunch of disks) or pass-through mode, exposing the individual physical storage devices to the operating system, and using OS tooling to manage redundancy. As this server arrived running VMware ESXi, an OS without those data redundancy management features, the server came using the SAS3516 RAID controller set up to present 2 virtual disks to the OS. Setting the controller to JBOD mode is possible, though. To get to the MegaRAID configuration without a running OS equipped with <code>storcli</code> and related tools, restart the server, and enter the BIOS setup interface by hitting <code>F2</code> before the boot loader is started. This can be done on the VGA console, CIMC virtual VGA console, or even the serial console. From the main screen, right arrow over to the Advanced screen and then down to the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu option. From there, navigate down to <code>Set Factory Defaults</code>, select it, and reboot server. Then enter the BIOS setup screens before control is passed to the boot loader again. This time through, the existing virtual drives will be deleted:
* From main BIOS config menu screen, arrow right to Advanced
* Arrow down to <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> and press Enter
* Arrow down to <code>View Server Profile</code> and press Enter
* Arrow down to <code>Virtual Drive Management</code> and press Enter
* A list of virtual drives is now presented. For each virtual drive in the list:
** Press Enter to see the Virtual Drive Details
** Press Enter again, to open the Operation menu
** Arrow down to <code>Delete Virtual Drive</code> and press Enter
** Select the <code>Go</code> option presented beneath the now selected <code>Delete Virtual Drive</code> operation.
** Change the <code>Confirm</code> setting in the next menu to <code>Enabled</code>.
** Select <code>Yes</code> in the menu.
** Select <code>OK</code> to return to the list of existing virtual drives.
* When the list of virtual drives is empty, return to the top level of the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu.
* Select <code>Configure</code> from the menu.
* Select <code>Make JBOD</code> from the menu.
* Select <code>Check All</code> from the menu. This option is displayed after the list of physical drives the controller has found.
* Select <code>OK</code> from the menu.
* Move up and change the <code>Confirm</code> setting to <code>Enabled</code>
* Select <code>Yes</code> from the menu.
* Return to the top level of the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu.
* Save configuration, reboot server, and enter the BIOS setup screen one more time (press <code>F2</code>).
* From main BIOS config menu screen, arrow right to Advanced
* Arrow down to <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> and press Enter
* Arrow down to <code>View Server Profile</code> and press Enter
* Arrow down to <code>Controller Management</code> and press Enter
* Arrow down to <code>Advanced Controller Properties</code> and press Enter
* Arrow down to <code>JBOD Mode</code> and set to <code>Enabled</code> if it is not already so.
* Save changes and reset the system one more time
From here, operating system installation can commence. Or storage device firmware updates. Or whatever...

== Storage media with SCSI Protection Information need not apply ==
Yeah, so the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> does not like these at all. These would be drives that do not have 512-byte or 4096-byte sectors. Often such drives are NetApp or EMC or 3PAR pulls. Drop into a Dell machine with perhaps a LSI SAS2008 instead and do something like
$ '''sudo sg_format --verbose --verbose --format --quick --size=512 --fmtpinfo=0 --pfu=0 /dev/sg3'''
to such a drive before trying anything in the UCS server. Ugh...

== JBOD operations ==
JBOD mode, right? Plug in a disk, maybe scan the bus, watch it go, right? Well, no. That would be too easy. Instead, one can SSH to the CIMC and do something like this:
C220-WZP22040X50# '''scope chassis'''
C220-WZP22040X50 /chassis # '''scope storageadapter MRAID'''
C220-WZP22040X50 /chassis/storageadapter # '''scope physical-drive 2'''
C220-WZP22040X50 /chassis/storageadapter/physical-drive # '''make-jbod
C220-WZP22040X50 /chassis/storageadapter/physical-drive* # '''commit-buffer'''
C220-WZP22040X50 /chassis/storageadapter/physical-drive #
And then search for it from the OS. Hopefully.

== Hardware Fault Resolution ==
Before arriving in the data center and being put into service, my first UCS C series server had a memory DIMM removed to count how much RAM was installed. It was won at auction and no hardware specs were supplied. The DIMM was not correctly re-seated before the server was powered on. The POST code correctly flagged the issue, deactivated the DIMM, and made entries in the System Event Log (SEL) '''and''' a Cisco specific Fault History Log. The SEL is easy enough to clear with <code>ipmitool</code> from the running OS, or from the CIMC. However, the Fault History Log is separate. <strike>Google</strike> Duck Duck Go search results have indicated the server's Fault History Log may be clearable with the Cisco UCS Manager software. [[UCS Manager]] is a whole separate thing and has its own page.

Cisco UCS Rack servers

2026-03-26T15:53:36Z

Adj: /* Server Power Control */

Acquired a Cisco UCS C220 M5 on 2026-03-10. Here are some notes.

== Network based management ==
CIMC, the Cisco Integrated Management Controller. On the machine I have obtained, this was not configured to use the dedicated management Ethernet interface on the back of the server. This is easily corrected while the machine starts up. Pressing F8 from a USB keyboard during the server's power on self test sequence will bring up the CIMC configuration screen. From there, the dedicated NIC option, IPv4 option, and DHCP enabled can be selected to do the expected things. After those are selected, press the <code><F1></code> key and fix the dedicated NIC port properties if the need any adjustment.

The CIMC is accessible over https and SSH. At first login (at least by HTTPS), a new password must be set. None of Dell's just warning "Heh, you have the default credentials. Not recommended!" warning stuff.

== Firmware updates ==
As acquired, server s/n WZP22040X50 is running BIOS version C220M5.3.1.2b.0.1025170315 and CIMC version 3.1(2c). Cisco's firmware update process is heavier weight than some other vendors'. More like HP(E)'s Service Pack for Proliant than Dells "get iDRAC to download update images from an FTP or web server." An ISO image must be downloaded and booted from. (This is maybe avoidable. And if is, updates will be made to document the process here.) As we are running 3.1 release code now, download the final update ISO image, 3.1(3k). This is free-as-in-beer with an account on Cisco's support site. No $$$ required. I have not read every line of the release notes to understand whether entire release trains can be skipped in the upgrade process.

One thing that is nice about getting the account created: they have a notification system a customer can sign up for when sends an email when new firmware is released. With options for weekly, daily, or monthly notications.

After the ISO is downloaded, open the KVM on the server's CIMC and attach it to the server using the appropriate item in the "Virtual Media" menu. Reboot server and press the <code><F6></code> key at the appropriate time to invoke the boot device selection menu. From there, boot from the Cisco vKVM attached ISO image. Wait patiently. After several minutes, you might catch Linux kernel boot messages. And shortly after that, something about the Cisco UCS Host Upgrade Utility Version 3.1.3k doing its work. Then wait some more. The Host Update Utility is not very good about giving progress updates. At least not this version. After 5-10 minutes an "accept the license" screen will appear. Do the obvious. And then, the easy thing is to use the "Update All" button and get all updates applied. Wait patiently some more. But at least we get some progress reports as the actual updates are applied. After all desired updates are complete, select "Exit" The CIMC will reboot if its code was updated. And the server will as well.

If the paranoia is very strong with you, Host Upgrade Utility ISO can be booted from a second time, and all updates verified. The boot and utility loading are just as excruciatingly slow as the actual update, but the verification itself goes swiftly.

First thing of note with going from CIMC 3.1(2c) to 3.1(3k) is that the CIMC's SSH server now supports more modern SSH host keys. No more need for <code>-o HostKeyAlgorithms=+ssh-rsa</code> when pointing an SSH client at the CIMC.

Also, following the release notes, jumping straight from 3.1(2c) to the most recent 4.3(2.260007) is almost certainly not a thing. I have been doing it in increments so far: 3.1(2c) -> 3.1(3k) -> 4.0(4n) -> 4.1(3n) -> 4.2(3l)

4.1(3n) includes some sort of cloud management functionality which I probably don't want. It is called "Cisco Intersight." It will get its own section.

== Cisco Intersight Cloud Management ==
Yeah, what I said just above. I am pretty sure I don't want this. Do not need this server's BMC phoning home to the Cisco mothership. Need to do some reading about turning it off.

It turns out to not to too difficult. Kinda hidden, but here's how I did it:

# Log into CIMC web UI. You will be looking at the Chassis / Summary page.
# Click on the icon in the top left corner. It is pretty much a three bar "hamburger" menu with a right-pointing arrow over its top.
# At the bottom of the just-exposed navigation pane, select the Admin / Device Connector link.
# On the right side of the Device Connector page, find the "Settings" link and follow it.
# Toggle Device Connector off, save settings, enjoy some peace about not having your server watching you quite as much.

The above is what I did on CIMC 4.1(3n). It may be different in newer releases. It may also be re-activated after further firmware updates. So make a note to check this one after more of those.

== Doing things with the CIMC ==
=== Restarting the CIMC ===
The web UI has a "CIMC Reboot" link toward the top right of the main page. At least while running CIMC 4.3 code. From an SSH session, do the following after logging in:
C220-WZP22040X50# '''scope cimc'''
C220-WZP22040X50 /cimc # '''reboot'''
This operation will reboot the Cisco IMC.
Continue?[y|N]'''y'''
and wait for it to restart.

=== Server Power Control ===
Pretty easy:
C220-WZP22040X50#
C220-WZP22040X50# '''scope chassis'''
C220-WZP22040X50 /chassis # '''power on'''
Do you want to continue?[y|N]'''y'''
C220-WZP22040X50 /chassis # '''exit'''
C220-WZP22040X50#

It also has some other options:
C220-WZP22040X50#
C220-WZP22040X50# '''scope chassis'''
C220-WZP22040X50 /chassis # '''power'''
cycle Power off then power on
hard-reset Equivalent to pressing Front Panel reset button or IPMI reset
off Turn power off (OS given short time to shutdown)
on Turn power on
shutdown Gracefully shutdown the host OS & power off

C220-WZP22040X50 /chassis # '''power'''

=== Serial over LAN ===
This is a bit different than Dell's iDRAC or HP(E)'s iLO. It gets its own dedicated SSH port. From an SSH connection to the CIMC, the configuration can be listed out like so:
C220-WZP22040X50# '''show sol detail'''
Serial Over LAN:
Enabled: no
Baud Rate(bps): 115200
Com Port: com0
SOL SSH Port: 2400
C220-WZP22040X50#

It does need to be explicitly enabled. Again, to do that from the CIMC SSH interface (there are maybe BIOS config and CMIC web interface toggles for this, too) do this:
C220-WZP22040X50# '''scope sol'''
C220-WZP22040X50 /sol # '''set enabled yes'''
C220-WZP22040X50 /sol *# '''commit'''
C220-WZP22040X50 /sol # '''exit'''
C220-WZP22040X50#

And that is done, so connect to the SSH server on port 2400. With OpenSSH, that is done by specifying <code>-p 2400</code> on the <code>ssh</code> command line. Something like:
$ '''ssh -p 2400 admin@172.16.10.160'''
admin@172.16.10.160's password:
CISCO Serial Over LAN:
Press Ctrl+x to Exit the session

CISCO Serial Over LAN exited
Connection to 172.16.10.160 closed.
$

== Mass storage devices, RAID, JBOD, and such ==
The ECS C220 M5 has a Broadcom/LSI MegaRAID Tri-Mode SAS3516 attached to the disk backplane in the front of the chassis.

I prefer running my disk interfaces in JBOD (just a bunch of disks) or pass-through mode, exposing the individual physical storage devices to the operating system, and using OS tooling to manage redundancy. As this server arrived running VMware ESXi, an OS without those data redundancy management features, the server came using the SAS3516 RAID controller set up to present 2 virtual disks to the OS. Setting the controller to JBOD mode is possible, though. To get to the MegaRAID configuration without a running OS equipped with <code>storcli</code> and related tools, restart the server, and enter the BIOS setup interface by hitting <code>F2</code> before the boot loader is started. This can be done on the VGA console, CIMC virtual VGA console, or even the serial console. From the main screen, right arrow over to the Advanced screen and then down to the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu option. From there, navigate down to <code>Set Factory Defaults</code>, select it, and reboot server. Then enter the BIOS setup screens before control is passed to the boot loader again. This time through, the existing virtual drives will be deleted:
* From main BIOS config menu screen, arrow right to Advanced
* Arrow down to <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> and press Enter
* Arrow down to <code>View Server Profile</code> and press Enter
* Arrow down to <code>Virtual Drive Management</code> and press Enter
* A list of virtual drives is now presented. For each virtual drive in the list:
** Press Enter to see the Virtual Drive Details
** Press Enter again, to open the Operation menu
** Arrow down to <code>Delete Virtual Drive</code> and press Enter
** Select the <code>Go</code> option presented beneath the now selected <code>Delete Virtual Drive</code> operation.
** Change the <code>Confirm</code> setting in the next menu to <code>Enabled</code>.
** Select <code>Yes</code> in the menu.
** Select <code>OK</code> to return to the list of existing virtual drives.
* When the list of virtual drives is empty, return to the top level of the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu.
* Select <code>Configure</code> from the menu.
* Select <code>Make JBOD</code> from the menu.
* Select <code>Check All</code> from the menu. This option is displayed after the list of physical drives the controller has found.
* Select <code>OK</code> from the menu.
* Move up and change the <code>Confirm</code> setting to <code>Enabled</code>
* Select <code>Yes</code> from the menu.
* Return to the top level of the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu.
* Save configuration, reboot server, and enter the BIOS setup screen one more time (press <code>F2</code>).
* From main BIOS config menu screen, arrow right to Advanced
* Arrow down to <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> and press Enter
* Arrow down to <code>View Server Profile</code> and press Enter
* Arrow down to <code>Controller Management</code> and press Enter
* Arrow down to <code>Advanced Controller Properties</code> and press Enter
* Arrow down to <code>JBOD Mode</code> and set to <code>Enabled</code> if it is not already so.
* Save changes and reset the system one more time
From here, operating system installation can commence. Or storage device firmware updates. Or whatever...

== Storage media with SCSI Protection Information need not apply ==
Yeah, so the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> does not like these at all. These would be drives that do not have 512-byte or 4096-byte sectors. Often such drives are NetApp or EMC or 3PAR pulls. Drop into a Dell machine with perhaps a LSI SAS2008 instead and do something like
$ '''sudo sg_format --verbose --verbose --format --quick --size=512 --fmtpinfo=0 --pfu=0 /dev/sg3'''
to such a drive before trying anything in the UCS server. Ugh...

== JBOD operations ==
JBOD mode, right? Plug in a disk, maybe scan the bus, watch it go, right? Well, no. That would be too easy. Instead, one can SSH to the CIMC and do something like this:
C220-WZP22040X50# '''scope chassis'''
C220-WZP22040X50 /chassis # '''scope storageadapter MRAID'''
C220-WZP22040X50 /chassis/storageadapter # '''scope physical-drive 2'''
C220-WZP22040X50 /chassis/storageadapter/physical-drive # '''make-jbod
C220-WZP22040X50 /chassis/storageadapter/physical-drive* # '''commit-buffer'''
C220-WZP22040X50 /chassis/storageadapter/physical-drive #
And then search for it from the OS. Hopefully.

== Hardware Fault Resolution ==
Before arriving in the data center and being put into service, my first UCS C series server had a memory DIMM removed to count how much RAM was installed. It was won at auction and no hardware specs were supplied. The DIMM was not correctly re-seated before the server was powered on. The POST code correctly flagged the issue, deactivated the DIMM, and made entries in the System Event Log (SEL) '''and''' a Cisco specific Fault History Log. The SEL is easy enough to clear with <code>ipmitool</code> from the running OS, or from the CIMC. However, the Fault History Log is separate. <strike>Google</strike> Duck Duck Go search results have indicated the server's Fault History Log may be clearable with the Cisco UCS Manager software. [[UCS Manager]] is a whole separate thing and has its own page.

Cisco UCS Rack servers

2026-03-26T15:30:33Z

Adj: /* Doing things with the CIMC */

Acquired a Cisco UCS C220 M5 on 2026-03-10. Here are some notes.

== Network based management ==
CIMC, the Cisco Integrated Management Controller. On the machine I have obtained, this was not configured to use the dedicated management Ethernet interface on the back of the server. This is easily corrected while the machine starts up. Pressing F8 from a USB keyboard during the server's power on self test sequence will bring up the CIMC configuration screen. From there, the dedicated NIC option, IPv4 option, and DHCP enabled can be selected to do the expected things. After those are selected, press the <code><F1></code> key and fix the dedicated NIC port properties if the need any adjustment.

The CIMC is accessible over https and SSH. At first login (at least by HTTPS), a new password must be set. None of Dell's just warning "Heh, you have the default credentials. Not recommended!" warning stuff.

== Firmware updates ==
As acquired, server s/n WZP22040X50 is running BIOS version C220M5.3.1.2b.0.1025170315 and CIMC version 3.1(2c). Cisco's firmware update process is heavier weight than some other vendors'. More like HP(E)'s Service Pack for Proliant than Dells "get iDRAC to download update images from an FTP or web server." An ISO image must be downloaded and booted from. (This is maybe avoidable. And if is, updates will be made to document the process here.) As we are running 3.1 release code now, download the final update ISO image, 3.1(3k). This is free-as-in-beer with an account on Cisco's support site. No $$$ required. I have not read every line of the release notes to understand whether entire release trains can be skipped in the upgrade process.

One thing that is nice about getting the account created: they have a notification system a customer can sign up for when sends an email when new firmware is released. With options for weekly, daily, or monthly notications.

After the ISO is downloaded, open the KVM on the server's CIMC and attach it to the server using the appropriate item in the "Virtual Media" menu. Reboot server and press the <code><F6></code> key at the appropriate time to invoke the boot device selection menu. From there, boot from the Cisco vKVM attached ISO image. Wait patiently. After several minutes, you might catch Linux kernel boot messages. And shortly after that, something about the Cisco UCS Host Upgrade Utility Version 3.1.3k doing its work. Then wait some more. The Host Update Utility is not very good about giving progress updates. At least not this version. After 5-10 minutes an "accept the license" screen will appear. Do the obvious. And then, the easy thing is to use the "Update All" button and get all updates applied. Wait patiently some more. But at least we get some progress reports as the actual updates are applied. After all desired updates are complete, select "Exit" The CIMC will reboot if its code was updated. And the server will as well.

If the paranoia is very strong with you, Host Upgrade Utility ISO can be booted from a second time, and all updates verified. The boot and utility loading are just as excruciatingly slow as the actual update, but the verification itself goes swiftly.

First thing of note with going from CIMC 3.1(2c) to 3.1(3k) is that the CIMC's SSH server now supports more modern SSH host keys. No more need for <code>-o HostKeyAlgorithms=+ssh-rsa</code> when pointing an SSH client at the CIMC.

Also, following the release notes, jumping straight from 3.1(2c) to the most recent 4.3(2.260007) is almost certainly not a thing. I have been doing it in increments so far: 3.1(2c) -> 3.1(3k) -> 4.0(4n) -> 4.1(3n) -> 4.2(3l)

4.1(3n) includes some sort of cloud management functionality which I probably don't want. It is called "Cisco Intersight." It will get its own section.

== Cisco Intersight Cloud Management ==
Yeah, what I said just above. I am pretty sure I don't want this. Do not need this server's BMC phoning home to the Cisco mothership. Need to do some reading about turning it off.

It turns out to not to too difficult. Kinda hidden, but here's how I did it:

# Log into CIMC web UI. You will be looking at the Chassis / Summary page.
# Click on the icon in the top left corner. It is pretty much a three bar "hamburger" menu with a right-pointing arrow over its top.
# At the bottom of the just-exposed navigation pane, select the Admin / Device Connector link.
# On the right side of the Device Connector page, find the "Settings" link and follow it.
# Toggle Device Connector off, save settings, enjoy some peace about not having your server watching you quite as much.

The above is what I did on CIMC 4.1(3n). It may be different in newer releases. It may also be re-activated after further firmware updates. So make a note to check this one after more of those.

== Doing things with the CIMC ==
=== Restarting the CIMC ===
The web UI has a "CIMC Reboot" link toward the top right of the main page. At least while running CIMC 4.3 code. From an SSH session, do the following after logging in:
C220-WZP22040X50# '''scope cimc'''
C220-WZP22040X50 /cimc # '''reboot'''
This operation will reboot the Cisco IMC.
Continue?[y|N]'''y'''
and wait for it to restart.

=== Server Power Control ===
Pretty easy:
C220-WZP22040X50#
C220-WZP22040X50# '''scope chassis'''
C220-WZP22040X50 /chassis # '''power on'''
Do you want to continue?[y|N]'''y'''
C220-WZP22040X50 /chassis # '''exit'''
C220-WZP22040X50#

=== Serial over LAN ===
This is a bit different than Dell's iDRAC or HP(E)'s iLO. It gets its own dedicated SSH port. From an SSH connection to the CIMC, the configuration can be listed out like so:
C220-WZP22040X50# '''show sol detail'''
Serial Over LAN:
Enabled: no
Baud Rate(bps): 115200
Com Port: com0
SOL SSH Port: 2400
C220-WZP22040X50#

It does need to be explicitly enabled. Again, to do that from the CIMC SSH interface (there are maybe BIOS config and CMIC web interface toggles for this, too) do this:
C220-WZP22040X50# '''scope sol'''
C220-WZP22040X50 /sol # '''set enabled yes'''
C220-WZP22040X50 /sol *# '''commit'''
C220-WZP22040X50 /sol # '''exit'''
C220-WZP22040X50#

And that is done, so connect to the SSH server on port 2400. With OpenSSH, that is done by specifying <code>-p 2400</code> on the <code>ssh</code> command line. Something like:
$ '''ssh -p 2400 admin@172.16.10.160'''
admin@172.16.10.160's password:
CISCO Serial Over LAN:
Press Ctrl+x to Exit the session

CISCO Serial Over LAN exited
Connection to 172.16.10.160 closed.
$

== Mass storage devices, RAID, JBOD, and such ==
The ECS C220 M5 has a Broadcom/LSI MegaRAID Tri-Mode SAS3516 attached to the disk backplane in the front of the chassis.

I prefer running my disk interfaces in JBOD (just a bunch of disks) or pass-through mode, exposing the individual physical storage devices to the operating system, and using OS tooling to manage redundancy. As this server arrived running VMware ESXi, an OS without those data redundancy management features, the server came using the SAS3516 RAID controller set up to present 2 virtual disks to the OS. Setting the controller to JBOD mode is possible, though. To get to the MegaRAID configuration without a running OS equipped with <code>storcli</code> and related tools, restart the server, and enter the BIOS setup interface by hitting <code>F2</code> before the boot loader is started. This can be done on the VGA console, CIMC virtual VGA console, or even the serial console. From the main screen, right arrow over to the Advanced screen and then down to the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu option. From there, navigate down to <code>Set Factory Defaults</code>, select it, and reboot server. Then enter the BIOS setup screens before control is passed to the boot loader again. This time through, the existing virtual drives will be deleted:
* From main BIOS config menu screen, arrow right to Advanced
* Arrow down to <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> and press Enter
* Arrow down to <code>View Server Profile</code> and press Enter
* Arrow down to <code>Virtual Drive Management</code> and press Enter
* A list of virtual drives is now presented. For each virtual drive in the list:
** Press Enter to see the Virtual Drive Details
** Press Enter again, to open the Operation menu
** Arrow down to <code>Delete Virtual Drive</code> and press Enter
** Select the <code>Go</code> option presented beneath the now selected <code>Delete Virtual Drive</code> operation.
** Change the <code>Confirm</code> setting in the next menu to <code>Enabled</code>.
** Select <code>Yes</code> in the menu.
** Select <code>OK</code> to return to the list of existing virtual drives.
* When the list of virtual drives is empty, return to the top level of the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu.
* Select <code>Configure</code> from the menu.
* Select <code>Make JBOD</code> from the menu.
* Select <code>Check All</code> from the menu. This option is displayed after the list of physical drives the controller has found.
* Select <code>OK</code> from the menu.
* Move up and change the <code>Confirm</code> setting to <code>Enabled</code>
* Select <code>Yes</code> from the menu.
* Return to the top level of the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu.
* Save configuration, reboot server, and enter the BIOS setup screen one more time (press <code>F2</code>).
* From main BIOS config menu screen, arrow right to Advanced
* Arrow down to <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> and press Enter
* Arrow down to <code>View Server Profile</code> and press Enter
* Arrow down to <code>Controller Management</code> and press Enter
* Arrow down to <code>Advanced Controller Properties</code> and press Enter
* Arrow down to <code>JBOD Mode</code> and set to <code>Enabled</code> if it is not already so.
* Save changes and reset the system one more time
From here, operating system installation can commence. Or storage device firmware updates. Or whatever...

== Storage media with SCSI Protection Information need not apply ==
Yeah, so the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> does not like these at all. These would be drives that do not have 512-byte or 4096-byte sectors. Often such drives are NetApp or EMC or 3PAR pulls. Drop into a Dell machine with perhaps a LSI SAS2008 instead and do something like
$ '''sudo sg_format --verbose --verbose --format --quick --size=512 --fmtpinfo=0 --pfu=0 /dev/sg3'''
to such a drive before trying anything in the UCS server. Ugh...

== JBOD operations ==
JBOD mode, right? Plug in a disk, maybe scan the bus, watch it go, right? Well, no. That would be too easy. Instead, one can SSH to the CIMC and do something like this:
C220-WZP22040X50# '''scope chassis'''
C220-WZP22040X50 /chassis # '''scope storageadapter MRAID'''
C220-WZP22040X50 /chassis/storageadapter # '''scope physical-drive 2'''
C220-WZP22040X50 /chassis/storageadapter/physical-drive # '''make-jbod
C220-WZP22040X50 /chassis/storageadapter/physical-drive* # '''commit-buffer'''
C220-WZP22040X50 /chassis/storageadapter/physical-drive #
And then search for it from the OS. Hopefully.

== Hardware Fault Resolution ==
Before arriving in the data center and being put into service, my first UCS C series server had a memory DIMM removed to count how much RAM was installed. It was won at auction and no hardware specs were supplied. The DIMM was not correctly re-seated before the server was powered on. The POST code correctly flagged the issue, deactivated the DIMM, and made entries in the System Event Log (SEL) '''and''' a Cisco specific Fault History Log. The SEL is easy enough to clear with <code>ipmitool</code> from the running OS, or from the CIMC. However, the Fault History Log is separate. <strike>Google</strike> Duck Duck Go search results have indicated the server's Fault History Log may be clearable with the Cisco UCS Manager software. [[UCS Manager]] is a whole separate thing and has its own page.

UCS Manager

2026-03-24T22:44:15Z

Adj: /* CIMC networking */

UCS Manager

2026-03-24T22:41:51Z

Adj: /* Server and Infrastructure issues */

Cisco UCS Rack servers

2026-03-24T21:13:51Z

Adj: /* JBOD operations */

Acquired a Cisco UCS C220 M5 on 2026-03-10. Here are some notes.

== Network based management ==
CIMC, the Cisco Integrated Management Controller. On the machine I have obtained, this was not configured to use the dedicated management Ethernet interface on the back of the server. This is easily corrected while the machine starts up. Pressing F8 from a USB keyboard during the server's power on self test sequence will bring up the CIMC configuration screen. From there, the dedicated NIC option, IPv4 option, and DHCP enabled can be selected to do the expected things. After those are selected, press the <code><F1></code> key and fix the dedicated NIC port properties if the need any adjustment.

The CIMC is accessible over https and SSH. At first login (at least by HTTPS), a new password must be set. None of Dell's just warning "Heh, you have the default credentials. Not recommended!" warning stuff.

== Firmware updates ==
As acquired, server s/n WZP22040X50 is running BIOS version C220M5.3.1.2b.0.1025170315 and CIMC version 3.1(2c). Cisco's firmware update process is heavier weight than some other vendors'. More like HP(E)'s Service Pack for Proliant than Dells "get iDRAC to download update images from an FTP or web server." An ISO image must be downloaded and booted from. (This is maybe avoidable. And if is, updates will be made to document the process here.) As we are running 3.1 release code now, download the final update ISO image, 3.1(3k). This is free-as-in-beer with an account on Cisco's support site. No $$$ required. I have not read every line of the release notes to understand whether entire release trains can be skipped in the upgrade process.

One thing that is nice about getting the account created: they have a notification system a customer can sign up for when sends an email when new firmware is released. With options for weekly, daily, or monthly notications.

After the ISO is downloaded, open the KVM on the server's CIMC and attach it to the server using the appropriate item in the "Virtual Media" menu. Reboot server and press the <code><F6></code> key at the appropriate time to invoke the boot device selection menu. From there, boot from the Cisco vKVM attached ISO image. Wait patiently. After several minutes, you might catch Linux kernel boot messages. And shortly after that, something about the Cisco UCS Host Upgrade Utility Version 3.1.3k doing its work. Then wait some more. The Host Update Utility is not very good about giving progress updates. At least not this version. After 5-10 minutes an "accept the license" screen will appear. Do the obvious. And then, the easy thing is to use the "Update All" button and get all updates applied. Wait patiently some more. But at least we get some progress reports as the actual updates are applied. After all desired updates are complete, select "Exit" The CIMC will reboot if its code was updated. And the server will as well.

If the paranoia is very strong with you, Host Upgrade Utility ISO can be booted from a second time, and all updates verified. The boot and utility loading are just as excruciatingly slow as the actual update, but the verification itself goes swiftly.

First thing of note with going from CIMC 3.1(2c) to 3.1(3k) is that the CIMC's SSH server now supports more modern SSH host keys. No more need for <code>-o HostKeyAlgorithms=+ssh-rsa</code> when pointing an SSH client at the CIMC.

Also, following the release notes, jumping straight from 3.1(2c) to the most recent 4.3(2.260007) is almost certainly not a thing. I have been doing it in increments so far: 3.1(2c) -> 3.1(3k) -> 4.0(4n) -> 4.1(3n) -> 4.2(3l)

4.1(3n) includes some sort of cloud management functionality which I probably don't want. It is called "Cisco Intersight." It will get its own section.

== Cisco Intersight Cloud Management ==
Yeah, what I said just above. I am pretty sure I don't want this. Do not need this server's BMC phoning home to the Cisco mothership. Need to do some reading about turning it off.

It turns out to not to too difficult. Kinda hidden, but here's how I did it:

# Log into CIMC web UI. You will be looking at the Chassis / Summary page.
# Click on the icon in the top left corner. It is pretty much a three bar "hamburger" menu with a right-pointing arrow over its top.
# At the bottom of the just-exposed navigation pane, select the Admin / Device Connector link.
# On the right side of the Device Connector page, find the "Settings" link and follow it.
# Toggle Device Connector off, save settings, enjoy some peace about not having your server watching you quite as much.

The above is what I did on CIMC 4.1(3n). It may be different in newer releases. It may also be re-activated after further firmware updates. So make a note to check this one after more of those.

== Doing things with the CIMC ==
=== Restarting the CIMC ===
The web UI has a "CIMC Reboot" link toward the top right of the main page. At least while running CIMC 4.3 code. From an SSH session, do the following after logging in:
C220-WZP22040X50# '''scope cimc'''
C220-WZP22040X50 /cimc # '''reboot'''
This operation will reboot the Cisco IMC.
Continue?[y|N]'''y'''
and wait for it to restart.

=== Serial over LAN ===
This is a bit different than Dell's iDRAC or HP(E)'s iLO. It gets its own dedicated SSH port. From an SSH connection to the CIMC, the configuration can be listed out like so:
C220-WZP22040X50# '''show sol detail'''
Serial Over LAN:
Enabled: no
Baud Rate(bps): 115200
Com Port: com0
SOL SSH Port: 2400
C220-WZP22040X50#

It does need to be explicitly enabled. Again, to do that from the CIMC SSH interface (there are maybe BIOS config and CMIC web interface toggles for this, too) do this:
C220-WZP22040X50# '''scope sol'''
C220-WZP22040X50 /sol # '''set enabled yes'''
C220-WZP22040X50 /sol *# '''commit'''
C220-WZP22040X50 /sol # '''exit'''
C220-WZP22040X50#

And that is done, so connect to the SSH server on port 2400. With OpenSSH, that is done by specifying <code>-p 2400</code> on the <code>ssh</code> command line. Something like:
$ '''ssh -p 2400 admin@172.16.10.160'''
admin@172.16.10.160's password:
CISCO Serial Over LAN:
Press Ctrl+x to Exit the session

CISCO Serial Over LAN exited
Connection to 172.16.10.160 closed.
$

== Mass storage devices, RAID, JBOD, and such ==
The ECS C220 M5 has a Broadcom/LSI MegaRAID Tri-Mode SAS3516 attached to the disk backplane in the front of the chassis.

I prefer running my disk interfaces in JBOD (just a bunch of disks) or pass-through mode, exposing the individual physical storage devices to the operating system, and using OS tooling to manage redundancy. As this server arrived running VMware ESXi, an OS without those data redundancy management features, the server came using the SAS3516 RAID controller set up to present 2 virtual disks to the OS. Setting the controller to JBOD mode is possible, though. To get to the MegaRAID configuration without a running OS equipped with <code>storcli</code> and related tools, restart the server, and enter the BIOS setup interface by hitting <code>F2</code> before the boot loader is started. This can be done on the VGA console, CIMC virtual VGA console, or even the serial console. From the main screen, right arrow over to the Advanced screen and then down to the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu option. From there, navigate down to <code>Set Factory Defaults</code>, select it, and reboot server. Then enter the BIOS setup screens before control is passed to the boot loader again. This time through, the existing virtual drives will be deleted:
* From main BIOS config menu screen, arrow right to Advanced
* Arrow down to <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> and press Enter
* Arrow down to <code>View Server Profile</code> and press Enter
* Arrow down to <code>Virtual Drive Management</code> and press Enter
* A list of virtual drives is now presented. For each virtual drive in the list:
** Press Enter to see the Virtual Drive Details
** Press Enter again, to open the Operation menu
** Arrow down to <code>Delete Virtual Drive</code> and press Enter
** Select the <code>Go</code> option presented beneath the now selected <code>Delete Virtual Drive</code> operation.
** Change the <code>Confirm</code> setting in the next menu to <code>Enabled</code>.
** Select <code>Yes</code> in the menu.
** Select <code>OK</code> to return to the list of existing virtual drives.
* When the list of virtual drives is empty, return to the top level of the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu.
* Select <code>Configure</code> from the menu.
* Select <code>Make JBOD</code> from the menu.
* Select <code>Check All</code> from the menu. This option is displayed after the list of physical drives the controller has found.
* Select <code>OK</code> from the menu.
* Move up and change the <code>Confirm</code> setting to <code>Enabled</code>
* Select <code>Yes</code> from the menu.
* Return to the top level of the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu.
* Save configuration, reboot server, and enter the BIOS setup screen one more time (press <code>F2</code>).
* From main BIOS config menu screen, arrow right to Advanced
* Arrow down to <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> and press Enter
* Arrow down to <code>View Server Profile</code> and press Enter
* Arrow down to <code>Controller Management</code> and press Enter
* Arrow down to <code>Advanced Controller Properties</code> and press Enter
* Arrow down to <code>JBOD Mode</code> and set to <code>Enabled</code> if it is not already so.
* Save changes and reset the system one more time
From here, operating system installation can commence. Or storage device firmware updates. Or whatever...

== Storage media with SCSI Protection Information need not apply ==
Yeah, so the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> does not like these at all. These would be drives that do not have 512-byte or 4096-byte sectors. Often such drives are NetApp or EMC or 3PAR pulls. Drop into a Dell machine with perhaps a LSI SAS2008 instead and do something like
$ '''sudo sg_format --verbose --verbose --format --quick --size=512 --fmtpinfo=0 --pfu=0 /dev/sg3'''
to such a drive before trying anything in the UCS server. Ugh...

== JBOD operations ==
JBOD mode, right? Plug in a disk, maybe scan the bus, watch it go, right? Well, no. That would be too easy. Instead, one can SSH to the CIMC and do something like this:
C220-WZP22040X50# '''scope chassis'''
C220-WZP22040X50 /chassis # '''scope storageadapter MRAID'''
C220-WZP22040X50 /chassis/storageadapter # '''scope physical-drive 2'''
C220-WZP22040X50 /chassis/storageadapter/physical-drive # '''make-jbod
C220-WZP22040X50 /chassis/storageadapter/physical-drive* # '''commit-buffer'''
C220-WZP22040X50 /chassis/storageadapter/physical-drive #
And then search for it from the OS. Hopefully.

== Hardware Fault Resolution ==
Before arriving in the data center and being put into service, my first UCS C series server had a memory DIMM removed to count how much RAM was installed. It was won at auction and no hardware specs were supplied. The DIMM was not correctly re-seated before the server was powered on. The POST code correctly flagged the issue, deactivated the DIMM, and made entries in the System Event Log (SEL) '''and''' a Cisco specific Fault History Log. The SEL is easy enough to clear with <code>ipmitool</code> from the running OS, or from the CIMC. However, the Fault History Log is separate. <strike>Google</strike> Duck Duck Go search results have indicated the server's Fault History Log may be clearable with the Cisco UCS Manager software. [[UCS Manager]] is a whole separate thing and has its own page.

Cisco UCS Rack servers

2026-03-24T21:07:17Z

Adj: /* Storage media with SCSI Protection Information need not apply */

Acquired a Cisco UCS C220 M5 on 2026-03-10. Here are some notes.

== Network based management ==
CIMC, the Cisco Integrated Management Controller. On the machine I have obtained, this was not configured to use the dedicated management Ethernet interface on the back of the server. This is easily corrected while the machine starts up. Pressing F8 from a USB keyboard during the server's power on self test sequence will bring up the CIMC configuration screen. From there, the dedicated NIC option, IPv4 option, and DHCP enabled can be selected to do the expected things. After those are selected, press the <code><F1></code> key and fix the dedicated NIC port properties if the need any adjustment.

The CIMC is accessible over https and SSH. At first login (at least by HTTPS), a new password must be set. None of Dell's just warning "Heh, you have the default credentials. Not recommended!" warning stuff.

== Firmware updates ==
As acquired, server s/n WZP22040X50 is running BIOS version C220M5.3.1.2b.0.1025170315 and CIMC version 3.1(2c). Cisco's firmware update process is heavier weight than some other vendors'. More like HP(E)'s Service Pack for Proliant than Dells "get iDRAC to download update images from an FTP or web server." An ISO image must be downloaded and booted from. (This is maybe avoidable. And if is, updates will be made to document the process here.) As we are running 3.1 release code now, download the final update ISO image, 3.1(3k). This is free-as-in-beer with an account on Cisco's support site. No $$$ required. I have not read every line of the release notes to understand whether entire release trains can be skipped in the upgrade process.

One thing that is nice about getting the account created: they have a notification system a customer can sign up for when sends an email when new firmware is released. With options for weekly, daily, or monthly notications.

After the ISO is downloaded, open the KVM on the server's CIMC and attach it to the server using the appropriate item in the "Virtual Media" menu. Reboot server and press the <code><F6></code> key at the appropriate time to invoke the boot device selection menu. From there, boot from the Cisco vKVM attached ISO image. Wait patiently. After several minutes, you might catch Linux kernel boot messages. And shortly after that, something about the Cisco UCS Host Upgrade Utility Version 3.1.3k doing its work. Then wait some more. The Host Update Utility is not very good about giving progress updates. At least not this version. After 5-10 minutes an "accept the license" screen will appear. Do the obvious. And then, the easy thing is to use the "Update All" button and get all updates applied. Wait patiently some more. But at least we get some progress reports as the actual updates are applied. After all desired updates are complete, select "Exit" The CIMC will reboot if its code was updated. And the server will as well.

If the paranoia is very strong with you, Host Upgrade Utility ISO can be booted from a second time, and all updates verified. The boot and utility loading are just as excruciatingly slow as the actual update, but the verification itself goes swiftly.

First thing of note with going from CIMC 3.1(2c) to 3.1(3k) is that the CIMC's SSH server now supports more modern SSH host keys. No more need for <code>-o HostKeyAlgorithms=+ssh-rsa</code> when pointing an SSH client at the CIMC.

Also, following the release notes, jumping straight from 3.1(2c) to the most recent 4.3(2.260007) is almost certainly not a thing. I have been doing it in increments so far: 3.1(2c) -> 3.1(3k) -> 4.0(4n) -> 4.1(3n) -> 4.2(3l)

4.1(3n) includes some sort of cloud management functionality which I probably don't want. It is called "Cisco Intersight." It will get its own section.

== Cisco Intersight Cloud Management ==
Yeah, what I said just above. I am pretty sure I don't want this. Do not need this server's BMC phoning home to the Cisco mothership. Need to do some reading about turning it off.

It turns out to not to too difficult. Kinda hidden, but here's how I did it:

# Log into CIMC web UI. You will be looking at the Chassis / Summary page.
# Click on the icon in the top left corner. It is pretty much a three bar "hamburger" menu with a right-pointing arrow over its top.
# At the bottom of the just-exposed navigation pane, select the Admin / Device Connector link.
# On the right side of the Device Connector page, find the "Settings" link and follow it.
# Toggle Device Connector off, save settings, enjoy some peace about not having your server watching you quite as much.

The above is what I did on CIMC 4.1(3n). It may be different in newer releases. It may also be re-activated after further firmware updates. So make a note to check this one after more of those.

== Doing things with the CIMC ==
=== Restarting the CIMC ===
The web UI has a "CIMC Reboot" link toward the top right of the main page. At least while running CIMC 4.3 code. From an SSH session, do the following after logging in:
C220-WZP22040X50# '''scope cimc'''
C220-WZP22040X50 /cimc # '''reboot'''
This operation will reboot the Cisco IMC.
Continue?[y|N]'''y'''
and wait for it to restart.

=== Serial over LAN ===
This is a bit different than Dell's iDRAC or HP(E)'s iLO. It gets its own dedicated SSH port. From an SSH connection to the CIMC, the configuration can be listed out like so:
C220-WZP22040X50# '''show sol detail'''
Serial Over LAN:
Enabled: no
Baud Rate(bps): 115200
Com Port: com0
SOL SSH Port: 2400
C220-WZP22040X50#

It does need to be explicitly enabled. Again, to do that from the CIMC SSH interface (there are maybe BIOS config and CMIC web interface toggles for this, too) do this:
C220-WZP22040X50# '''scope sol'''
C220-WZP22040X50 /sol # '''set enabled yes'''
C220-WZP22040X50 /sol *# '''commit'''
C220-WZP22040X50 /sol # '''exit'''
C220-WZP22040X50#

And that is done, so connect to the SSH server on port 2400. With OpenSSH, that is done by specifying <code>-p 2400</code> on the <code>ssh</code> command line. Something like:
$ '''ssh -p 2400 admin@172.16.10.160'''
admin@172.16.10.160's password:
CISCO Serial Over LAN:
Press Ctrl+x to Exit the session

CISCO Serial Over LAN exited
Connection to 172.16.10.160 closed.
$

== Mass storage devices, RAID, JBOD, and such ==
The ECS C220 M5 has a Broadcom/LSI MegaRAID Tri-Mode SAS3516 attached to the disk backplane in the front of the chassis.

I prefer running my disk interfaces in JBOD (just a bunch of disks) or pass-through mode, exposing the individual physical storage devices to the operating system, and using OS tooling to manage redundancy. As this server arrived running VMware ESXi, an OS without those data redundancy management features, the server came using the SAS3516 RAID controller set up to present 2 virtual disks to the OS. Setting the controller to JBOD mode is possible, though. To get to the MegaRAID configuration without a running OS equipped with <code>storcli</code> and related tools, restart the server, and enter the BIOS setup interface by hitting <code>F2</code> before the boot loader is started. This can be done on the VGA console, CIMC virtual VGA console, or even the serial console. From the main screen, right arrow over to the Advanced screen and then down to the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu option. From there, navigate down to <code>Set Factory Defaults</code>, select it, and reboot server. Then enter the BIOS setup screens before control is passed to the boot loader again. This time through, the existing virtual drives will be deleted:
* From main BIOS config menu screen, arrow right to Advanced
* Arrow down to <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> and press Enter
* Arrow down to <code>View Server Profile</code> and press Enter
* Arrow down to <code>Virtual Drive Management</code> and press Enter
* A list of virtual drives is now presented. For each virtual drive in the list:
** Press Enter to see the Virtual Drive Details
** Press Enter again, to open the Operation menu
** Arrow down to <code>Delete Virtual Drive</code> and press Enter
** Select the <code>Go</code> option presented beneath the now selected <code>Delete Virtual Drive</code> operation.
** Change the <code>Confirm</code> setting in the next menu to <code>Enabled</code>.
** Select <code>Yes</code> in the menu.
** Select <code>OK</code> to return to the list of existing virtual drives.
* When the list of virtual drives is empty, return to the top level of the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu.
* Select <code>Configure</code> from the menu.
* Select <code>Make JBOD</code> from the menu.
* Select <code>Check All</code> from the menu. This option is displayed after the list of physical drives the controller has found.
* Select <code>OK</code> from the menu.
* Move up and change the <code>Confirm</code> setting to <code>Enabled</code>
* Select <code>Yes</code> from the menu.
* Return to the top level of the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu.
* Save configuration, reboot server, and enter the BIOS setup screen one more time (press <code>F2</code>).
* From main BIOS config menu screen, arrow right to Advanced
* Arrow down to <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> and press Enter
* Arrow down to <code>View Server Profile</code> and press Enter
* Arrow down to <code>Controller Management</code> and press Enter
* Arrow down to <code>Advanced Controller Properties</code> and press Enter
* Arrow down to <code>JBOD Mode</code> and set to <code>Enabled</code> if it is not already so.
* Save changes and reset the system one more time
From here, operating system installation can commence. Or storage device firmware updates. Or whatever...

== Storage media with SCSI Protection Information need not apply ==
Yeah, so the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> does not like these at all. These would be drives that do not have 512-byte or 4096-byte sectors. Often such drives are NetApp or EMC or 3PAR pulls. Drop into a Dell machine with perhaps a LSI SAS2008 instead and do something like
$ '''sudo sg_format --verbose --verbose --format --quick --size=512 --fmtpinfo=0 --pfu=0 /dev/sg3'''
to such a drive before trying anything in the UCS server. Ugh...

== JBOD operations ==
JBOD mode, right? Plug in a disk, maybe scan the bus, watch it go, right? Well, no. That would be too easy. Instead, one can SSH to the CIMC and do this:
C220-WZP22040X50# '''scope chassis'''
C220-WZP22040X50 /chassis # '''scope storageadapter MRAID'''
C220-WZP22040X50 /chassis/storageadapter # '''scope physical-drive 2'''
C220-WZP22040X50 /chassis/storageadapter/physical-drive # '''make-jbod
C220-WZP22040X50 /chassis/storageadapter/physical-drive* # '''commit-buffer'''
C220-WZP22040X50 /chassis/storageadapter/physical-drive #
And then search for it from the OS. Hopefully.

== Hardware Fault Resolution ==
Before arriving in the data center and being put into service, my first UCS C series server had a memory DIMM removed to count how much RAM was installed. It was won at auction and no hardware specs were supplied. The DIMM was not correctly re-seated before the server was powered on. The POST code correctly flagged the issue, deactivated the DIMM, and made entries in the System Event Log (SEL) '''and''' a Cisco specific Fault History Log. The SEL is easy enough to clear with <code>ipmitool</code> from the running OS, or from the CIMC. However, the Fault History Log is separate. <strike>Google</strike> Duck Duck Go search results have indicated the server's Fault History Log may be clearable with the Cisco UCS Manager software. [[UCS Manager]] is a whole separate thing and has its own page.

Cisco UCS Rack servers

2026-03-24T21:04:06Z

Adj: /* Mass storage devices, RAID, JBOD, and such */

Acquired a Cisco UCS C220 M5 on 2026-03-10. Here are some notes.

== Network based management ==
CIMC, the Cisco Integrated Management Controller. On the machine I have obtained, this was not configured to use the dedicated management Ethernet interface on the back of the server. This is easily corrected while the machine starts up. Pressing F8 from a USB keyboard during the server's power on self test sequence will bring up the CIMC configuration screen. From there, the dedicated NIC option, IPv4 option, and DHCP enabled can be selected to do the expected things. After those are selected, press the <code><F1></code> key and fix the dedicated NIC port properties if the need any adjustment.

The CIMC is accessible over https and SSH. At first login (at least by HTTPS), a new password must be set. None of Dell's just warning "Heh, you have the default credentials. Not recommended!" warning stuff.

== Firmware updates ==
As acquired, server s/n WZP22040X50 is running BIOS version C220M5.3.1.2b.0.1025170315 and CIMC version 3.1(2c). Cisco's firmware update process is heavier weight than some other vendors'. More like HP(E)'s Service Pack for Proliant than Dells "get iDRAC to download update images from an FTP or web server." An ISO image must be downloaded and booted from. (This is maybe avoidable. And if is, updates will be made to document the process here.) As we are running 3.1 release code now, download the final update ISO image, 3.1(3k). This is free-as-in-beer with an account on Cisco's support site. No $$$ required. I have not read every line of the release notes to understand whether entire release trains can be skipped in the upgrade process.

One thing that is nice about getting the account created: they have a notification system a customer can sign up for when sends an email when new firmware is released. With options for weekly, daily, or monthly notications.

After the ISO is downloaded, open the KVM on the server's CIMC and attach it to the server using the appropriate item in the "Virtual Media" menu. Reboot server and press the <code><F6></code> key at the appropriate time to invoke the boot device selection menu. From there, boot from the Cisco vKVM attached ISO image. Wait patiently. After several minutes, you might catch Linux kernel boot messages. And shortly after that, something about the Cisco UCS Host Upgrade Utility Version 3.1.3k doing its work. Then wait some more. The Host Update Utility is not very good about giving progress updates. At least not this version. After 5-10 minutes an "accept the license" screen will appear. Do the obvious. And then, the easy thing is to use the "Update All" button and get all updates applied. Wait patiently some more. But at least we get some progress reports as the actual updates are applied. After all desired updates are complete, select "Exit" The CIMC will reboot if its code was updated. And the server will as well.

If the paranoia is very strong with you, Host Upgrade Utility ISO can be booted from a second time, and all updates verified. The boot and utility loading are just as excruciatingly slow as the actual update, but the verification itself goes swiftly.

First thing of note with going from CIMC 3.1(2c) to 3.1(3k) is that the CIMC's SSH server now supports more modern SSH host keys. No more need for <code>-o HostKeyAlgorithms=+ssh-rsa</code> when pointing an SSH client at the CIMC.

Also, following the release notes, jumping straight from 3.1(2c) to the most recent 4.3(2.260007) is almost certainly not a thing. I have been doing it in increments so far: 3.1(2c) -> 3.1(3k) -> 4.0(4n) -> 4.1(3n) -> 4.2(3l)

4.1(3n) includes some sort of cloud management functionality which I probably don't want. It is called "Cisco Intersight." It will get its own section.

== Cisco Intersight Cloud Management ==
Yeah, what I said just above. I am pretty sure I don't want this. Do not need this server's BMC phoning home to the Cisco mothership. Need to do some reading about turning it off.

It turns out to not to too difficult. Kinda hidden, but here's how I did it:

# Log into CIMC web UI. You will be looking at the Chassis / Summary page.
# Click on the icon in the top left corner. It is pretty much a three bar "hamburger" menu with a right-pointing arrow over its top.
# At the bottom of the just-exposed navigation pane, select the Admin / Device Connector link.
# On the right side of the Device Connector page, find the "Settings" link and follow it.
# Toggle Device Connector off, save settings, enjoy some peace about not having your server watching you quite as much.

The above is what I did on CIMC 4.1(3n). It may be different in newer releases. It may also be re-activated after further firmware updates. So make a note to check this one after more of those.

== Doing things with the CIMC ==
=== Restarting the CIMC ===
The web UI has a "CIMC Reboot" link toward the top right of the main page. At least while running CIMC 4.3 code. From an SSH session, do the following after logging in:
C220-WZP22040X50# '''scope cimc'''
C220-WZP22040X50 /cimc # '''reboot'''
This operation will reboot the Cisco IMC.
Continue?[y|N]'''y'''
and wait for it to restart.

=== Serial over LAN ===
This is a bit different than Dell's iDRAC or HP(E)'s iLO. It gets its own dedicated SSH port. From an SSH connection to the CIMC, the configuration can be listed out like so:
C220-WZP22040X50# '''show sol detail'''
Serial Over LAN:
Enabled: no
Baud Rate(bps): 115200
Com Port: com0
SOL SSH Port: 2400
C220-WZP22040X50#

It does need to be explicitly enabled. Again, to do that from the CIMC SSH interface (there are maybe BIOS config and CMIC web interface toggles for this, too) do this:
C220-WZP22040X50# '''scope sol'''
C220-WZP22040X50 /sol # '''set enabled yes'''
C220-WZP22040X50 /sol *# '''commit'''
C220-WZP22040X50 /sol # '''exit'''
C220-WZP22040X50#

And that is done, so connect to the SSH server on port 2400. With OpenSSH, that is done by specifying <code>-p 2400</code> on the <code>ssh</code> command line. Something like:
$ '''ssh -p 2400 admin@172.16.10.160'''
admin@172.16.10.160's password:
CISCO Serial Over LAN:
Press Ctrl+x to Exit the session

CISCO Serial Over LAN exited
Connection to 172.16.10.160 closed.
$

== Mass storage devices, RAID, JBOD, and such ==
The ECS C220 M5 has a Broadcom/LSI MegaRAID Tri-Mode SAS3516 attached to the disk backplane in the front of the chassis.

I prefer running my disk interfaces in JBOD (just a bunch of disks) or pass-through mode, exposing the individual physical storage devices to the operating system, and using OS tooling to manage redundancy. As this server arrived running VMware ESXi, an OS without those data redundancy management features, the server came using the SAS3516 RAID controller set up to present 2 virtual disks to the OS. Setting the controller to JBOD mode is possible, though. To get to the MegaRAID configuration without a running OS equipped with <code>storcli</code> and related tools, restart the server, and enter the BIOS setup interface by hitting <code>F2</code> before the boot loader is started. This can be done on the VGA console, CIMC virtual VGA console, or even the serial console. From the main screen, right arrow over to the Advanced screen and then down to the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu option. From there, navigate down to <code>Set Factory Defaults</code>, select it, and reboot server. Then enter the BIOS setup screens before control is passed to the boot loader again. This time through, the existing virtual drives will be deleted:
* From main BIOS config menu screen, arrow right to Advanced
* Arrow down to <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> and press Enter
* Arrow down to <code>View Server Profile</code> and press Enter
* Arrow down to <code>Virtual Drive Management</code> and press Enter
* A list of virtual drives is now presented. For each virtual drive in the list:
** Press Enter to see the Virtual Drive Details
** Press Enter again, to open the Operation menu
** Arrow down to <code>Delete Virtual Drive</code> and press Enter
** Select the <code>Go</code> option presented beneath the now selected <code>Delete Virtual Drive</code> operation.
** Change the <code>Confirm</code> setting in the next menu to <code>Enabled</code>.
** Select <code>Yes</code> in the menu.
** Select <code>OK</code> to return to the list of existing virtual drives.
* When the list of virtual drives is empty, return to the top level of the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu.
* Select <code>Configure</code> from the menu.
* Select <code>Make JBOD</code> from the menu.
* Select <code>Check All</code> from the menu. This option is displayed after the list of physical drives the controller has found.
* Select <code>OK</code> from the menu.
* Move up and change the <code>Confirm</code> setting to <code>Enabled</code>
* Select <code>Yes</code> from the menu.
* Return to the top level of the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> menu.
* Save configuration, reboot server, and enter the BIOS setup screen one more time (press <code>F2</code>).
* From main BIOS config menu screen, arrow right to Advanced
* Arrow down to <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> and press Enter
* Arrow down to <code>View Server Profile</code> and press Enter
* Arrow down to <code>Controller Management</code> and press Enter
* Arrow down to <code>Advanced Controller Properties</code> and press Enter
* Arrow down to <code>JBOD Mode</code> and set to <code>Enabled</code> if it is not already so.
* Save changes and reset the system one more time
From here, operating system installation can commence. Or storage device firmware updates. Or whatever...

== Storage media with SCSI Protection Information need not apply ==
Yeah, so the <code>AVAGO <Cisco 12G Modular Raid Controller with 2GB cache (max 16 drives)> Configuration</code> does not like these at all. Drop into a Dell machine with perhaps a LSI SAS2008 instead and do something like
$ '''sudo sg_format --verbose --verbose --format --quick --size=512 --fmtpinfo=0 --pfu=0 /dev/sg3'''
to such a drive before trying anything in the UCS server. Ugh...

== JBOD operations ==
JBOD mode, right? Plug in a disk, maybe scan the bus, watch it go, right? Well, no. That would be too easy. Instead, one can SSH to the CIMC and do this:
C220-WZP22040X50# '''scope chassis'''
C220-WZP22040X50 /chassis # '''scope storageadapter MRAID'''
C220-WZP22040X50 /chassis/storageadapter # '''scope physical-drive 2'''
C220-WZP22040X50 /chassis/storageadapter/physical-drive # '''make-jbod
C220-WZP22040X50 /chassis/storageadapter/physical-drive* # '''commit-buffer'''
C220-WZP22040X50 /chassis/storageadapter/physical-drive #
And then search for it from the OS. Hopefully.

== Hardware Fault Resolution ==
Before arriving in the data center and being put into service, my first UCS C series server had a memory DIMM removed to count how much RAM was installed. It was won at auction and no hardware specs were supplied. The DIMM was not correctly re-seated before the server was powered on. The POST code correctly flagged the issue, deactivated the DIMM, and made entries in the System Event Log (SEL) '''and''' a Cisco specific Fault History Log. The SEL is easy enough to clear with <code>ipmitool</code> from the running OS, or from the CIMC. However, the Fault History Log is separate. <strike>Google</strike> Duck Duck Go search results have indicated the server's Fault History Log may be clearable with the Cisco UCS Manager software. [[UCS Manager]] is a whole separate thing and has its own page.

UCS Manager

2026-03-24T13:23:48Z

Adj: /* I can power on now? */

UCS Manager

2026-03-24T00:41:13Z

Adj: /* I can power on now? */

UCS Manager

2026-03-23T23:30:24Z

Adj: /* Enter an org, see what we have for basic policies */

UCS Manager

2026-03-23T04:52:23Z

Adj: /* Telling the FI to discover the blade chassis */

UCS Manager

2026-03-23T04:25:41Z

Adj:

UCS Manager

2026-03-23T04:12:28Z

Adj: /* SFPs go into the FI and the UCS 5108 FEXes */

UCS Manager

2026-03-22T05:01:48Z

Adj: /* Telling the FI to discover the blade chassis */

UCS Manager

2026-03-22T04:46:30Z

Adj: /* STOP! Don't forget the other UCS things! */

UCS Manager

2026-03-22T03:20:46Z

Adj: /* Software upgrade time! */

UCS Manager

2026-03-22T03:11:56Z

Adj: /* Clock discipline is good */

UCS Manager

2026-03-22T03:11:35Z

Adj: /* Basic Fabric Interconnect configuration */