UCS Manager: Difference between revisions
| Line 1,364: | Line 1,364: | ||
ucs-fi-6248up-0-A /firmware # |
ucs-fi-6248up-0-A /firmware # |
||
Note that the UCS Manager SSH client can now talk to a Debian 12 OpenSSH server. Small wins are still wins! |
Note that the UCS Manager SSH client can now talk to a Debian 12 OpenSSH server. Small wins are still wins! |
||
=== A new step for firmware infrastructure updates === |
|||
It seems that one must accept that a fabric interconnect reboot has happened before applying a new infrastructure firmware pack. Something like this: |
|||
ucs-fi-6248up-0-A# '''scope monitoring''' |
|||
ucs-fi-6248up-0-A /monitoring # '''show new-faults''' |
|||
Severity Code Last Transition Time ID Description |
|||
--------- -------- ------------------------ -------- ----------- |
|||
Info F0440 2026-04-02T18:37:14.718 113208 Chassis discovery policy conflict: Link IOM 1/1/2 to peer port A:N/A/1/2 not configured |
|||
ucs-fi-6248up-0-A /monitoring # '''top''' |
|||
ucs-fi-6248up-0-A# '''scope firmware''' |
|||
ucs-fi-6248up-0-A /firmware # '''scope auto-install''' |
|||
ucs-fi-6248up-0-A /firmware/auto-install # '''acknowledge primary fabric-interconnect reboot ''' |
|||
ucs-fi-6248up-0-A /firmware/auto-install* # '''commit-buffer''' |
|||
ucs-fi-6248up-0-A /firmware/auto-install # '''install infra infra-vers 3.2(3p)A''' |
|||
This operation upgrades firmware on UCS Infrastructure Components |
|||
(UCS manager, Fabric Interconnects and IOMs). |
|||
Here is the checklist of things that are recommended before starting Auto-Install |
|||
(1) Review current critical/major faults |
|||
(2) Initiate a configuration backup |
|||
(3) Check if Management Interface Monitoring Policy is enabled |
|||
(4) Check if there is a pending Fabric Interconnect Reboot activitiy |
|||
(5) Ensure NTP is configured |
|||
(6) Check if any hardware (fabric interconnects, io-modules, servers or adapters) is unsupported in the target release |
|||
(7) Some fabric-interconnect service-pack install/uninstall will do additional FI reboots in order to complete install/uninstall |
|||
To ensure before triggering auto-install, check manual service-pack activation/remove where warning is given if reboots are required |
|||
Do you want to proceed? (yes/no):'''yes''' |
|||
Triggering Install-Infra with: |
|||
Infrastructure Pack Version: 3.2(3p)A |
|||
Warning: Any Service Pack installed in the system will be removed |
|||
ucs-fi-6248up-0-A /firmware/auto-install # '''acknowledge primary fabric-interconnect reboot''' |
|||
Warning: Check the outstanding faults (scope monitoring <enter> show new-faults) since last FI reboot. |
|||
Please make sure the data paths are recovered before proceeding with this FI reboot to ensure there is no interruption to the data traffic. |
|||
ucs-fi-6248up-0-A /firmware/auto-install* # '''commit-buffer ''' |
|||
ucs-fi-6248up-0-A /firmware/auto-install # |
|||
Broadcast message from root (Thu Apr 2 23:36:05 2026): |
|||
The system is going down for reboot NOW! |
|||
Connection to 172.16.10.176 closed by remote host. |
|||
Connection to 172.16.10.176 closed. |
|||
Revision as of 23:37, 2 April 2026
There are a few pieces of Cisco UCS equipment in the data center:
- One UCS 5108 blade enclosure, accommodating up to 8 B200 server blades of various generations
- 2 UCS-FI-6248UP fabric interconnects, which sit between UCS servers or blade chassis and the rest of the network
- 1 UCS C220 M5 server
Cisco UCS Manager gets installed on the Fabric Interconnects and does Cisco proprietary magic things. Like powering on a single blade. This is way more complicated that it needs to be. So this document is being written to capture knowledge of installation, upgrades, and server management tasks.
How to get
Much (all?) of Cisco's UCS software is available directly without a support contract. But a customer account is required to download. Registration is free as in beer. I have had one since July 2021 and have not been troubled with Cisco spam or any sort of "are you a real person at a real company?" validation.
As this is written, UCS Infrastructure and UCS Manager Software is the place to get it. As I have some older, end-of-life, end-of-support fabric interconnects (62xx series) which are not supported on the current 6.0 release train, I have instead selected the 4.2(3p) version for download. This comes as a 1Gbyte-ish ucs-k9-bundle-infra.4.2.3p.A.bin file.
Untested Fabric Interconnect, what do we do?
5ish years ago (summer 2021), I acquired three of these 6248UP FIs. And I think I got one of them configured for use on the management network and updated to then-current software. I'm not sure which of the three that was. And I am sure I did no documentation at the time. So here we are with new notes.
Console and power are at the front?!?!
So, this is weird. Console and power connections are at the front side of the chassis. All of the connectivity is at the rear as it should be. There is sufficient room on the back for a couple of C14 power connectors and 2 more 8P8C modular connectors for management LAN and serial console. But they are at the front. So cabling is kinda sorta extra fun. sigh
Console serial cable is the usual Cisco modular pinout. Signal on pins 3 and 6. 9600 bps, 8-n-1 are the serial port settings. Nothing unusual there.
Mystery solved regarding the connections at front of the chassis. My FIs have 32 ports for traffic, but there is (well, was, back when this was a current product) a 16 port expansion module available. See https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-6200-series-fabric-interconnects/data_sheet_c78-675245.html for some diagrams.
Basic Fabric Interconnect configuration
I might have lucked out on the first one I received, but it booted normally and started the interactive system setup dialog. It is all pretty basic. Set a password for the admin user and an IP address, netmask, IPv4 gateway address, and DNS server address for the management Ethernet interface (again, on the front). That's pretty much it.
After this is done, make sure the management Ethernet interface is connected to a switch on the correct VLAN and subnet. Then do all the things over an SSH connection. At least for version 2.1, the SSH server only knows about RSA host keys, so a modern OpenSSH client will need something like -o HostKeyAlgorithms=+ssh-rsa listed as a command line option.
Clock discipline is good
To set the time in the FI. Or just get it close. NTP will come. Eventually.
ucs-fi-6248up-0-A# scope system ucs-fi-6248up-0-A /system # scope services ucs-fi-6248up-0-A /system/services # set clock Mar 22 2026 02 53 35 Sun Mar 22 02:53:35 UTC 2026 ucs-fi-6248up-0-A /system/services #
And now for the NTP:
ucs-fi-6248up-0-A# scope system
ucs-fi-6248up-0-A /system # scope services
ucs-fi-6248up-0-A /system/services # create ntp-server 172.16.10.2
ucs-fi-6248up-0-A /system/services* # commit-buffer
ucs-fi-6248up-0-A /system/services # create ntp-server 172.16.10.3
ucs-fi-6248up-0-A /system/services* # commit-buffer
ucs-fi-6248up-0-A /system/services # show ntp
NTP Servers:
Name: 172.16.10.2
Name: 172.16.10.3
ucs-fi-6248up-0-A /system/services #
Software upgrade time!
Again, on the first one I have tried, let's log in as admin and see what's running:
Cisco UCS 6200 Series Fabric Interconnect
ucs-fi-6248up-0-A login: admin Password: Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2009, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php ucs-fi-6248up-0-A# show version System version: 2.2(1c) ucs-fi-6248up-0-A#
And, more verbosely (for reasons that are beyond me, "brief" mode shows more output?):
ucs-fi-6248up-0-A# show version brief System version: 2.2(1c) Package-Vers: 2.2(1c)A ucs-fi-6248up-0-A#
So that's really very super dooper old. I downloaded a copy of the 4.1(3i) code in August 2022. Off do so some reading to see if I can skip all the in-between versions.
Did not find anything authoritative in the release notes (but I only looked very briefly.) So, we can do it stepwise. Hopefully. Process looks something like this:
ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # download image tftp://172.17.0.18/ucs-k9-bundle-infra.2.2.8m.A.bin
ucs-fi-6248up-0-A /firmware # show download-task
Download task:
File Name Protocol Server Userid State
--------- -------- --------------- --------------- -----
ucs-k9-bundle-infra.2.2.8m.A.bin
Tftp 172.17.0.18 Downloaded
ucs-fi-6248up-0-A /firmware #
Yes, you read that right. I delivered the software image to the FI over TFTP. SSH (scp and sftp) are supported protocols, but could not connect to an OpenSSH server on Debian 13. This might be better with newer FI firmware, but for now, I already have a TFTP server handy for netbooting other machines in the environment.
And from here, the firmware bundle can actually be installed on the Fabric Interconnect like so:
ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # scope auto-install
ucs-fi-6248up-0-A /firmware/auto-install # install infra infra-vers 2.2(8m)A
This operation upgrades firmware on UCS Infrastructure Components
(UCS manager, Fabric Interconnects and IOMs). Do you want to proceed?
(yes/no):yes
Triggering Install-Infra with:
Infrastructure Pack Version: 2.2(8m)A
ucs-fi-6248up-0-A /firmware/auto-install # acknowledge primary fabric-interconnect reboot
ucs-fi-6248up-0-A /firmware/auto-install* # commit-buffer
ucs-fi-6248up-0-A /firmware/auto-install # show fsm status expand
FSM Status:
Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: In Progress
Completion Time:
Progress (%): 87
FSM Stage:
Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployActivateUCSM Success 0
6 DeployPollActivateOfUCSM In Progress 1
7 DeployUpdateIOM Pending 0
8 DeployPollUpdateOfIOM Pending 0
9 DeployActivateIOM Pending 0
10 DeployPollActivateOfIOM Pending 0
11 DeployActivateRemoteFI Pending 0
12 DeployPollActivateOfRemoteFI Pending 0
13 DeployWaitForUserAck Pending 0
14 DeployActivateLocalFI Pending 0
15 DeployPollActivateOfLocalFI Pending 0
ucs-fi-6248up-0-A /firmware/auto-install # Connection to 172.16.10.176 closed.
Somewhere along the way, the SSH server hung up on me. But that seems like the sort of thing that should happen as the FI's OS is being upgraded. Progress can also be tracked on the serial console port like so:
ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # scope auto-install
ucs-fi-6248up-0-A /firmware/auto-install # show fsm status expand | no-more
FSM Status:
Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: In Progress
Completion Time:
Progress (%): 98
FSM Stage:
Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployDownloadImages Skip 0
6 DeployCopyAllImagesToPeer Skip 0
7 DeployInternalBackup Success 0
8 DeployPollInternalBackup Success 2
9 DeployActivateUCSM Skip 0
10 DeployPollActivateOfUCSM Success 0
11 DeployUpdateIOM Success 0
12 DeployPollUpdateOfIOM Success 0
13 DeployActivateIOM Success 0
14 DeployPollActivateOfIOM Success 0
15 DeployActivateRemoteFI Skip 0
16 DeployPollActivateOfRemoteFI Skip 0
17 DeployWaitForUserAck Skip 0
18 DeployPollWaitForUserAck Success 0
19 DeployActivateLocalFI Success 0
20 DeployPollActivateOfLocalFI In Progress 2
ucs-fi-6248up-0-A /firmware/auto-install #
At some point, the FI will reboot itself.
Confirm new code is running like so:
ucs-fi-6248up-0-A# show version System version: 2.2(8m) ucs-fi-6248up-0-A# show version brief System version: 2.2(8m) Package-Vers: 2.2(8m)A ucs-fi-6248up-0-A#
STOP! Don't forget the other UCS things!
Um, yeah. So there is this UCS 5108 blade chassis in the data center, too. No idea what its firmware versions are. But they are sure to also need firmware updates. And it would be distressing if those components were left in a state where they are unusable because the FIs are so much newer than the blade enclosure and its servers. So I guess we need to see about warming the ZIP code 5degC by attaching the blade chassis to the mains power and see what we can see.
Correct cabling for single FI to UCS 5108
According to https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/chassis-install-guide/ucs5108_install/ucs5108_install_chapter_010.html, a single Fabric Interconnect can only be connected to a single Fabric Extender on a UCS chassis. So for now, enable 6248UP ports 1 and 3, and cable to ports 1 and 2 on the left side (as seen from the rear) Fabric Extender.
SFPs go into the FI and the UCS 5108 FEXes
Pretty self explanatory, right. Find Cisco 10GbE SR SFP+ modules, insert into port one on each of the FEXes, insert into ports 1 and 3 on the FI. Attach nice aqua colored cable. And don't panic as the lights don't come on.
Telling the FI to discover the blade chassis
This is way more complicated that it needs to be. I'm sure it's totally amazing if you have racks full of Cisco UCS hardware. But I have just the one blade chassis and the one C series server and the 2 fabric interconnects. And the friendly FI management software is all lovely out of date (completely unrunnable in 2026!) Java stuff. So fumbling around the CLI, hoping to find my way out of the darkness here.
The process for this seems to be something like:
- Configure FI Ethernet ports as server ports
- Ensure the attached chassis has been discovered and acknowledge any faults
ucs-fi-6248up-0-A# scope eth-server
ucs-fi-6248up-0-A /eth-server # show fabric a detail
Fabric:
Id: A
Current Task:
ucs-fi-6248up-0-A /eth-server # scope fabric a
ucs-fi-6248up-0-A /eth-server/fabric # create interface 1 1
ucs-fi-6248up-0-A /eth-server/fabric/interface* # enable
ucs-fi-6248up-0-A /eth-server/fabric/interface* # show interface detail
ucs-fi-6248up-0-A /eth-server/fabric/interface* # commit-buffer
ucs-fi-6248up-0-A /eth-server/fabric/interface # exit
ucs-fi-6248up-0-A /eth-server/fabric # create interface 1 3
ucs-fi-6248up-0-A /eth-server/fabric/interface* # enable
ucs-fi-6248up-0-A /eth-server/fabric/interface* # commit-buffer
ucs-fi-6248up-0-A /eth-server/fabric/interface # exit
ucs-fi-6248up-0-A /eth-server/fabric # exit
ucs-fi-6248up-0-A /eth-server # show interface
Interface:
Fabric Slot Port Admin State Oper State State Reason Chassis Lic State Grace Prd
------- ----- ----- ------------ ------------ ------------------------------- -------- -------------------- ---------
A 1 1 Enabled Up 1 License Ok 0
A 1 3 Enabled Up 1 License Ok 0
ucs-fi-6248up-0-A /eth-server # exit
ucs-fi-6248up-0-A# show chassis inventory
Chassis PID Vendor Serial (SN) HW Revision
---------- --------------- ----------------- ----------- -----------
1 N20-C6508 Cisco Systems Inc FOX1808GLJG 0
ucs-fi-6248up-0-A#
Success! Serial number reported in the UCS Manager matches the one I found on the label.
So how do I power on a server?
Still fumbling through here. But: servers (blade servers for sure, not sure about rack servers) belong to UCS "Organizations." These organizations are managed by the UCS Manager, too. And since an organization owns a server (blade or otherwise) the organization can power servers on and off.
What orgs exist?
This is what we get from a factory default UCS Manager:
ucs-fi-6248up-0-A# show org
Organizations:
Name
----
/ (root)
ucs-fi-6248up-0-A# show org detail
Organizations:
Name: / (root)
ucs-fi-6248up-0-A#
Enter an org, see what we have for basic policies
ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # show bladeserver-disc-policy
ucs-fi-6248up-0-A /org # show chassis-conn-policy
Chassis/FEX Connectivity Policy:
Chassis/FEX Id Chassis/FEX Name Fabric Id Link Aggregation Preference
-------------- ---------------- --------- ---------------------------
1 sys/chassis-1 A Global
ucs-fi-6248up-0-A /org # show chassis-disc-policy
Chassis/FEX Discovery Policy:
Description Qualifier Action Rebalance Link Aggregation Pref Multicast Hw Hash
----------- ---------- ----------------- ----------------- --------------------- -----------------
none 1 Link User Acknowledged None Disabled
ucs-fi-6248up-0-A /org # show chassis-disc-policy detail
Chassis/FEX Discovery Policy:
Description:
Qualifier: none
Action: 1 Link
Rebalance: User Acknowledged
Link Aggregation Pref: None
Multicast Hw Hash: Disabled
ucs-fi-6248up-0-A /org # show rackserver-disc-policy
Rack Server Discovery Policy:
Action Scrub Policy
----------------- ------------
Immediate
ucs-fi-6248up-0-A /org # show rackserver-disc-policy detail
Rack Server Discovery Policy:
Action: Immediate
Scrub Policy:
Description:
Current Task:
ucs-fi-6248up-0-A /org # show rackserver-mgmt-policy
Rack Server Management Policy:
Action
------
Auto Acknowledged
ucs-fi-6248up-0-A /org # show rackserver-mgmt-policy detail
Rack Server Management Policy:
Action: Auto Acknowledged
ucs-fi-6248up-0-A /org #
Create a profile and add a (blade) server to it
ucs-fi-6248up-0-A# show chassis inventory 1
Chassis PID Vendor Serial (SN) HW Revision
---------- --------------- ----------------- ----------- -----------
1 N20-C6508 Cisco Systems Inc FOX1808GLJG 0
ucs-fi-6248up-0-A# show chassis inventory 1 expand
Chassis 1:
Servers:
Server 1/1:
Equipped Product Name: Cisco UCS B200 M3
Equipped PID: UCSB-B200-M3
Equipped VID: V03
Equipped Serial (SN): FCH162871NA
Slot Status: Equipped
Acknowledged Product Name: Cisco UCS B200 M3
Acknowledged PID: UCSB-B200-M3
Acknowledged VID: V03
Acknowledged Serial (SN): FCH162871NA
Acknowledged Memory (MB): 196608
Acknowledged Effective Memory (MB): 196608
Acknowledged Cores: 0
Acknowledged Adapters: 2
Server 1/2:
Equipped Product Name: Cisco UCS B200 M3
Equipped PID: UCSB-B200-M3
Equipped VID: V06
Equipped Serial (SN): FCH1811JELC
Slot Status: Equipped
Acknowledged Product Name: Cisco UCS B200 M3
[...]
ucs-fi-6248up-0-A# top
ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # create service-profile
ucs-fi-6248up-0-A /org # create service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile* # associate server 1/1
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile #
I can power on now?
Well, no. Maybe it should have worked. But for this blade, there was a Power On Self Test (POST) failure. Let's dig in. To power on the server included in the service profile, do this:
ucs-fi-6248up-0-A# scope org / ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA ucs-fi-6248up-0-A /org/service-profile # power up ucs-fi-6248up-0-A /org/service-profile* # commit-buffer ucs-fi-6248up-0-A /org/service-profile # top ucs-fi-6248up-0-A #
And let's have a look at its status:
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # show
Server:
Slot Overall Status Service Profile Availability
------- --------------------- -------------------- ------------
1 Compute Failed Unavailable
ucs-fi-6248up-0-A /chassis/server # show detail
Server:
Slot: 1
Name:
User Label:
Overall Status: Compute Failed
Oper Qualifier: Compute Post Failure
Service Profile:
Association: None
Availability: Unavailable
Discovery: Failed
Conn Path: A
Conn Status: A
Managing Instance: A
Admin Power: Policy
Oper Power: Off
Admin State: In Service
Product Name: Cisco UCS B200 M3
PID: UCSB-B200-M3
VID: V03
Vendor: Cisco Systems Inc
Serial (SN): FCH162871NA
HW Revision: 0
Mfg Date: 2012-07-26T00:00:00.000
Part Number: 73-13217-08
Memory (MB): 196608
Effective Memory (MB): 196608
Operating Memory Speed (MHz): 1333
Operating Memory Voltage: Regular Voltage
Cores: 0
Num Of Cores Enabled: 0
Adapters: 2
Eth Host Interfaces: 0
FC Host Interfaces: 0
Burned-In UUID: 00000000-0000-0000-0000-000000000000
Dynamic UUID: 00000000-0000-0000-0000-000000000000
Current Task 1: Checking hardware configuration server 1/1(FSM-STAGE:sam:dme:ComputeBladeDiscover:Sanitize)
Current Task 2:
Current Task 3:
ucs-fi-6248up-0-A /chassis/server # show post
POST:
Global ID Code Severity Affected Object Description
--------- --------- --------- -------------------------------- -----------
6215 POST-6215 Critical sys/chassis-1/blade-1 Board Programmable version not valid for Processor Type
ucs-fi-6248up-0-A /chassis/server # show post detail
POST:
Global ID: 6215
Code: POST-6215
Local ID: 1536
Severity: Critical
Affected Object: sys/chassis-1/blade-1
Description: Board Programmable version not valid for Processor Type
Type: server: Cisco Systems Inc UCSB-B200-M3
Recoverable: Non Recoverable
Recovery Action: Refer the CPU upgrade guide.
Timestamp: 2026-03-22T16:17:06.001
ucs-fi-6248up-0-A /chassis/server #
Making an inferences from "Board Programmable version not valid for Processor Type" I am thinking that it does not like the Xeon E5-2650v2 I installed. Now to find a not v2 E5-26xx CPU and see if that works any better. I found a not obviously damaged E5-2609 in storage (er, piles in the garage) and installed that in the blade. Let's see what we have now:
ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # power up
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile # top
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # show inventory
Server 1/1:
Name:
User Label:
Equipped PID: UCSB-B200-M3
Equipped VID: V03
Equipped Serial (SN): FCH162871NA
Slot Status: Equipped
Acknowledged Product Name: Cisco UCS B200 M3
Acknowledged PID: UCSB-B200-M3
Acknowledged VID: V03
Acknowledged Serial (SN): FCH162871NA
Acknowledged Memory (MB): 16384
Acknowledged Effective Memory (MB): 16384
Acknowledged Cores: 4
Acknowledged Adapters: 2
ucs-fi-6248up-0-A /chassis/server # top
ucs-fi-6248up-0-A#
Great Success! The Xeon E5-2609 is a 4 core CPU. And there are 4x 4GiByte memory DIMMs installed. So inventory looks good. Huzzah! (And what an utter pain.)
Server and Infrastructure issues
So, we just saw the first one of these above. The B200 M3 blades do not like Xeon E5-26xx v2 processors. At least with the firmware installed at present. Let's figure that one out first. https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/blade-servers/B200M3.html#reference_CD3077C1064743F99C0F9A935778BC2F has a little table listing minimum versions for various components to get the Xeon E5-2600 v2 CPUs to run. These are the required versions for E5-26xx V2 CPUs:
| Software or Firmware | Minimum Version |
|---|---|
| Server CIMC | 2.1(3) |
| Server BIOS | 2.1(3) |
| Cisco UCS Manager | 2.1(3) |
| Board controller firmware | 8.0 |
Let's see if we can see what we are running in one of the blades I have and the UCS Manager itself:
ucs-fi-6248up-0-A# scope chassis 1
ucs-fi-6248up-0-A /chassis # scope server 1/1
ucs-fi-6248up-0-A /chassis/server # show firmware cimc
Server Running-Vers Package-Vers Update-Status Activate-Status
------- --------------- --------------- --------------- ---------------
1/1 3.1(23c) Ready Ready
ucs-fi-6248up-0-A /chassis/server # show firmware bios
Server 1/1:
BIOS:
Running-Vers: B200M3.2.2.6d.0.062220160055
Package-Vers:
Update-Status: Ready
Activate-Status: Ready
ucs-fi-6248up-0-A /chassis/server # show firmware boardcontroller
Management Controller:
Server Running-Vers Package-Vers Activate-Status
------- --------------- --------------- ---------------
1/1 15.0 Ready
ucs-fi-6248up-0-A /chassis/server # top
ucs-fi-6248up-0-A# show version brief
System version: 2.2(8m)
Package-Vers: 2.2(8m)A
ucs-fi-6248up-0-A#
So that is odd. The requirements listed in Cisco's documentation for supporting Xeon E5-26xx v2 CPUs are met. But the blade complains about "Board Programmable version not valid for Processor Type". Maybe the blade server's CIMC could tell us more?
CIMC networking
This is going to get us started with attaching the Fabric Interconnect(s) to the rest of the network. Hopefully things go mostly OK. The network at large is all Brocade and Mellanox gear. But here's the strategy statement:
- Create a Link Aggregation Group interface on the appropriate top-of-rack Brocade ICX switches. Pipe all VLANs we care about (management and generic server, 10 and 1000) to that LAG.
- Configure the Fabric Interconnect with some uplink ports to connect to the Brocade LAG interfaces.
- Configure some VLANs on the FI uplink ports ???
- Plumb the management VLAN to the blade servers' CIMC network.
Brocade config
Fairly simple, really. Add this to the switch's config (from elevated privileges prompt (enable mode) do configure terminal and save (write memory) when done.
lag rack-2-to-ucs-fi-6248up-0 dynamic id 3 ports ethernet 1/3/2 ethernet 2/3/2 primary-port 1/3/2 deploy
And to validate:
SSH@brocade-icx6610-48p-rack-2#show lag rack-2-to-ucs-fi-6248up-0
Total number of LAGs: 3
Total number of deployed LAGs: 3
Total number of trunks created:3 (117 available)
LACP System Priority / ID: 1 / 748e.f8dd.6228
LACP Long timeout: 120, default: 120
LACP Short timeout: 3, default: 3
=== LAG "rack-2-to-ucs-fi-6248up-0" ID 3 (dynamic Deployed) ===
LAG Configuration:
Ports: e 1/3/2 e 2/3/2
Port Count: 2
Primary Port: 1/3/2
Trunk Type: hash-based
LACP Key: 20003
Deployment: HW Trunk ID 3
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/3/2 Down None None None 3 Yes N/A 0 748e.f8dd.6264
2/3/2 Down None None None 3 Yes N/A 0 748e.f8dd.6264
Port [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/3/2 1 1 20003 Yes S Agg Syn No No Def No Dwn
2/3/2 1 1 20003 Yes S Agg Syn No No Def No Dwn
Partner Info and PDU Statistics
Port Partner Partner LACP LACP
System ID Key Rx Count Tx Count
1/3/2 1-0000.0000.0000 129 0 0
2/3/2 1-0000.0000.0000 385 0 0
SSH@brocade-icx6610-48p-rack-2#
So that was not terrible.
UCS Fabric Interconnect uplink ports
Need to make some. Kinda like this:
ucs-fi-6248up-0-A# scope eth-uplink
ucs-fi-6248up-0-A /eth-uplink # show fabric a
Fabric:
Id
--
A
ucs-fi-6248up-0-A /eth-uplink # show fabric a detail
Fabric:
Id: A
Current Task 1:
Current Task 2:
ucs-fi-6248up-0-A /eth-uplink # show fabric a expand
Fabric:
Id: A
ucs-fi-6248up-0-A /eth-uplink # scope fabric a
ucs-fi-6248up-0-A /eth-uplink/fabric # create interface 1 17
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # enable
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # commit-buffer
ucs-fi-6248up-0-A /eth-uplink/fabric/interface # up
ucs-fi-6248up-0-A /eth-uplink/fabric # create interface 1 19
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # enable
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # commit-buffer
ucs-fi-6248up-0-A /eth-uplink/fabric/interface # exit
ucs-fi-6248up-0-A /eth-uplink/fabric # show interface
Interface:
Slot Id Port Id Admin State Oper State Lic State Grace Period State Reason Ethernet Link Profile name Oper Ethernet Link Profile name
---------- ---------- ----------- ---------------- -------------------- --------------- ------------ -------------------------- -------------------------------
1 17 Enabled Up License Ok 0 default fabric/lan/eth-link-prof-default
1 19 Enabled Up License Ok 0 default fabric/lan/eth-link-prof-default
ucs-fi-6248up-0-A /eth-uplink/fabric #
Another UCS Manager (and Fabric Interconnect) update
I have acquired a pair of B200 M4 blades. And the FI (and UCS Manager) need upgrading to support them. Hopefully nothing breaks in the process. Hopefully...
Now that we are on 2.2(8m), the SSH client is able to fetch the .bin file using scp. Or maybe I just got the syntax correct this time. So here we are:
ucs-fi-6248up-0-A# show version
System version: 2.2(8m)
ucs-fi-6248up-0-A# top
ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # download image scp://adj@172.17.0.18/home/adj/Downloads/ucs-k9-bundle-infra.2.5.2a.A.bin
Password:
ucs-fi-6248up-0-A /firmware # show download-task
Download task:
File Name Protocol Server Userid State
--------- -------- --------------- --------------- -----
ucs-k9-bundle-infra.2.2.8m.A.bin
Tftp 172.17.0.18 Downloaded
ucs-k9-bundle-infra.2.5.2a.A.bin
Scp 172.17.0.18 adj Downloading
ucs-fi-6248up-0-A /firmware # show download-task
Download task:
File Name Protocol Server Userid State
--------- -------- --------------- --------------- -----
ucs-k9-bundle-infra.2.2.8m.A.bin
Tftp 172.17.0.18 Downloaded
ucs-k9-bundle-infra.2.5.2a.A.bin
Scp 172.17.0.18 adj Downloading
ucs-fi-6248up-0-A /firmware # show download-task
Download task:
File Name Protocol Server Userid State
--------- -------- --------------- --------------- -----
ucs-k9-bundle-infra.2.2.8m.A.bin
Tftp 172.17.0.18 Downloaded
ucs-k9-bundle-infra.2.5.2a.A.bin
Scp 172.17.0.18 adj Downloaded
ucs-fi-6248up-0-A /firmware # scope auto-install
ucs-fi-6248up-0-A /firmware/auto-install # install infra infra-vers 2.5(2a)A
This operation upgrades firmware on UCS Infrastructure Components
(UCS manager, Fabric Interconnects and IOMs).
Here is the checklist of things that are recommended before starting Auto-Install
(1) Review current critical/major faults
(2) Initiate a configuration backup
(3) Check if Management Interface Monitoring Policy is enabled
(4) Check if there is a pending Fabric Interconnect Reboot activitiy
(5) Ensure NTP is configured
(6) Check if any hardware (fabric interconnects, io-modules, servers or adapters) is unsupported in the target release
Do you want to proceed? (yes/no):yes
Triggering Install-Infra with:
Infrastructure Pack Version: 2.5(2a)A
ucs-fi-6248up-0-A /firmware/auto-install # acknowledge primary fabric-interconnect reboot
ucs-fi-6248up-0-A /firmware/auto-install* # commit-buffer
ucs-fi-6248up-0-A /firmware/auto-install # show fsm status expand
FSM Status:
Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: Success
Completion Time: 2009-01-01T01:59:13.146
Progress (%): 100
FSM Stage:
Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployDownloadImages Skip 0
6 DeployCopyAllImagesToPeer Skip 0
7 DeployInternalBackup Success 0
8 DeployPollInternalBackup Success 2
9 DeployActivateUCSM Skip 0
10 DeployPollActivateOfUCSM Success 0
11 DeployUpdateIOM Success 0
12 DeployPollUpdateOfIOM Success 0
13 DeployActivateIOM Success 0
14 DeployPollActivateOfIOM Success 0
15 DeployActivateRemoteFI Skip 0
16 DeployPollActivateOfRemoteFI Skip 0
17 DeployWaitForUserAck Skip 0
18 DeployPollWaitForUserAck Success 0
19 DeployActivateLocalFI Success 0
20 DeployPollActivateOfLocalFI Success 2
ucs-fi-6248up-0-A /firmware/auto-install # show fsm status expand | no-more
FSM Status:
Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: In Progress
Completion Time:
Progress (%): 95
FSM Stage:
Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployDownloadImages Skip 0
6 DeployCopyAllImagesToPeer Skip 0
7 DeployInternalBackup Success 0
8 DeployPollInternalBackup In Progress 1
9 DeployActivateUCSM Pending 0
10 DeployPollActivateOfUCSM Pending 0
11 DeployUpdateIOM Pending 0
12 DeployPollUpdateOfIOM Pending 0
13 DeployActivateIOM Pending 0
14 DeployPollActivateOfIOM Pending 0
15 DeployActivateRemoteFI Pending 0
16 DeployPollActivateOfRemoteFI Pending 0
17 DeployWaitForUserAck Pending 0
18 DeployPollWaitForUserAck Pending 0
19 DeployActivateLocalFI Pending 0
20 DeployPollActivateOfLocalFI Pending 0
ucs-fi-6248up-0-A /firmware/auto-install # show fsm status expand | no-more
FSM Status:
Affected Object: sys/fw-system/fsm
Current FSM: Deploy
Status: In Progress
Completion Time:
Progress (%): 96
FSM Stage:
Order Stage Name Status Try
------ ---------------------------------------- ------------ ---
1 DeployWaitForDeploy Success 0
2 DeployResolveDistributableNames Skip 0
3 DeployResolveDistributable Skip 0
4 DeployResolveImages Skip 0
5 DeployDownloadImages Skip 0
6 DeployCopyAllImagesToPeer Skip 0
7 DeployInternalBackup Success 0
8 DeployPollInternalBackup Success 2
9 DeployActivateUCSM Success 0
10 DeployPollActivateOfUCSM In Progress 1
11 DeployUpdateIOM Pending 0
12 DeployPollUpdateOfIOM Pending 0
13 DeployActivateIOM Pending 0
14 DeployPollActivateOfIOM Pending 0
15 DeployActivateRemoteFI Pending 0
16 DeployPollActivateOfRemoteFI Pending 0
17 DeployWaitForUserAck Pending 0
18 DeployPollWaitForUserAck Pending 0
19 DeployActivateLocalFI Pending 0
20 DeployPollActivateOfLocalFI Pending 0
ucs-fi-6248up-0-A /firmware/auto-install # Connection to 172.16.10.176 closed.
$ ssh -o HostKeyAlgorithms=+ssh-rsa admin@172.16.10.176
Cisco UCS 6200 Series Fabric Interconnect
(admin@172.16.10.176) Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
ucs-fi-6248up-0-A# show version
System version: 2.5(2a)
ucs-fi-6248up-0-A# show version brief
System version: 2.5(2a)
Package-Vers: 2.5(2a)A
ucs-fi-6248up-0-A#
And after some more time and one more disconnect, we see that another OpenSSH client compatibility option is required to connect:
$ ssh -o HostKeyAlgorithms=+ssh-rsa admin@172.16.10.176 Unable to negotiate with 172.16.10.176 port 22: no matching cipher found. Their offer: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se $ ssh -o HostKeyAlgorithms=+ssh-rsa -o Ciphers=+aes256-cbc admin@172.16.10.176 Cisco UCS 6200 Series Fabric Interconnect (admin@172.16.10.176) Password: Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2009, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php ucs-fi-6248up-0-A# show version System version: 2.5(2a) ucs-fi-6248up-0-A# show version brief System version: 2.5(2a) Package-Vers: 2.5(2a)A ucs-fi-6248up-0-A#
But I'm still trying to get the blades' CIMCs connected!
The struggle is real! But I did find https://www.cisco.com/c/en/us/support/docs/servers-unified-computing/integrated-management-controller/118367-configure-cimc-00.pdf which is a GUI-specific outline of the process involved in getting UCS servers' (blade and maybe also rackmount) CIMCs attached to the greater network environment.
In-band? Out-of-band? Huh?
First decision to be made here. In UCS versions before 2.2, the servers' CIMCs always had Ethernet connectivity through the Fabric Interconnects' management port. This connection amounts to a single point of failure for CIMC connectivity.
In UCS version 2.2 and newer, there are two options for CIMC connectivity: In-band using the FIs' high bandwidth network uplink ports, and out-of band using an FI's 1000baseT management port. Naturally, being me, I will opt for the more complicated alternative. And probably complain the whole way through.
In-band CIMC connectivity setup
Log in to the FI. Go to the organization (probably /. And create an IP pool. We are skipping the IPv6 stuff for now and the management VLAN is IPv4-only.
ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # show ip-pool
IP Pool:
Name Size Assigned Management mode
-------------------- ---------- ---------- ---------------
ext-mgmt 0 0 Internal
iscsi-initiator-pool 0 0 Internal
ucs-fi-6248up-0-A /org # create ip-pool CIMC_inband_pool
ucs-fi-6248up-0-A /org/ip-pool* # create block 172.16.10.168 172.16.10.175 172.16.10.1 255.255.255.0
ucs-fi-6248up-0-A /org/ip-pool/block* # set primary-dns 172.16.10.1 secondary-dns 172.16.10.2
ucs-fi-6248up-0-A /org/ip-pool/block* # commit-buffer
ucs-fi-6248up-0-A /org/ip-pool/block # show detail
Block of IP Addresses:
From: 172.16.10.168
To: 172.16.10.175
Default Gateway: 172.16.10.1
Subnet Mask: 255.255.255.0
Primary DNS: 172.16.10.1
Secondary DNS: 172.16.10.2
ucs-fi-6248up-0-A /org/ip-pool/block # show expand
Block of IP Addresses:
From To Default Gateway Subnet Mask
--------------- --------------- --------------- -----------
172.16.10.168 172.16.10.175 172.16.10.1 255.255.255.0
ucs-fi-6248up-0-A /org/ip-pool/block # up
ucs-fi-6248up-0-A /org/ip-pool # show
IP Pool:
Name Size Assigned Management mode
-------------------- ---------- ---------- ---------------
CIMC_inband_pool 8 0 Internal
ucs-fi-6248up-0-A /org/ip-pool # show detail
IP Pool:
Name: CIMC_inband_pool
Size: 8
Assigned: 0
IPv4 Size: 8
IPv4 Assigned: 0
IPv6 Size: 0
IPv6 Assigned: 0
Descr:
Assignment Order: Default
Management mode: Internal
Guid: 00000000-0000-0000-0000-000000000000
Net bios enabled or disabled: Not Active
DHCP enaled or disabled: Not Supported
ucs-fi-6248up-0-A /org/ip-pool # show expand
IP Pool:
Name: CIMC_inband_pool
Size: 8
Assigned: 0
Management mode: Internal
Block of IP Addresses:
From To Default Gateway Subnet Mask
--------------- --------------- --------------- -----------
172.16.10.168 172.16.10.175 172.16.10.1 255.255.255.0
Pooled:
Id Subnet Assigned Assigned To
--------------- --------------- -------- -----------
172.16.10.168 255.255.255.0 No
172.16.10.169 255.255.255.0 No
172.16.10.170 255.255.255.0 No
172.16.10.171 255.255.255.0 No
172.16.10.172 255.255.255.0 No
172.16.10.173 255.255.255.0 No
172.16.10.174 255.255.255.0 No
172.16.10.175 255.255.255.0 No
ucs-fi-6248up-0-A /org/ip-pool #
So we now have a pool of IPv4 addresses to hand out to CIMCs in our UCS domain. And we can also LOL at Cisco's DHCP enaled or disabled spelling error. But not too loud. It may well be fixed in newer versions.
Create VLAN and VLAN group to connect the CIMC IP pool to
VLAN on the Ethernet uplinks:
ucs-fi-6248up-0-A /org/ip-pool # top ucs-fi-6248up-0-A# scope eth-uplink ucs-fi-6248up-0-A /eth-uplink # create vlan Management 10 ucs-fi-6248up-0-A /eth-uplink/vlan* # set native no ucs-fi-6248up-0-A /eth-uplink/vlan* # set vlan-id 10 ucs-fi-6248up-0-A /eth-uplink/vlan* # commit-buffer
VLAN group now:
ucs-fi-6248up-0-A /eth-uplink/vlan # up ucs-fi-6248up-0-A /eth-uplink # show vlan-group ucs-fi-6248up-0-A /eth-uplink # create vlan-group Management_group ucs-fi-6248up-0-A /eth-uplink/vlan-group* # create member-vlan Management ucs-fi-6248up-0-A /eth-uplink/vlan-group/member-vlan* # commit-buffer
Add IP pool, VLAN, VLAN group to in-band profile:
ucs-fi-6248up-0-A /eth-uplink/vlan-group/member-vlan # top ucs-fi-6248up-0-A# scope eth-uplink ucs-fi-6248up-0-A /eth-uplink # scope inband-profile ucs-fi-6248up-0-A /eth-uplink/inband-profile # set default-pool-name CIMC_inband_pool ucs-fi-6248up-0-A /eth-uplink/inband-profile* # set default-vlan-name Management ucs-fi-6248up-0-A /eth-uplink/inband-profile* # set net-group-name Management_group ucs-fi-6248up-0-A /eth-uplink/inband-profile* # commit-buffer ucs-fi-6248up-0-A /eth-uplink/inband-profile #
Now that the in-band profile is set, update a blade's service profile to reference it:
ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # show service-profile
Service Profile:
Service Profile Name Type Server Assignment Association
-------------------- ----------------- ------- ---------- -----------
UCS_B200_M3_FCH162871NA
Instance 1/1 Assigned Associating
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # set ext-mgmt-ip-pool-name CIMC_inband_pool
ucs-fi-6248up-0-A /org/service-profile* # set ext-mgmt-ip-state pooled
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A# top
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # reset hard-reset-immediate
ucs-fi-6248up-0-A /chassis/server* # commit-buffer
ucs-fi-6248up-0-A /chassis/server # reset-kvm
ucs-fi-6248up-0-A /chassis/server* # commit-buffer
ucs-fi-6248up-0-A /chassis/server* # reset-ipmi
ucs-fi-6248up-0-A /chassis/server* # commit-buffer
ucs-fi-6248up-0-A /chassis/server # scope cimc
ucs-fi-6248up-0-A /chassis/server/cimc # show
CIMC:
PID Serial (SN) HW Revision
---------------- ---------------- -----------
UCSB-B200-M3 FCH162871NA 0
ucs-fi-6248up-0-A /chassis/server/cimc # show detail
CIMC:
Product Name: Cisco UCS B200 M3
PID: UCSB-B200-M3
VID: V01
Vendor: Cisco Systems Inc
Serial (SN): FCH162871NA
HW Revision: 0
GUID:
Current Task:
ucs-fi-6248up-0-A /chassis/server/cimc # show expand
CIMC:
PID: UCSB-B200-M3
Serial (SN): FCH162871NA
HW Revision: 0
Vmedia Mapping List:
Full Name: sys/chassis-1/blade-1/mgmt/actual-mount-list
Vmedia Mapping:
Vdisk Id Mapping Name Device Type Mount Protocol Mount Status
-------- ------------ ----------- -------------- ------------
1 Cdd Unknown Not Mounted
2 Hdd Unknown Not Mounted
External Management Interface:
Mode: In Band
Ip V4 State: Pooled
Ip V6 State: Pooled
External Management Virtual LAN:
Network Name: Management
Id: 10
External Management Pooled IP:
Name IP Address Default Gateway Subnet Primary DNS IP Secondary DNS IP
---------- --------------- --------------- --------------- --------------- ----------------
CIMC_inband_pool
172.16.10.169 172.16.10.1 255.255.255.0 172.16.10.1 172.16.10.2
External Management Pooled IPv6:
Name IP Address Default Gateway Prefix Primary DNS IP Secondary DNS IP
---------- ---------- --------------- ------ -------------- ----------------
CIMC_inband_pool
:: :: 64 :: ::
Management Interface:
Access Type: Unspecified
IP Address: 0.0.0.0
Netmask: 0.0.0.0
Gateway: 0.0.0.0
MAC Address: 30:F7:0D:BE:89:EA
Fabric ID: A
Peer Port:
Peer Port Id: 1
Peer Slot Id: 1
Peer Chassis Id: 1
Discovery: Absent
Management Endpoint Log Control:
Type ID Capacity
------ --- --------
SEL 0 Available
ucs-fi-6248up-0-A /chassis/server/cimc #
(Some) Success! The blade's CIMC is assigned an IPv4 address. And it is pingable on the local network. SSH is a no-go, though. Maybe not a thing at all on B series blades? And some further fiddling seems to be needed to get IPMI over LAN working. Something like this:
ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # create ipmi-access-profile ipmi_profile
ucs-fi-6248up-0-A /org/ipmi-access-profile* # create ipmi-user admin
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user* # set password
Enter a password:
Confirm the password:
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user* # commit-buffer
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user # set privilege admin
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user* # commit-buffer
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user # up
ucs-fi-6248up-0-A /org/ipmi-access-profile # set ipmi-over-lan enable
ucs-fi-6248up-0-A /org/ipmi-access-profile # commit-buffer
ucs-fi-6248up-0-A /org/ipmi-access-profile # up
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # set ipmi-access-profile ipmi_profile
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile # top
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # scope cimc
ucs-fi-6248up-0-A /chassis/server/cimc # reset
ucs-fi-6248up-0-A /chassis/server/cimc* # commit-buffer
ucs-fi-6248up-0-A /chassis/server/cimc # top
ucs-fi-6248up-0-A# connect cimc 1/1
Trying 127.5.1.1...
Connected to 127.5.1.1.
Escape character is '^]'.
CIMC Debug Firmware Utility Shell [ support ]
[ help ]# help
__________________________________________
Debug Firmware Utility
__________________________________________
Command List
__________________________________________
alarms
cores
exit
i2cstats
images
mctools
memory
messages
mrcout
network
obfl
post
power
programmables
sensors
sel
fru
tasks
top
update
users
version
mezz1fru
mezz2fru
sldp
help [COMMAND]
__________________________________________
Notes:
"enter Key" will execute last command
"COMMAND ?" will execute help for that command
__________________________________________
[ help ]# users
0. [ "admin" ] [ "" ] [ 2 ]
1. [ Empty Entry ]
2. [ Empty Entry ]
3. [ Empty Entry ]
4. [ Empty Entry ]
5. [ Empty Entry ]
6. [ Empty Entry ]
7. [ Empty Entry ]
8. [ Empty Entry ]
9. [ Empty Entry ]
10. [ Empty Entry ]
11. [ Empty Entry ]
12. [ Empty Entry ]
13. [ Empty Entry ]
14. [ Empty Entry ]
15. [ Empty Entry ]
[ users ]# exit
Connection closed by foreign host.
ucs-fi-6248up-0-A#
So that looks like we should have IPMI over LAN. Let's poke at it a bit and see what happens:
$ ping -c 5 172.16.10.169 PING 172.16.10.169 (172.16.10.169) 56(84) bytes of data. 64 bytes from 172.16.10.169: icmp_seq=1 ttl=63 time=0.930 ms 64 bytes from 172.16.10.169: icmp_seq=2 ttl=63 time=0.655 ms 64 bytes from 172.16.10.169: icmp_seq=3 ttl=63 time=0.677 ms 64 bytes from 172.16.10.169: icmp_seq=4 ttl=63 time=0.658 ms 64 bytes from 172.16.10.169: icmp_seq=5 ttl=63 time=0.732 ms --- 172.16.10.169 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4058ms rtt min/avg/max/mdev = 0.655/0.730/0.930/0.103 ms $ ipmitool -I lanplus -H 172.16.10.169 -U admin chassis status Password: Error: Unable to establish IPMI v2 / RMCP+ session $
And that is a promising start with a disappointing finish. Just for grins, let's try from a machine on the same subnet. Maybe ICMP echo replies get routed but the IPMI over LAN traffic doesn't?
router$ ping -c 5 172.16.10.169
PING 172.16.10.169 (172.16.10.169) 56(84) bytes of data.
64 bytes from 172.16.10.169: icmp_seq=1 ttl=64 time=1.85 ms
64 bytes from 172.16.10.169: icmp_seq=2 ttl=64 time=0.498 ms
64 bytes from 172.16.10.169: icmp_seq=3 ttl=64 time=0.463 ms
64 bytes from 172.16.10.169: icmp_seq=4 ttl=64 time=0.452 ms
64 bytes from 172.16.10.169: icmp_seq=5 ttl=64 time=0.460 ms
--- 172.16.10.169 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4067ms
rtt min/avg/max/mdev = 0.452/0.745/1.853/0.554 ms
router$ ipmitool -I lanplus -H 172.16.10.169 -U admin mc info
Password:
Device ID : 32
Device Revision : 0
Firmware Revision : 3.01
IPMI Version : 2.0
Manufacturer ID : 5771
Manufacturer Name : Cisco Systems, Inc.
Product ID : 9 (0x0009)
Product Name : Unknown (0x09)
Device Available : yes
Provides Device SDRs : yes
Additional Device Support :
Sensor Device
SDR Repository Device
SEL Device
FRU Inventory Device
IPMB Event Receiver
IPMB Event Generator
Aux Firmware Rev Info :
0x00
0x00
0x2a
0x8b
router$ ipmitool -I lanplus -H 172.16.10.169 -U admin lan print
Password:
Set in Progress : Set Complete
Auth Type Support : NONE MD2 MD5 PASSWORD
Auth Type Enable : Callback :
: User :
: Operator :
: Admin :
: OEM :
IP Address Source : Static Address
IP Address : 0.0.0.0
Subnet Mask : 0.0.0.0
MAC Address : 30:f7:0d:be:89:ea
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x00 Precedence=0x00 TOS=0x10
BMC ARP Control : ARP Responses Disabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 2.0 seconds
Default Gateway IP : 0.0.0.0
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,128
Cipher Suite Priv Max : XXXaaaXXaaaXaaa
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
Bad Password Threshold : Not Available
router$
Yay! Very strange that it can be pinged from a different subnet, but ipmitool cannot talk to it that way. Also very strange that it says it has no IP address assigned. I am just guessing here, but perhaps the code in the FI is doing that for us and I have not found the documentation describing how it works.
Serial console for blades?
I am fine with low bit rate serial stuff. No need for a fancy bit mapped screen to manage my computers. And it saves me having to run crazy Avocent Java applications in the process. So let's see what happens when we try to connect from a machine on the same subnet:
router$ ipmitool -I lanplus -H 172.16.10.169 -U admin sol activate Password: Error activating SOL payload: Invalid data field in request router$
Disappointing. But I think I might have seem something about serial over LAN in the service profiles somewhere. Let's see if we can get it turned on:
ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # show sol-policy
ucs-fi-6248up-0-A /org # create sol-policy sol_pol
ucs-fi-6248up-0-A /org/sol-policy* # enable
ucs-fi-6248up-0-A /org/sol-policy* # set speed 115200
ucs-fi-6248up-0-A /org/sol-policy* # commit-buffer
ucs-fi-6248up-0-A /org/sol-policy # up
ucs-fi-6248up-0-A /org # show service-profile
Service Profile:
Service Profile Name Type Server Assignment Association
-------------------- ----------------- ------- ---------- -----------
UCS_B200_M3_FCH162871NA
Instance 1/1 Assigned Associating
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # set sol-policy sol_pol
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile # top
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # scope cimc
ucs-fi-6248up-0-A /chassis/server/cimc # reset
ucs-fi-6248up-0-A /chassis/server/cimc* # commit-buffer
ucs-fi-6248up-0-A /chassis/server/cimc #
And trying again from a machine on the same subnet:
router$ ipmitool -I lanplus -H 172.16.10.169 -U admin sol activate Password: Error activating SOL payload: Invalid data field in request router$
So still no. Grrrr. Maybe it needs to be powered on first?
router$ ipmitool -I lanplus -H 172.16.10.169 -U admin chassis power status Password: Chassis Power is off router$ ipmitool -I lanplus -H 172.16.10.169 -U admin chassis power on Password: Chassis Power Control: Up/On router$ sleep 30 router$ ipmitool -I lanplus -H 172.16.10.169 -U admin sol activate Password: Error activating SOL payload: Invalid data field in request router$
More grrrr. Manuals suggest I might need a BIOS policy that enables the serial port, too?
ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # create bios-policy serial_is_good
ucs-fi-6248up-0-A /org/bios-policy* # set quiet-boot-config quiet-boot disabled
ucs-fi-6248up-0-A /org/bios-policy* # set serial-port-a-config serial-port-a enabled
ucs-fi-6248up-0-A /org/bios-policy* # set console-redir-config baud-rate 115200
ucs-fi-6248up-0-A /org/bios-policy* # set console-redir-config console-redir serial-port-a
ucs-fi-6248up-0-A /org/bios-policy* # set console-redir-config flow-control none
ucs-fi-6248up-0-A /org/bios-policy* # set console-redir-config legacy-os-redir disabled
ucs-fi-6248up-0-A /org/bios-policy* # set console-redir-config terminal-type vt100-plus
ucs-fi-6248up-0-A /org/bios-policy* # commit-buffer
ucs-fi-6248up-0-A /org/bios-policy # up
ucs-fi-6248up-0-A /org # show service-profile
Service Profile:
Service Profile Name Type Server Assignment Association
-------------------- ----------------- ------- ---------- -----------
UCS_B200_M3_FCH162871NA
Instance 1/1 Assigned Associating
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # set bios-policy serial_is_good
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile # top
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # scope cimc
ucs-fi-6248up-0-A /chassis/server/cimc # reset
ucs-fi-6248up-0-A /chassis/server/cimc* # commit-buffer
ucs-fi-6248up-0-A /chassis/server/cimc # top
ucs-fi-6248up-0-A#
So there is still no working serial console on the blade at this point. https://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-b-series-blade-servers/200105-Utilizing-SoL-logging-for-Serial-Redirec.html says to SSH to the CIMC IP address. Just getting a TCP RST back ("connection refused") when trying that. And the IPMI over LAN serial just does not seem to be supported at all. Maybe the blade needs a firmware update, too?
Applying firmware update to blades
So far (I think) I have only done firmware updates to the fabric interconnect and the blade chassis FEXes. First up, let's see what is available for our B200 M3 blades:
ucs-fi-6248up-0-A# scope firmware ucs-fi-6248up-0-A /firmware # show image | i b200-m3 ucs-b200-m3-bios.B200M3.2.0.4a.0.080920121557.bin ucs-b200-m3-bios.B200M3.2.2.1a.0.111220131105.bin ucs-b200-m3-brdprog.11.0.bin Board Controller 11.0 ucs-b200-m3-k9-cimc.2.0.4b.bin CIMC 2.0(4b) ucs-b200-m3-k9-cimc.2.2.1c.bin CIMC 2.2(1c) ucs-b200-m3-mrsasctlr.20.10.1-0100_4.30.00_NA.bin ucs-b200-m3-mrsasctlr.20.12.1-0160_4.37.00_NA.bin ucs-fi-6248up-0-A /firmware #
Those might be kinda old? Let's see what is installed on the B200 M3 that is in the chassis:
ucs-fi-6248up-0-A /firmware # top
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # show firmware
Server 1/1:
Adapter 1:
Running-Vers: 4.1(3a)
Package-Vers:
Update-Status: Ready
Activate-Status: Ready
Adapter 2:
Running-Vers: 4.1(3a)
Package-Vers:
Update-Status: Ready
Activate-Status: Ready
BIOS:
Running-Vers: B200M3.2.2.6d.0.062220160055
Package-Vers:
Update-Status: Ready
Activate-Status: Ready
RAID Controller 1:
Running-Vers: 20.13.1-0255
Package-Vers:
Activate-Status: Ready
BoardController:
Running-Vers: 15.0
ucs-fi-6248up-0-A /chassis/server #
From this, it would seem that the blade in the chassis is already running a newer firmware bundle than the one running on the fabric interconnect. So, let's find out what we are running on the FI, and see about some more updates to it:
ucs-fi-6248up-0-A /chassis/server # top ucs-fi-6248up-0-A# show version brief System version: 2.5(2a) Package-Vers: 2.5(2a)A ucs-fi-6248up-0-A#
It would seem that 3.0(2f) is the latest in the next-newer release train. So acquire that from Cisco's download site and install on the FI and apply following the process documented above. And there is a hiccup with the SSH in 2.5(2a) not being able to get a host key from the SSH servers running on Debian 12 or Debian 13. Here's an example Debian server log of that:
adj@zarathud:~$ sudo grep ssh.*172.16.10.176 /var/log/auth.log 2026-04-02T17:24:54.156462+00:00 zarathud sshd[3090225]: Unable to negotiate with 172.16.10.176 port 33905: no matching host key type found. Their offer: x509v3-sign-rsa,x509v3-sign-dss,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss [preauth] 2026-04-02T17:41:20.766493+00:00 zarathud sshd[3092657]: Unable to negotiate with 172.16.10.176 port 58959: no matching host key type found. Their offer: x509v3-sign-rsa,x509v3-sign-dss,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss [preauth] 2026-04-02T17:50:44.154486+00:00 zarathud sshd[3094045]: Unable to negotiate with 172.16.10.176 port 45452: no matching host key type found. Their offer: x509v3-sign-rsa,x509v3-sign-dss,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss [preauth] 2026-04-02T17:51:36.645000+00:00 zarathud sudo: adj : TTY=pts/22 ; PWD=/home/adj ; USER=root ; COMMAND=/usr/bin/grep ssh.*172.16.10.176 /var/log/auth.log 2026-04-02T17:53:33.561802+00:00 zarathud sudo: adj : TTY=pts/22 ; PWD=/home/adj ; USER=root ; COMMAND=/usr/bin/grep ssh.*172.16.10.176 /var/log/auth.log adj@zarathud:~$
So, for expediency's sake, we will just do it over TFTP this time. Maybe the SSH client in the 3.1(3l) release will be better able to talk to my SSH servers? That last remains to be seen. But the SSH server is a bit nearer to 2026 standards:
adj@yetanotherthinclient:~$ ssh admin@172.16.10.176 Unable to negotiate with 172.16.10.176 port 22: no matching host key type found. Their offer: ssh-rsa adj@yetanotherthinclient:~$ ssh -o HostKeyAlgorithms=+ssh-rsa admin@172.16.10.176 Cisco UCS 6200 Series Fabric Interconnect (admin@172.16.10.176) Password: Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2009, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php ucs-fi-6248up-0-A# show version System version: 3.1(3l) ucs-fi-6248up-0-A# show version brief System version: 3.1(3l) Package-Vers: 3.1(3l)A Service pack version: 3.1(3)SP0(Default) Running-Modules: Package-Vers: ucs-fi-6248up-0-A#
Still only does RSA host keys. But it does support more recent symmetric encryption schemes than the SSH server in 2.5(2a). But major excitement! The web UI now just needs HTML rendering. And ECMAscript. And CSS. But no more need for Java just to see what is going on with the UCS infrastructure. It does seem to still be needed for the KVM. But this is a major step forward. I very strongly recommend getting to the 3.1 UCS Manager releases.
And now for some server (blade) firmware. Available at the same place as the UCS Manager download, a section called "Related Software" has the server firmware images. Download and copy those to the UCS Manager running on the fabric interconnects, too:
ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # download image scp://adj@172.17.0.18/home/adj/Downloads/ucs-k9-bundle-b-series.3.1.3l.B.bin
ucs-fi-6248up-0-A /firmware # show download-task
Download task:
File Name Protocol Server Userid State
--------- -------- ------------------------------------- --------------- -----
ucs-k9-bundle-b-series.3.1.3l.B.bin
Scp 172.17.0.18 adj Downloading
ucs-k9-bundle-infra.2.2.8m.A.bin
Tftp 172.17.0.18 Downloaded
ucs-k9-bundle-infra.2.5.2a.A.bin
Scp 172.17.0.18 adj Downloaded
ucs-k9-bundle-infra.3.1.3l.A.bin
Tftp 172.17.0.18 Downloaded
ucs-mini-k9-bundle-infra.3.0.2f.A.bin
Tftp 172.17.0.18 Downloaded
ucs-fi-6248up-0-A /firmware # show download-task ucs-k9-bundle-b-series.3.1.3l.B.bin detail
Download task:
File Name: ucs-k9-bundle-b-series.3.1.3l.B.bin
Protocol: Scp
Server: 172.17.0.18
Userid: adj
Path: /home/adj/Downloads
Downloaded Image Size (KB): 517633
State: Downloading
Current Task: unpacking image ucs-k9-bundle-b-series.3.1.3l.B.bin on primary(FSM-STAGE:sam:dme:FirmwareDownloaderDownload:UnpackLocal)
ucs-fi-6248up-0-A /firmware # show download-task ucs-k9-bundle-b-series.3.1.3l.B.bin detail
Download task:
File Name: ucs-k9-bundle-b-series.3.1.3l.B.bin
Protocol: Scp
Server: 172.17.0.18
Userid: adj
Path: /home/adj/Downloads
Downloaded Image Size (KB): 517633
State: Downloaded
Current Task:
ucs-fi-6248up-0-A /firmware #
Note that the UCS Manager SSH client can now talk to a Debian 12 OpenSSH server. Small wins are still wins!
A new step for firmware infrastructure updates
It seems that one must accept that a fabric interconnect reboot has happened before applying a new infrastructure firmware pack. Something like this:
ucs-fi-6248up-0-A# scope monitoring
ucs-fi-6248up-0-A /monitoring # show new-faults
Severity Code Last Transition Time ID Description
--------- -------- ------------------------ -------- -----------
Info F0440 2026-04-02T18:37:14.718 113208 Chassis discovery policy conflict: Link IOM 1/1/2 to peer port A:N/A/1/2 not configured
ucs-fi-6248up-0-A /monitoring # top
ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # scope auto-install
ucs-fi-6248up-0-A /firmware/auto-install # acknowledge primary fabric-interconnect reboot
ucs-fi-6248up-0-A /firmware/auto-install* # commit-buffer
ucs-fi-6248up-0-A /firmware/auto-install # install infra infra-vers 3.2(3p)A
This operation upgrades firmware on UCS Infrastructure Components
(UCS manager, Fabric Interconnects and IOMs).
Here is the checklist of things that are recommended before starting Auto-Install
(1) Review current critical/major faults
(2) Initiate a configuration backup
(3) Check if Management Interface Monitoring Policy is enabled
(4) Check if there is a pending Fabric Interconnect Reboot activitiy
(5) Ensure NTP is configured
(6) Check if any hardware (fabric interconnects, io-modules, servers or adapters) is unsupported in the target release
(7) Some fabric-interconnect service-pack install/uninstall will do additional FI reboots in order to complete install/uninstall
To ensure before triggering auto-install, check manual service-pack activation/remove where warning is given if reboots are required
Do you want to proceed? (yes/no):yes
Triggering Install-Infra with:
Infrastructure Pack Version: 3.2(3p)A
Warning: Any Service Pack installed in the system will be removed
ucs-fi-6248up-0-A /firmware/auto-install # acknowledge primary fabric-interconnect reboot
Warning: Check the outstanding faults (scope monitoring <enter> show new-faults) since last FI reboot.
Please make sure the data paths are recovered before proceeding with this FI reboot to ensure there is no interruption to the data traffic.
ucs-fi-6248up-0-A /firmware/auto-install* # commit-buffer
ucs-fi-6248up-0-A /firmware/auto-install #
Broadcast message from root (Thu Apr 2 23:36:05 2026):
The system is going down for reboot NOW!
Connection to 172.16.10.176 closed by remote host.
Connection to 172.16.10.176 closed.