UCS Manager

There are a few pieces of Cisco UCS equipment in the data center:

• One UCS 5108 blade enclosure, accommodating up to 8 B200 server blades of various generations
• 2 UCS-FI-6248UP fabric interconnects, which sit between UCS servers or blade chassis and the rest of the network
• 1 UCS C220 M5 server

Cisco UCS Manager gets installed on the Fabric Interconnects and does Cisco proprietary magic things. Like powering on a single blade. This is way more complicated that it needs to be. So this document is being written to capture knowledge of installation, upgrades, and server management tasks.

How to get

Much (all?) of Cisco's UCS software is available directly without a support contract. But a customer account is required to download. Registration is free as in beer. I have had one since July 2021 and have not been troubled with Cisco spam or any sort of "are you a real person at a real company?" validation.

As this is written, UCS Infrastructure and UCS Manager Software is the place to get it. As I have some older, end-of-life, end-of-support fabric interconnects (62xx series) which are not supported on the current 6.0 release train, I have instead selected the 4.2(3p) version for download. This comes as a 1Gbyte-ish ucs-k9-bundle-infra.4.2.3p.A.bin file.

Untested Fabric Interconnect, what do we do?

5ish years ago (summer 2021), I acquired three of these 6248UP FIs. And I think I got one of them configured for use on the management network and updated to then-current software. I'm not sure which of the three that was. And I am sure I did no documentation at the time. So here we are with new notes.

Console and power are at the front?!?!

So, this is weird. Console and power connections are at the front side of the chassis. All of the connectivity is at the rear as it should be. There is sufficient room on the back for a couple of C14 power connectors and 2 more 8P8C modular connectors for management LAN and serial console. But they are at the front. So cabling is kinda sorta extra fun. sigh

Console serial cable is the usual Cisco modular pinout. Signal on pins 3 and 6. 9600 bps, 8-n-1 are the serial port settings. Nothing unusual there.

Mystery solved regarding the connections at front of the chassis. My FIs have 32 ports for traffic, but there is (well, was, back when this was a current product) a 16 port expansion module available. See https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-6200-series-fabric-interconnects/data_sheet_c78-675245.html for some diagrams.

Basic Fabric Interconnect configuration

I might have lucked out on the first one I received, but it booted normally and started the interactive system setup dialog. It is all pretty basic. Set a password for the admin user and an IP address, netmask, IPv4 gateway address, and DNS server address for the management Ethernet interface (again, on the front). That's pretty much it.

After this is done, make sure the management Ethernet interface is connected to a switch on the correct VLAN and subnet. Then do all the things over an SSH connection. At least for version 2.1, the SSH server only knows about RSA host keys, so a modern OpenSSH client will need something like -o HostKeyAlgorithms=+ssh-rsa listed as a command line option.

Clock discipline is good

To set the time in the FI. Or just get it close. NTP will come. Eventually.

ucs-fi-6248up-0-A# scope system
ucs-fi-6248up-0-A /system # scope services
ucs-fi-6248up-0-A /system/services # set clock Mar 22 2026 02 53 35
Sun Mar 22 02:53:35 UTC 2026
ucs-fi-6248up-0-A /system/services #

And now for the NTP:

ucs-fi-6248up-0-A# scope system
ucs-fi-6248up-0-A /system # scope services
ucs-fi-6248up-0-A /system/services # create ntp-server 172.16.10.2
ucs-fi-6248up-0-A /system/services* # commit-buffer
ucs-fi-6248up-0-A /system/services # create ntp-server 172.16.10.3
ucs-fi-6248up-0-A /system/services* # commit-buffer
ucs-fi-6248up-0-A /system/services # show ntp
NTP Servers:
    Name: 172.16.10.2
    Name: 172.16.10.3
ucs-fi-6248up-0-A /system/services #

Software upgrade time!

Again, on the first one I have tried, let's log in as admin and see what's running:

Cisco UCS 6200 Series Fabric Interconnect

ucs-fi-6248up-0-A login: admin
Password: 
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

ucs-fi-6248up-0-A# show version
System version: 2.2(1c)
ucs-fi-6248up-0-A#

And, more verbosely (for reasons that are beyond me, "brief" mode shows more output?):

ucs-fi-6248up-0-A# show version brief
System version: 2.2(1c)
Package-Vers: 2.2(1c)A
ucs-fi-6248up-0-A# 

So that's really very super dooper old. I downloaded a copy of the 4.1(3i) code in August 2022. Off do so some reading to see if I can skip all the in-between versions.

Did not find anything authoritative in the release notes (but I only looked very briefly.) So, we can do it stepwise. Hopefully. Process looks something like this:

ucs-fi-6248up-0-A# scope firmware 
ucs-fi-6248up-0-A /firmware # download image tftp://172.17.0.18/ucs-k9-bundle-infra.2.2.8m.A.bin
ucs-fi-6248up-0-A /firmware # show download-task

Download task:
    File Name Protocol Server          Userid          State
    --------- -------- --------------- --------------- -----
    ucs-k9-bundle-infra.2.2.8m.A.bin
              Tftp     172.17.0.18                     Downloaded
ucs-fi-6248up-0-A /firmware # 

Yes, you read that right. I delivered the software image to the FI over TFTP. SSH (scp and sftp) are supported protocols, but could not connect to an OpenSSH server on Debian 13. This might be better with newer FI firmware, but for now, I already have a TFTP server handy for netbooting other machines in the environment.

And from here, the firmware bundle can actually be installed on the Fabric Interconnect like so:

ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # scope auto-install
ucs-fi-6248up-0-A /firmware/auto-install # install infra infra-vers 2.2(8m)A
This operation upgrades firmware on UCS Infrastructure Components
(UCS manager, Fabric Interconnects and IOMs). Do you want to proceed?
 (yes/no):yes

Triggering Install-Infra with:
   Infrastructure Pack Version: 2.2(8m)A
ucs-fi-6248up-0-A /firmware/auto-install # acknowledge primary fabric-interconnect reboot
ucs-fi-6248up-0-A /firmware/auto-install* # commit-buffer
ucs-fi-6248up-0-A /firmware/auto-install # show fsm status expand


    FSM Status:

        Affected Object: sys/fw-system/fsm
        Current FSM: Deploy
        Status: In Progress
        Completion Time:
        Progress (%): 87

        FSM Stage:

        Order  Stage Name                               Status       Try
        ------ ---------------------------------------- ------------ ---
        1      DeployWaitForDeploy                      Success      0
        2      DeployResolveDistributableNames          Skip         0
        3      DeployResolveDistributable               Skip         0
        4      DeployResolveImages                      Skip         0
        5      DeployActivateUCSM                       Success      0
        6      DeployPollActivateOfUCSM                 In Progress  1
        7      DeployUpdateIOM                          Pending      0
        8      DeployPollUpdateOfIOM                    Pending      0
        9      DeployActivateIOM                        Pending      0
        10     DeployPollActivateOfIOM                  Pending      0
        11     DeployActivateRemoteFI                   Pending      0
        12     DeployPollActivateOfRemoteFI             Pending      0
        13     DeployWaitForUserAck                     Pending      0
        14     DeployActivateLocalFI                    Pending      0
        15     DeployPollActivateOfLocalFI              Pending      0
ucs-fi-6248up-0-A /firmware/auto-install #  Connection to 172.16.10.176 closed.

Somewhere along the way, the SSH server hung up on me. But that seems like the sort of thing that should happen as the FI's OS is being upgraded. Progress can also be tracked on the serial console port like so:

ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # scope auto-install
ucs-fi-6248up-0-A /firmware/auto-install # show fsm status expand | no-more


    FSM Status:

        Affected Object: sys/fw-system/fsm
        Current FSM: Deploy
        Status: In Progress
        Completion Time:
        Progress (%): 98

        FSM Stage:

        Order  Stage Name                               Status       Try
        ------ ---------------------------------------- ------------ ---
        1      DeployWaitForDeploy                      Success      0
        2      DeployResolveDistributableNames          Skip         0
        3      DeployResolveDistributable               Skip         0
        4      DeployResolveImages                      Skip         0
        5      DeployDownloadImages                     Skip         0
        6      DeployCopyAllImagesToPeer                Skip         0
        7      DeployInternalBackup                     Success      0
        8      DeployPollInternalBackup                 Success      2
        9      DeployActivateUCSM                       Skip         0
        10     DeployPollActivateOfUCSM                 Success      0
        11     DeployUpdateIOM                          Success      0
        12     DeployPollUpdateOfIOM                    Success      0
        13     DeployActivateIOM                        Success      0
        14     DeployPollActivateOfIOM                  Success      0
        15     DeployActivateRemoteFI                   Skip         0
        16     DeployPollActivateOfRemoteFI             Skip         0
        17     DeployWaitForUserAck                     Skip         0
        18     DeployPollWaitForUserAck                 Success      0
        19     DeployActivateLocalFI                    Success      0
        20     DeployPollActivateOfLocalFI              In Progress  2
ucs-fi-6248up-0-A /firmware/auto-install # 

At some point, the FI will reboot itself.

Confirm new code is running like so:

ucs-fi-6248up-0-A# show version
System version: 2.2(8m)
ucs-fi-6248up-0-A# show version brief
System version: 2.2(8m)
Package-Vers: 2.2(8m)A
ucs-fi-6248up-0-A#

STOP! Don't forget the other UCS things!

Um, yeah. So there is this UCS 5108 blade chassis in the data center, too. No idea what its firmware versions are. But they are sure to also need firmware updates. And it would be distressing if those components were left in a state where they are unusable because the FIs are so much newer than the blade enclosure and its servers. So I guess we need to see about warming the ZIP code 5degC by attaching the blade chassis to the mains power and see what we can see.

Correct cabling for single FI to UCS 5108

According to https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/chassis-install-guide/ucs5108_install/ucs5108_install_chapter_010.html, a single Fabric Interconnect can only be connected to a single Fabric Extender on a UCS chassis. So for now, enable 6248UP ports 1 and 3, and cable to ports 1 and 2 on the left side (as seen from the rear) Fabric Extender.

SFPs go into the FI and the UCS 5108 FEXes

Pretty self explanatory, right. Find Cisco 10GbE SR SFP+ modules, insert into port one on each of the FEXes, insert into ports 1 and 3 on the FI. Attach nice aqua colored cable. And don't panic as the lights don't come on.

Telling the FI to discover the blade chassis

This is way more complicated that it needs to be. I'm sure it's totally amazing if you have racks full of Cisco UCS hardware. But I have just the one blade chassis and the one C series server and the 2 fabric interconnects. And the friendly FI management software is all lovely out of date (completely unrunnable in 2026!) Java stuff. So fumbling around the CLI, hoping to find my way out of the darkness here.

The process for this seems to be something like:

• Configure FI Ethernet ports as server ports
• Ensure the attached chassis has been discovered and acknowledge any faults

ucs-fi-6248up-0-A# scope eth-server
ucs-fi-6248up-0-A /eth-server # show fabric a detail

Fabric:
    Id: A
    Current Task:

ucs-fi-6248up-0-A /eth-server # scope fabric a
ucs-fi-6248up-0-A /eth-server/fabric # create interface 1 1
ucs-fi-6248up-0-A /eth-server/fabric/interface* # enable
ucs-fi-6248up-0-A /eth-server/fabric/interface* # show interface detail
ucs-fi-6248up-0-A /eth-server/fabric/interface* # commit-buffer
ucs-fi-6248up-0-A /eth-server/fabric/interface # exit
ucs-fi-6248up-0-A /eth-server/fabric # create interface 1 3
ucs-fi-6248up-0-A /eth-server/fabric/interface* # enable
ucs-fi-6248up-0-A /eth-server/fabric/interface* # commit-buffer
ucs-fi-6248up-0-A /eth-server/fabric/interface # exit
ucs-fi-6248up-0-A /eth-server/fabric # exit
ucs-fi-6248up-0-A /eth-server # show interface

Interface:

Fabric  Slot  Port  Admin State  Oper State   State Reason                    Chassis  Lic State            Grace Prd
------- ----- ----- ------------ ------------ ------------------------------- -------- -------------------- ---------
A           1     1 Enabled      Up                                           1        License Ok                         0
A           1     3 Enabled      Up                                           1        License Ok                         0
ucs-fi-6248up-0-A /eth-server # exit
ucs-fi-6248up-0-A# show chassis inventory
Chassis    PID             Vendor            Serial (SN) HW Revision
---------- --------------- ----------------- ----------- -----------
         1 N20-C6508       Cisco Systems Inc FOX1808GLJG 0
ucs-fi-6248up-0-A#

Success! Serial number reported in the UCS Manager matches the one I found on the label.

So how do I power on a server?

Still fumbling through here. But: servers (blade servers for sure, not sure about rack servers) belong to UCS "Organizations." These organizations are managed by the UCS Manager, too. And since an organization owns a server (blade or otherwise) the organization can power servers on and off.

What orgs exist?

This is what we get from a factory default UCS Manager:

ucs-fi-6248up-0-A# show org

Organizations:
    Name
    ----
    / (root)
ucs-fi-6248up-0-A# show org detail

Organizations:
    Name: / (root)
ucs-fi-6248up-0-A#

Enter an org, see what we have for basic policies

ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # show bladeserver-disc-policy
ucs-fi-6248up-0-A /org # show chassis-conn-policy

Chassis/FEX Connectivity Policy:
    Chassis/FEX Id Chassis/FEX Name Fabric Id Link Aggregation Preference
    -------------- ---------------- --------- ---------------------------
                 1 sys/chassis-1    A         Global
ucs-fi-6248up-0-A /org # show chassis-disc-policy

Chassis/FEX Discovery Policy:
    Description Qualifier  Action            Rebalance         Link Aggregation Pref Multicast Hw Hash
    ----------- ---------- ----------------- ----------------- --------------------- -----------------
                none       1 Link            User Acknowledged None                  Disabled
ucs-fi-6248up-0-A /org # show chassis-disc-policy detail

Chassis/FEX Discovery Policy:
    Description:
    Qualifier: none
    Action: 1 Link
    Rebalance: User Acknowledged
    Link Aggregation Pref: None
    Multicast Hw Hash: Disabled
ucs-fi-6248up-0-A /org # show rackserver-disc-policy

Rack Server Discovery Policy:
    Action            Scrub Policy
    ----------------- ------------
    Immediate
ucs-fi-6248up-0-A /org # show rackserver-disc-policy detail

Rack Server Discovery Policy:
    Action: Immediate
    Scrub Policy:
    Description:
    Current Task:
ucs-fi-6248up-0-A /org # show rackserver-mgmt-policy

Rack Server Management Policy:
    Action
    ------
    Auto Acknowledged
ucs-fi-6248up-0-A /org # show rackserver-mgmt-policy detail

Rack Server Management Policy:
    Action: Auto Acknowledged
ucs-fi-6248up-0-A /org #

Create a profile and add a (blade) server to it

ucs-fi-6248up-0-A# show chassis inventory 1
Chassis    PID             Vendor            Serial (SN) HW Revision
---------- --------------- ----------------- ----------- -----------
         1 N20-C6508       Cisco Systems Inc FOX1808GLJG 0
ucs-fi-6248up-0-A# show chassis inventory 1 expand 
Chassis 1:
    Servers:
        Server 1/1:
            Equipped Product Name: Cisco UCS B200 M3
            Equipped PID: UCSB-B200-M3
            Equipped VID: V03
            Equipped Serial (SN): FCH162871NA
            Slot Status: Equipped
            Acknowledged Product Name: Cisco UCS B200 M3
            Acknowledged PID: UCSB-B200-M3
            Acknowledged VID: V03
            Acknowledged Serial (SN): FCH162871NA
            Acknowledged Memory (MB): 196608
            Acknowledged Effective Memory (MB): 196608
            Acknowledged Cores: 0
            Acknowledged Adapters: 2

        Server 1/2:
            Equipped Product Name: Cisco UCS B200 M3
            Equipped PID: UCSB-B200-M3
            Equipped VID: V06
            Equipped Serial (SN): FCH1811JELC
            Slot Status: Equipped
            Acknowledged Product Name: Cisco UCS B200 M3
        [...]
ucs-fi-6248up-0-A# top
ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # create service-profile
ucs-fi-6248up-0-A /org # create service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile* # associate server 1/1
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile #

I can power on now?

Well, no. Maybe it should have worked. But for this blade, there was a Power On Self Test (POST) failure. Let's dig in. To power on the server included in the service profile, do this:

ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # power up
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile # top
ucs-fi-6248up-0-A #

And let's have a look at its status:

ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # show

Server:
    Slot    Overall Status        Service Profile      Availability
    ------- --------------------- -------------------- ------------
          1 Compute Failed                             Unavailable
ucs-fi-6248up-0-A /chassis/server # show detail

Server:
    Slot: 1
    Name:
    User Label:
    Overall Status: Compute Failed
    Oper Qualifier: Compute Post Failure
    Service Profile:
    Association: None
    Availability: Unavailable
    Discovery: Failed
    Conn Path: A
    Conn Status: A
    Managing Instance: A
    Admin Power: Policy
    Oper Power: Off
    Admin State: In Service
    Product Name: Cisco UCS B200 M3
    PID: UCSB-B200-M3
    VID: V03
    Vendor: Cisco Systems Inc
    Serial (SN): FCH162871NA
    HW Revision: 0
    Mfg Date: 2012-07-26T00:00:00.000
    Part Number: 73-13217-08
    Memory (MB): 196608
    Effective Memory (MB): 196608
    Operating Memory Speed (MHz): 1333
    Operating Memory Voltage: Regular Voltage
    Cores: 0
    Num Of Cores Enabled: 0
    Adapters: 2
    Eth Host Interfaces: 0
    FC Host Interfaces: 0
    Burned-In UUID: 00000000-0000-0000-0000-000000000000
    Dynamic UUID: 00000000-0000-0000-0000-000000000000
    Current Task 1: Checking hardware configuration server 1/1(FSM-STAGE:sam:dme:ComputeBladeDiscover:Sanitize)
    Current Task 2:
    Current Task 3:
ucs-fi-6248up-0-A /chassis/server # show post

POST:
    Global ID Code      Severity  Affected Object                  Description
    --------- --------- --------- -------------------------------- -----------
    6215      POST-6215 Critical  sys/chassis-1/blade-1            Board Programmable version not valid for Processor Type
ucs-fi-6248up-0-A /chassis/server # show post detail

POST:
    Global ID: 6215
    Code: POST-6215
    Local ID: 1536
    Severity: Critical
    Affected Object: sys/chassis-1/blade-1
    Description: Board Programmable version not valid for Processor Type
    Type: server: Cisco Systems Inc UCSB-B200-M3
    Recoverable: Non Recoverable
    Recovery Action: Refer the CPU upgrade guide.
    Timestamp: 2026-03-22T16:17:06.001
ucs-fi-6248up-0-A /chassis/server #

Making an inferences from "Board Programmable version not valid for Processor Type" I am thinking that it does not like the Xeon E5-2650v2 I installed. Now to find a not v2 E5-26xx CPU and see if that works any better. I found a not obviously damaged E5-2609 in storage (er, piles in the garage) and installed that in the blade. Let's see what we have now:

ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # power up
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile # top
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # show inventory
Server 1/1:
    Name:
    User Label:
    Equipped PID: UCSB-B200-M3
    Equipped VID: V03
    Equipped Serial (SN): FCH162871NA
    Slot Status: Equipped
    Acknowledged Product Name: Cisco UCS B200 M3
    Acknowledged PID: UCSB-B200-M3
    Acknowledged VID: V03
    Acknowledged Serial (SN): FCH162871NA
    Acknowledged Memory (MB): 16384
    Acknowledged Effective Memory (MB): 16384
    Acknowledged Cores: 4
    Acknowledged Adapters: 2
ucs-fi-6248up-0-A /chassis/server # top
ucs-fi-6248up-0-A# 

Great Success! The Xeon E5-2609 is a 4 core CPU. And there are 4x 4GiByte memory DIMMs installed. So inventory looks good. Huzzah! (And what an utter pain.)

Server and Infrastructure issues

So, we just saw the first one of these above. The B200 M3 blades do not like Xeon E5-26xx v2 processors. At least with the firmware installed at present. Let's figure that one out first. https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/blade-servers/B200M3.html#reference_CD3077C1064743F99C0F9A935778BC2F has a little table listing minimum versions for various components to get the Xeon E5-2600 v2 CPUs to run. These are the required versions for E5-26xx V2 CPUs:

Software or Firmware	Minimum Version
Server CIMC	2.1(3)
Server BIOS	2.1(3)
Cisco UCS Manager	2.1(3)
Board controller firmware	8.0

Let's see if we can see what we are running in one of the blades I have and the UCS Manager itself:

ucs-fi-6248up-0-A# scope chassis 1
ucs-fi-6248up-0-A /chassis # scope server 1/1
ucs-fi-6248up-0-A /chassis/server # show firmware cimc
Server  Running-Vers    Package-Vers    Update-Status   Activate-Status
------- --------------- --------------- --------------- ---------------
1/1     3.1(23c)                        Ready           Ready
ucs-fi-6248up-0-A /chassis/server # show firmware bios
Server 1/1:
    BIOS:
        Running-Vers: B200M3.2.2.6d.0.062220160055
        Package-Vers:
        Update-Status: Ready
        Activate-Status: Ready


ucs-fi-6248up-0-A /chassis/server # show firmware boardcontroller

Management Controller:

Server  Running-Vers    Package-Vers    Activate-Status
------- --------------- --------------- ---------------
1/1     15.0                            Ready
ucs-fi-6248up-0-A /chassis/server # top
ucs-fi-6248up-0-A# show version brief
System version: 2.2(8m)
Package-Vers: 2.2(8m)A
ucs-fi-6248up-0-A#

So that is odd. The requirements listed in Cisco's documentation for supporting Xeon E5-26xx v2 CPUs are met. But the blade complains about "Board Programmable version not valid for Processor Type". Maybe the blade server's CIMC could tell us more?

CIMC networking

This is going to get us started with attaching the Fabric Interconnect(s) to the rest of the network. Hopefully things go mostly OK. The network at large is all Brocade and Mellanox gear. But here's the strategy statement:

• Create a Link Aggregation Group interface on the appropriate top-of-rack Brocade ICX switches. Pipe all VLANs we care about (management and generic server, 10 and 1000) to that LAG.
• Configure the Fabric Interconnect with some uplink ports to connect to the Brocade LAG interfaces.
• Configure some VLANs on the FI uplink ports ???
• Plumb the management VLAN to the blade servers' CIMC network.

Brocade config

Fairly simple, really. Add this to the switch's config (from elevated privileges prompt (enable mode) do configure terminal and save (write memory) when done.

lag rack-2-to-ucs-fi-6248up-0 dynamic id 3
 ports ethernet 1/3/2 ethernet 2/3/2 
 primary-port 1/3/2
 deploy

And to validate:

SSH@brocade-icx6610-48p-rack-2#show lag rack-2-to-ucs-fi-6248up-0
Total number of LAGs:          3
Total number of deployed LAGs: 3
Total number of trunks created:3 (117 available)
LACP System Priority / ID:     1 / 748e.f8dd.6228
LACP Long timeout:             120, default: 120
LACP Short timeout:            3, default: 3

=== LAG "rack-2-to-ucs-fi-6248up-0" ID 3 (dynamic Deployed) ===
LAG Configuration:
   Ports:         e 1/3/2 e 2/3/2 
   Port Count:    2
   Primary Port:  1/3/2
   Trunk Type:    hash-based
   LACP Key:      20003
Deployment: HW Trunk ID 3
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/3/2      Down    None    None None  3     Yes N/A  0   748e.f8dd.6264                 
2/3/2      Down    None    None None  3     Yes N/A  0   748e.f8dd.6264                 

Port       [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/3/2           1        1   20003   Yes   S   Agg  Syn  No   No   Def  No   Dwn
2/3/2           1        1   20003   Yes   S   Agg  Syn  No   No   Def  No   Dwn

                                                                 
 Partner Info and PDU Statistics 
Port          Partner         Partner     LACP      LACP     
             System ID         Key     Rx Count  Tx Count  
1/3/2    1-0000.0000.0000      129        0         0
2/3/2    1-0000.0000.0000      385        0         0


SSH@brocade-icx6610-48p-rack-2#

So that was not terrible.

UCS Fabric Interconnect uplink ports

Need to make some. Kinda like this:

ucs-fi-6248up-0-A# scope eth-uplink
ucs-fi-6248up-0-A /eth-uplink # show fabric a

Fabric:
    Id
    --
    A
ucs-fi-6248up-0-A /eth-uplink # show fabric a detail

Fabric:
    Id: A
    Current Task 1:
    Current Task 2:
ucs-fi-6248up-0-A /eth-uplink # show fabric a expand

Fabric:
    Id: A
ucs-fi-6248up-0-A /eth-uplink # scope fabric a
ucs-fi-6248up-0-A /eth-uplink/fabric # create interface 1 17
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # enable
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # commit-buffer
ucs-fi-6248up-0-A /eth-uplink/fabric/interface # up
ucs-fi-6248up-0-A /eth-uplink/fabric # create interface 1 19
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # enable
ucs-fi-6248up-0-A /eth-uplink/fabric/interface* # commit-buffer 
ucs-fi-6248up-0-A /eth-uplink/fabric/interface # exit
ucs-fi-6248up-0-A /eth-uplink/fabric # show interface 

Interface:

Slot Id    Port Id    Admin State Oper State       Lic State            Grace Period    State Reason Ethernet Link Profile name Oper Ethernet Link Profile name
---------- ---------- ----------- ---------------- -------------------- --------------- ------------ -------------------------- -------------------------------
1          17         Enabled     Up               License Ok                         0              default                    fabric/lan/eth-link-prof-default
1          19         Enabled     Up               License Ok                         0              default                    fabric/lan/eth-link-prof-default
ucs-fi-6248up-0-A /eth-uplink/fabric #

Another UCS Manager (and Fabric Interconnect) update

I have acquired a pair of B200 M4 blades. And the FI (and UCS Manager) need upgrading to support them. Hopefully nothing breaks in the process. Hopefully...

Now that we are on 2.2(8m), the SSH client is able to fetch the .bin file using scp. Or maybe I just got the syntax correct this time. So here we are:

ucs-fi-6248up-0-A# show version
System version: 2.2(8m)
ucs-fi-6248up-0-A# top
ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # download image scp://adj@172.17.0.18/home/adj/Downloads/ucs-k9-bundle-infra.2.5.2a.A.bin
Password:
ucs-fi-6248up-0-A /firmware # show download-task

Download task:
    File Name Protocol Server          Userid          State
    --------- -------- --------------- --------------- -----
    ucs-k9-bundle-infra.2.2.8m.A.bin
              Tftp     172.17.0.18                     Downloaded
    ucs-k9-bundle-infra.2.5.2a.A.bin
              Scp      172.17.0.18     adj             Downloading
ucs-fi-6248up-0-A /firmware # show download-task

Download task:
    File Name Protocol Server          Userid          State
    --------- -------- --------------- --------------- -----
    ucs-k9-bundle-infra.2.2.8m.A.bin
              Tftp     172.17.0.18                     Downloaded
    ucs-k9-bundle-infra.2.5.2a.A.bin
              Scp      172.17.0.18     adj             Downloading
ucs-fi-6248up-0-A /firmware # show download-task
 
 Download task:
     File Name Protocol Server          Userid          State
     --------- -------- --------------- --------------- -----
     ucs-k9-bundle-infra.2.2.8m.A.bin
               Tftp     172.17.0.18                     Downloaded
     ucs-k9-bundle-infra.2.5.2a.A.bin
               Scp      172.17.0.18     adj             Downloaded
ucs-fi-6248up-0-A /firmware # scope auto-install
ucs-fi-6248up-0-A /firmware/auto-install # install infra infra-vers 2.5(2a)A
This operation upgrades firmware on UCS Infrastructure Components
(UCS manager, Fabric Interconnects and IOMs).
Here is the checklist of things that are recommended before starting Auto-Install
(1) Review current critical/major faults
(2) Initiate a configuration backup
(3) Check if Management Interface Monitoring Policy is enabled
(4) Check if there is a pending Fabric Interconnect Reboot activitiy
(5) Ensure NTP is configured
(6) Check if any hardware (fabric interconnects, io-modules, servers or adapters) is unsupported in the target release
Do you want to proceed? (yes/no):yes

Triggering Install-Infra with:
   Infrastructure Pack Version: 2.5(2a)A

ucs-fi-6248up-0-A /firmware/auto-install # acknowledge primary fabric-interconnect reboot
ucs-fi-6248up-0-A /firmware/auto-install* # commit-buffer
ucs-fi-6248up-0-A /firmware/auto-install # show fsm status expand


    FSM Status:

        Affected Object: sys/fw-system/fsm
        Current FSM: Deploy
        Status: Success
        Completion Time: 2009-01-01T01:59:13.146
        Progress (%): 100

        FSM Stage:

        Order  Stage Name                               Status       Try
        ------ ---------------------------------------- ------------ ---
        1      DeployWaitForDeploy                      Success      0
        2      DeployResolveDistributableNames          Skip         0
        3      DeployResolveDistributable               Skip         0
        4      DeployResolveImages                      Skip         0
        5      DeployDownloadImages                     Skip         0
        6      DeployCopyAllImagesToPeer                Skip         0
        7      DeployInternalBackup                     Success      0
        8      DeployPollInternalBackup                 Success      2
        9      DeployActivateUCSM                       Skip         0
        10     DeployPollActivateOfUCSM                 Success      0
        11     DeployUpdateIOM                          Success      0
        12     DeployPollUpdateOfIOM                    Success      0
        13     DeployActivateIOM                        Success      0
        14     DeployPollActivateOfIOM                  Success      0
        15     DeployActivateRemoteFI                   Skip         0
        16     DeployPollActivateOfRemoteFI             Skip         0
        17     DeployWaitForUserAck                     Skip         0
        18     DeployPollWaitForUserAck                 Success      0
        19     DeployActivateLocalFI                    Success      0
        20     DeployPollActivateOfLocalFI              Success      2
ucs-fi-6248up-0-A /firmware/auto-install # show fsm status expand | no-more


   FSM Status:

        Affected Object: sys/fw-system/fsm
        Current FSM: Deploy
        Status: In Progress
        Completion Time:
        Progress (%): 95

        FSM Stage:

        Order  Stage Name                               Status       Try
        ------ ---------------------------------------- ------------ ---
        1      DeployWaitForDeploy                      Success      0
        2      DeployResolveDistributableNames          Skip         0
        3      DeployResolveDistributable               Skip         0
        4      DeployResolveImages                      Skip         0
        5      DeployDownloadImages                     Skip         0
        6      DeployCopyAllImagesToPeer                Skip         0
        7      DeployInternalBackup                     Success      0
        8      DeployPollInternalBackup                 In Progress  1
        9      DeployActivateUCSM                       Pending      0
        10     DeployPollActivateOfUCSM                 Pending      0
        11     DeployUpdateIOM                          Pending      0
        12     DeployPollUpdateOfIOM                    Pending      0
        13     DeployActivateIOM                        Pending      0
        14     DeployPollActivateOfIOM                  Pending      0
        15     DeployActivateRemoteFI                   Pending      0
        16     DeployPollActivateOfRemoteFI             Pending      0
        17     DeployWaitForUserAck                     Pending      0
        18     DeployPollWaitForUserAck                 Pending      0
        19     DeployActivateLocalFI                    Pending      0
        20     DeployPollActivateOfLocalFI              Pending      0
ucs-fi-6248up-0-A /firmware/auto-install # show fsm status expand | no-more


    FSM Status:

        Affected Object: sys/fw-system/fsm
        Current FSM: Deploy
        Status: In Progress
        Completion Time:
        Progress (%): 96

        FSM Stage:

        Order  Stage Name                               Status       Try
        ------ ---------------------------------------- ------------ ---
        1      DeployWaitForDeploy                      Success      0
        2      DeployResolveDistributableNames          Skip         0
        3      DeployResolveDistributable               Skip         0
        4      DeployResolveImages                      Skip         0
        5      DeployDownloadImages                     Skip         0
        6      DeployCopyAllImagesToPeer                Skip         0
        7      DeployInternalBackup                     Success      0
        8      DeployPollInternalBackup                 Success      2
        9      DeployActivateUCSM                       Success      0
        10     DeployPollActivateOfUCSM                 In Progress  1
        11     DeployUpdateIOM                          Pending      0
        12     DeployPollUpdateOfIOM                    Pending      0
        13     DeployActivateIOM                        Pending      0
        14     DeployPollActivateOfIOM                  Pending      0
        15     DeployActivateRemoteFI                   Pending      0
        16     DeployPollActivateOfRemoteFI             Pending      0
        17     DeployWaitForUserAck                     Pending      0
        18     DeployPollWaitForUserAck                 Pending      0
        19     DeployActivateLocalFI                    Pending      0
        20     DeployPollActivateOfLocalFI              Pending      0
ucs-fi-6248up-0-A /firmware/auto-install # Connection to 172.16.10.176 closed.
$ ssh -o HostKeyAlgorithms=+ssh-rsa admin@172.16.10.176
Cisco UCS 6200 Series Fabric Interconnect
(admin@172.16.10.176) Password: 
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

ucs-fi-6248up-0-A# show version
System version: 2.5(2a)
ucs-fi-6248up-0-A# show version brief
System version: 2.5(2a)
Package-Vers: 2.5(2a)A
ucs-fi-6248up-0-A#

And after some more time and one more disconnect, we see that another OpenSSH client compatibility option is required to connect:

$ ssh -o HostKeyAlgorithms=+ssh-rsa admin@172.16.10.176
Unable to negotiate with 172.16.10.176 port 22: no matching cipher found. Their offer: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se
$ ssh -o HostKeyAlgorithms=+ssh-rsa -o Ciphers=+aes256-cbc admin@172.16.10.176
Cisco UCS 6200 Series Fabric Interconnect
(admin@172.16.10.176) Password: 
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

ucs-fi-6248up-0-A# show version
System version: 2.5(2a)
ucs-fi-6248up-0-A# show version brief
System version: 2.5(2a)
Package-Vers: 2.5(2a)A
ucs-fi-6248up-0-A#

But I'm still trying to get the blades' CIMCs connected!

The struggle is real! But I did find https://www.cisco.com/c/en/us/support/docs/servers-unified-computing/integrated-management-controller/118367-configure-cimc-00.pdf which is a GUI-specific outline of the process involved in getting UCS servers' (blade and maybe also rackmount) CIMCs attached to the greater network environment.

In-band? Out-of-band? Huh?

First decision to be made here. In UCS versions before 2.2, the servers' CIMCs always had Ethernet connectivity through the Fabric Interconnects' management port. This connection amounts to a single point of failure for CIMC connectivity.

In UCS version 2.2 and newer, there are two options for CIMC connectivity: In-band using the FIs' high bandwidth network uplink ports, and out-of band using an FI's 1000baseT management port. Naturally, being me, I will opt for the more complicated alternative. And probably complain the whole way through.

In-band CIMC connectivity setup

Log in to the FI. Go to the organization (probably /. And create an IP pool. We are skipping the IPv6 stuff for now and the management VLAN is IPv4-only.

ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # show ip-pool

IP Pool:
    Name                 Size       Assigned   Management mode
    -------------------- ---------- ---------- ---------------
    ext-mgmt                      0          0 Internal
    iscsi-initiator-pool          0          0 Internal
ucs-fi-6248up-0-A /org # create ip-pool CIMC_inband_pool
ucs-fi-6248up-0-A /org/ip-pool* # create block 172.16.10.168 172.16.10.175 172.16.10.1 255.255.255.0
ucs-fi-6248up-0-A /org/ip-pool/block* # set primary-dns 172.16.10.1 secondary-dns 172.16.10.2
ucs-fi-6248up-0-A /org/ip-pool/block* # commit-buffer
ucs-fi-6248up-0-A /org/ip-pool/block # show detail

Block of IP Addresses:
    From: 172.16.10.168
    To: 172.16.10.175
    Default Gateway: 172.16.10.1
    Subnet Mask: 255.255.255.0
    Primary DNS: 172.16.10.1
    Secondary DNS: 172.16.10.2
ucs-fi-6248up-0-A /org/ip-pool/block # show expand

Block of IP Addresses:
    From            To              Default Gateway Subnet Mask
    --------------- --------------- --------------- -----------
    172.16.10.168   172.16.10.175   172.16.10.1     255.255.255.0
ucs-fi-6248up-0-A /org/ip-pool/block # up
ucs-fi-6248up-0-A /org/ip-pool # show

IP Pool:
    Name                 Size       Assigned   Management mode
    -------------------- ---------- ---------- ---------------
    CIMC_inband_pool              8          0 Internal
ucs-fi-6248up-0-A /org/ip-pool # show detail

IP Pool:
    Name: CIMC_inband_pool
    Size: 8
    Assigned: 0
    IPv4 Size: 8
    IPv4 Assigned: 0
    IPv6 Size: 0
    IPv6 Assigned: 0
    Descr:
    Assignment Order: Default
    Management mode: Internal
    Guid: 00000000-0000-0000-0000-000000000000
    Net bios enabled or disabled: Not Active
    DHCP enaled or disabled: Not Supported
ucs-fi-6248up-0-A /org/ip-pool # show expand

IP Pool:
    Name: CIMC_inband_pool
    Size: 8
    Assigned: 0
    Management mode: Internal

    Block of IP Addresses:
        From            To              Default Gateway Subnet Mask
        --------------- --------------- --------------- -----------
        172.16.10.168   172.16.10.175   172.16.10.1     255.255.255.0

    Pooled:
        Id              Subnet          Assigned Assigned To
        --------------- --------------- -------- -----------
        172.16.10.168   255.255.255.0   No
        172.16.10.169   255.255.255.0   No
        172.16.10.170   255.255.255.0   No
        172.16.10.171   255.255.255.0   No
        172.16.10.172   255.255.255.0   No
        172.16.10.173   255.255.255.0   No
        172.16.10.174   255.255.255.0   No
        172.16.10.175   255.255.255.0   No
ucs-fi-6248up-0-A /org/ip-pool #

So we now have a pool of IPv4 addresses to hand out to CIMCs in our UCS domain. And we can also LOL at Cisco's DHCP enaled or disabled spelling error. But not too loud. It may well be fixed in newer versions.

Create VLAN and VLAN group to connect the CIMC IP pool to

VLAN on the Ethernet uplinks:

ucs-fi-6248up-0-A /org/ip-pool # top
ucs-fi-6248up-0-A# scope eth-uplink
ucs-fi-6248up-0-A /eth-uplink # create vlan Management 10
ucs-fi-6248up-0-A /eth-uplink/vlan* # set native no
ucs-fi-6248up-0-A /eth-uplink/vlan* # set vlan-id 10
ucs-fi-6248up-0-A /eth-uplink/vlan* # commit-buffer

VLAN group now:

ucs-fi-6248up-0-A /eth-uplink/vlan # up
ucs-fi-6248up-0-A /eth-uplink # show vlan-group
ucs-fi-6248up-0-A /eth-uplink # create vlan-group Management_group
ucs-fi-6248up-0-A /eth-uplink/vlan-group* # create member-vlan Management
ucs-fi-6248up-0-A /eth-uplink/vlan-group/member-vlan* # commit-buffer

Add IP pool, VLAN, VLAN group to in-band profile:

ucs-fi-6248up-0-A /eth-uplink/vlan-group/member-vlan # top
ucs-fi-6248up-0-A# scope eth-uplink
ucs-fi-6248up-0-A /eth-uplink # scope inband-profile
ucs-fi-6248up-0-A /eth-uplink/inband-profile # set default-pool-name CIMC_inband_pool
ucs-fi-6248up-0-A /eth-uplink/inband-profile* # set default-vlan-name Management
ucs-fi-6248up-0-A /eth-uplink/inband-profile* # set net-group-name Management_group
ucs-fi-6248up-0-A /eth-uplink/inband-profile* # commit-buffer
ucs-fi-6248up-0-A /eth-uplink/inband-profile # 

Now that the in-band profile is set, update a blade's service profile to reference it:

ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # show service-profile

Service Profile:
    Service Profile Name Type              Server  Assignment Association
    -------------------- ----------------- ------- ---------- -----------
    UCS_B200_M3_FCH162871NA
                         Instance          1/1     Assigned   Associating
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # set ext-mgmt-ip-pool-name CIMC_inband_pool
ucs-fi-6248up-0-A /org/service-profile* # set ext-mgmt-ip-state pooled
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A# top
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # reset hard-reset-immediate
ucs-fi-6248up-0-A /chassis/server* # commit-buffer
ucs-fi-6248up-0-A /chassis/server # reset-kvm
ucs-fi-6248up-0-A /chassis/server* # commit-buffer
ucs-fi-6248up-0-A /chassis/server* # reset-ipmi
ucs-fi-6248up-0-A /chassis/server* # commit-buffer
ucs-fi-6248up-0-A /chassis/server # scope cimc
ucs-fi-6248up-0-A /chassis/server/cimc # show

CIMC:
    PID              Serial (SN)      HW Revision
    ---------------- ---------------- -----------
    UCSB-B200-M3     FCH162871NA      0
ucs-fi-6248up-0-A /chassis/server/cimc # show detail

CIMC:
    Product Name: Cisco UCS B200 M3
    PID: UCSB-B200-M3
    VID: V01
    Vendor: Cisco Systems Inc
    Serial (SN): FCH162871NA
    HW Revision: 0
    GUID:
    Current Task:
ucs-fi-6248up-0-A /chassis/server/cimc # show expand

CIMC:
    PID: UCSB-B200-M3
    Serial (SN): FCH162871NA
    HW Revision: 0

    Vmedia Mapping List:
        Full Name: sys/chassis-1/blade-1/mgmt/actual-mount-list

        Vmedia Mapping:
            Vdisk Id Mapping Name Device Type Mount Protocol Mount Status
            -------- ------------ ----------- -------------- ------------
            1                     Cdd         Unknown        Not Mounted
            2                     Hdd         Unknown        Not Mounted

    External Management Interface:
        Mode: In Band
        Ip V4 State: Pooled
        Ip V6 State: Pooled

        External Management Virtual LAN:
            Network Name: Management
            Id: 10

            External Management Pooled IP:
                Name       IP Address      Default Gateway Subnet          Primary DNS IP  Secondary DNS IP
                ---------- --------------- --------------- --------------- --------------- ----------------
                CIMC_inband_pool
                           172.16.10.169   172.16.10.1     255.255.255.0   172.16.10.1     172.16.10.2

            External Management Pooled IPv6:
                Name       IP Address Default Gateway Prefix Primary DNS IP Secondary DNS IP
                ---------- ---------- --------------- ------ -------------- ----------------
                CIMC_inband_pool
                           ::         ::              64     ::             ::

    Management Interface:
        Access Type: Unspecified
        IP Address: 0.0.0.0
        Netmask: 0.0.0.0
        Gateway: 0.0.0.0
        MAC Address: 30:F7:0D:BE:89:EA
        Fabric ID: A
        Peer Port:
        Peer Port Id: 1
        Peer Slot Id: 1
        Peer Chassis Id: 1
        Discovery: Absent
        
    Management Endpoint Log Control:
        Type   ID  Capacity
        ------ --- --------
        SEL    0   Available
ucs-fi-6248up-0-A /chassis/server/cimc #

(Some) Success! The blade's CIMC is assigned an IPv4 address. And it is pingable on the local network. SSH is a no-go, though. Maybe not a thing at all on B series blades? And some further fiddling seems to be needed to get IPMI over LAN working. Something like this:

ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # create ipmi-access-profile ipmi_profile
ucs-fi-6248up-0-A /org/ipmi-access-profile* # create ipmi-user admin
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user* # set password
Enter a password:
Confirm the password:
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user* # commit-buffer
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user # set privilege admin
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user* # commit-buffer
ucs-fi-6248up-0-A /org/ipmi-access-profile/ipmi-user # up
ucs-fi-6248up-0-A /org/ipmi-access-profile # set ipmi-over-lan enable
ucs-fi-6248up-0-A /org/ipmi-access-profile # commit-buffer
ucs-fi-6248up-0-A /org/ipmi-access-profile # up
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # set ipmi-access-profile ipmi_profile
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile # top
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # scope cimc
ucs-fi-6248up-0-A /chassis/server/cimc # reset
ucs-fi-6248up-0-A /chassis/server/cimc* # commit-buffer
ucs-fi-6248up-0-A /chassis/server/cimc # top
ucs-fi-6248up-0-A# connect cimc 1/1
Trying 127.5.1.1...
Connected to 127.5.1.1.
Escape character is '^]'.

CIMC Debug Firmware Utility Shell [ support ]
[ help ]# help
__________________________________________
          Debug Firmware Utility          
__________________________________________
Command List
__________________________________________
alarms
cores
exit
i2cstats
images
mctools
memory
messages
mrcout
network
obfl
post
power
programmables
sensors
sel
fru
tasks
top
update
users
version
mezz1fru
mezz2fru
sldp
help [COMMAND]
__________________________________________
 Notes: 
"enter Key" will execute last command
"COMMAND ?" will execute help for that command
__________________________________________
[ help ]# users
0. [ "admin" ] [ "" ] [ 2 ]
1. [ Empty Entry ]
2. [ Empty Entry ]
3. [ Empty Entry ]
4. [ Empty Entry ]
5. [ Empty Entry ]
6. [ Empty Entry ]
7. [ Empty Entry ]
8. [ Empty Entry ]
9. [ Empty Entry ]
10. [ Empty Entry ]
11. [ Empty Entry ]
12. [ Empty Entry ]
13. [ Empty Entry ]
14. [ Empty Entry ]
15. [ Empty Entry ]
[ users ]# exit
Connection closed by foreign host.
ucs-fi-6248up-0-A#

So that looks like we should have IPMI over LAN. Let's poke at it a bit and see what happens:

$ ping -c 5 172.16.10.169
PING 172.16.10.169 (172.16.10.169) 56(84) bytes of data.
64 bytes from 172.16.10.169: icmp_seq=1 ttl=63 time=0.930 ms
64 bytes from 172.16.10.169: icmp_seq=2 ttl=63 time=0.655 ms
64 bytes from 172.16.10.169: icmp_seq=3 ttl=63 time=0.677 ms
64 bytes from 172.16.10.169: icmp_seq=4 ttl=63 time=0.658 ms
64 bytes from 172.16.10.169: icmp_seq=5 ttl=63 time=0.732 ms

--- 172.16.10.169 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4058ms
rtt min/avg/max/mdev = 0.655/0.730/0.930/0.103 ms
$ ipmitool -I lanplus -H 172.16.10.169 -U admin chassis status
Password: 
Error: Unable to establish IPMI v2 / RMCP+ session
$

And that is a promising start with a disappointing finish. Just for grins, let's try from a machine on the same subnet. Maybe ICMP echo replies get routed but the IPMI over LAN traffic doesn't?

router$ ping -c 5 172.16.10.169
PING 172.16.10.169 (172.16.10.169) 56(84) bytes of data.
64 bytes from 172.16.10.169: icmp_seq=1 ttl=64 time=1.85 ms
64 bytes from 172.16.10.169: icmp_seq=2 ttl=64 time=0.498 ms
64 bytes from 172.16.10.169: icmp_seq=3 ttl=64 time=0.463 ms
64 bytes from 172.16.10.169: icmp_seq=4 ttl=64 time=0.452 ms
64 bytes from 172.16.10.169: icmp_seq=5 ttl=64 time=0.460 ms 

--- 172.16.10.169 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4067ms
rtt min/avg/max/mdev = 0.452/0.745/1.853/0.554 ms
router$ ipmitool -I lanplus -H 172.16.10.169 -U admin mc info
Password: 
Device ID                 : 32
Device Revision           : 0
Firmware Revision         : 3.01
IPMI Version              : 2.0
Manufacturer ID           : 5771
Manufacturer Name         : Cisco Systems, Inc.
Product ID                : 9 (0x0009)
Product Name              : Unknown (0x09)
Device Available          : yes
Provides Device SDRs      : yes
Additional Device Support :
    Sensor Device
    SDR Repository Device
    SEL Device
    FRU Inventory Device
    IPMB Event Receiver
    IPMB Event Generator
Aux Firmware Rev Info     : 
    0x00
    0x00
    0x2a
    0x8b
router$ ipmitool -I lanplus -H 172.16.10.169 -U admin lan print
Password: 
Set in Progress         : Set Complete
Auth Type Support       : NONE MD2 MD5 PASSWORD 
Auth Type Enable        : Callback : 
                        : User     : 
                        : Operator : 
                        : Admin    : 
                        : OEM      : 
IP Address Source       : Static Address
IP Address              : 0.0.0.0
Subnet Mask             : 0.0.0.0
MAC Address             : 30:f7:0d:be:89:ea
SNMP Community String   : public
IP Header               : TTL=0x40 Flags=0x00 Precedence=0x00 TOS=0x10
BMC ARP Control         : ARP Responses Disabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl   : 2.0 seconds
Default Gateway IP      : 0.0.0.0
Default Gateway MAC     : 00:00:00:00:00:00
Backup Gateway IP       : 0.0.0.0
Backup Gateway MAC      : 00:00:00:00:00:00
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,128
Cipher Suite Priv Max   : XXXaaaXXaaaXaaa
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM
Bad Password Threshold  : Not Available
router$

Yay! Very strange that it can be pinged from a different subnet, but ipmitool cannot talk to it that way. Also very strange that it says it has no IP address assigned. I am just guessing here, but perhaps the code in the FI is doing that for us and I have not found the documentation describing how it works.

Serial console for blades?

I am fine with low bit rate serial stuff. No need for a fancy bit mapped screen to manage my computers. And it saves me having to run crazy Avocent Java applications in the process. So let's see what happens when we try to connect from a machine on the same subnet:

router$ ipmitool -I lanplus -H 172.16.10.169 -U admin sol activate
Password: 
Error activating SOL payload: Invalid data field in request
router$

Disappointing. But I think I might have seem something about serial over LAN in the service profiles somewhere. Let's see if we can get it turned on:

ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # show sol-policy
ucs-fi-6248up-0-A /org # create sol-policy sol_pol
ucs-fi-6248up-0-A /org/sol-policy* # enable
ucs-fi-6248up-0-A /org/sol-policy* # set speed 115200
ucs-fi-6248up-0-A /org/sol-policy* # commit-buffer
ucs-fi-6248up-0-A /org/sol-policy # up
ucs-fi-6248up-0-A /org # show service-profile

Service Profile:
    Service Profile Name Type              Server  Assignment Association
    -------------------- ----------------- ------- ---------- -----------
    UCS_B200_M3_FCH162871NA
                         Instance          1/1     Assigned   Associating
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # set sol-policy sol_pol
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile # top
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # scope cimc
ucs-fi-6248up-0-A /chassis/server/cimc # reset
ucs-fi-6248up-0-A /chassis/server/cimc* # commit-buffer
ucs-fi-6248up-0-A /chassis/server/cimc #

And trying again from a machine on the same subnet:

router$ ipmitool -I lanplus -H 172.16.10.169 -U admin sol activate
Password: 
Error activating SOL payload: Invalid data field in request
router$ 

So still no. Grrrr. Maybe it needs to be powered on first?

router$ ipmitool -I lanplus -H 172.16.10.169 -U admin chassis power status
Password: 
Chassis Power is off
router$ ipmitool -I lanplus -H 172.16.10.169 -U admin chassis power on
Password: 
Chassis Power Control: Up/On
router$ sleep 30
router$ ipmitool -I lanplus -H 172.16.10.169 -U admin sol activate
Password: 
Error activating SOL payload: Invalid data field in request
router$

More grrrr. Manuals suggest I might need a BIOS policy that enables the serial port, too?

ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # create bios-policy serial_is_good
ucs-fi-6248up-0-A /org/bios-policy* # set quiet-boot-config quiet-boot disabled
ucs-fi-6248up-0-A /org/bios-policy* # set serial-port-a-config serial-port-a enabled
ucs-fi-6248up-0-A /org/bios-policy* # set console-redir-config baud-rate 115200
ucs-fi-6248up-0-A /org/bios-policy* # set console-redir-config console-redir serial-port-a
ucs-fi-6248up-0-A /org/bios-policy* # set console-redir-config flow-control none
ucs-fi-6248up-0-A /org/bios-policy* # set console-redir-config legacy-os-redir disabled
ucs-fi-6248up-0-A /org/bios-policy* # set console-redir-config terminal-type vt100-plus
ucs-fi-6248up-0-A /org/bios-policy* # commit-buffer
ucs-fi-6248up-0-A /org/bios-policy # up
ucs-fi-6248up-0-A /org # show service-profile

Service Profile:
    Service Profile Name Type              Server  Assignment Association
    -------------------- ----------------- ------- ---------- -----------
    UCS_B200_M3_FCH162871NA
                         Instance          1/1     Assigned   Associating
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # set bios-policy serial_is_good
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile # top
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # scope cimc
ucs-fi-6248up-0-A /chassis/server/cimc # reset
ucs-fi-6248up-0-A /chassis/server/cimc* # commit-buffer
ucs-fi-6248up-0-A /chassis/server/cimc # top
ucs-fi-6248up-0-A#

So there is still no working serial console on the blade at this point. https://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-b-series-blade-servers/200105-Utilizing-SoL-logging-for-Serial-Redirec.html says to SSH to the CIMC IP address. Just getting a TCP RST back ("connection refused") when trying that. And the IPMI over LAN serial just does not seem to be supported at all. Maybe the blade needs a firmware update, too?

Applying firmware update to blades

So far (I think) I have only done firmware updates to the fabric interconnect and the blade chassis FEXes. First up, let's see what is available for our B200 M3 blades:

ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # show image | i b200-m3
ucs-b200-m3-bios.B200M3.2.0.4a.0.080920121557.bin
ucs-b200-m3-bios.B200M3.2.2.1a.0.111220131105.bin
ucs-b200-m3-brdprog.11.0.bin                  Board Controller     11.0
ucs-b200-m3-k9-cimc.2.0.4b.bin                CIMC                 2.0(4b)
ucs-b200-m3-k9-cimc.2.2.1c.bin                CIMC                 2.2(1c)
ucs-b200-m3-mrsasctlr.20.10.1-0100_4.30.00_NA.bin
ucs-b200-m3-mrsasctlr.20.12.1-0160_4.37.00_NA.bin
ucs-fi-6248up-0-A /firmware # 

Those might be kinda old? Let's see what is installed on the B200 M3 that is in the chassis:

ucs-fi-6248up-0-A /firmware # top
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # show firmware
Server 1/1:
    Adapter 1:
        Running-Vers: 4.1(3a)
        Package-Vers:
        Update-Status: Ready
        Activate-Status: Ready
    Adapter 2:
        Running-Vers: 4.1(3a)
        Package-Vers:
        Update-Status: Ready
        Activate-Status: Ready
    BIOS:
        Running-Vers: B200M3.2.2.6d.0.062220160055
        Package-Vers:
        Update-Status: Ready
        Activate-Status: Ready

    RAID Controller 1:
        Running-Vers: 20.13.1-0255
        Package-Vers:
        Activate-Status: Ready

    BoardController:
        Running-Vers: 15.0
ucs-fi-6248up-0-A /chassis/server #

From this, it would seem that the blade in the chassis is already running a newer firmware bundle than the one running on the fabric interconnect. So, let's find out what we are running on the FI, and see about some more updates to it:

ucs-fi-6248up-0-A /chassis/server # top
ucs-fi-6248up-0-A# show version brief
System version: 2.5(2a)
Package-Vers: 2.5(2a)A
ucs-fi-6248up-0-A# 

It would seem that 3.0(2f) is the latest in the next-newer release train. So acquire that from Cisco's download site and install on the FI and apply following the process documented above. And there is a hiccup with the SSH in 2.5(2a) not being able to get a host key from the SSH servers running on Debian 12 or Debian 13. Here's an example Debian server log of that:

adj@zarathud:~$ sudo grep ssh.*172.16.10.176 /var/log/auth.log
2026-04-02T17:24:54.156462+00:00 zarathud sshd[3090225]: Unable to negotiate with 172.16.10.176 port 33905: no matching host key type found. Their offer: x509v3-sign-rsa,x509v3-sign-dss,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss [preauth]
2026-04-02T17:41:20.766493+00:00 zarathud sshd[3092657]: Unable to negotiate with 172.16.10.176 port 58959: no matching host key type found. Their offer: x509v3-sign-rsa,x509v3-sign-dss,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss [preauth]
2026-04-02T17:50:44.154486+00:00 zarathud sshd[3094045]: Unable to negotiate with 172.16.10.176 port 45452: no matching host key type found. Their offer: x509v3-sign-rsa,x509v3-sign-dss,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss [preauth]
2026-04-02T17:51:36.645000+00:00 zarathud sudo:      adj : TTY=pts/22 ; PWD=/home/adj ; USER=root ; COMMAND=/usr/bin/grep ssh.*172.16.10.176 /var/log/auth.log
2026-04-02T17:53:33.561802+00:00 zarathud sudo:      adj : TTY=pts/22 ; PWD=/home/adj ; USER=root ; COMMAND=/usr/bin/grep ssh.*172.16.10.176 /var/log/auth.log
adj@zarathud:~$

So, for expediency's sake, we will just do it over TFTP this time. Maybe the SSH client in the 3.1(3l) release will be better able to talk to my SSH servers? That last remains to be seen. But the SSH server is a bit nearer to 2026 standards:

adj@yetanotherthinclient:~$ ssh admin@172.16.10.176
Unable to negotiate with 172.16.10.176 port 22: no matching host key type found. Their offer: ssh-rsa
adj@yetanotherthinclient:~$ ssh -o HostKeyAlgorithms=+ssh-rsa admin@172.16.10.176
Cisco UCS 6200 Series Fabric Interconnect
(admin@172.16.10.176) Password: 
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

ucs-fi-6248up-0-A# show version
System version: 3.1(3l)
ucs-fi-6248up-0-A# show version brief
System version: 3.1(3l)
Package-Vers: 3.1(3l)A
Service pack version: 3.1(3)SP0(Default)
Running-Modules:
Package-Vers:
ucs-fi-6248up-0-A# 

Still only does RSA host keys. But it does support more recent symmetric encryption schemes than the SSH server in 2.5(2a). But major excitement! The web UI now just needs HTML rendering. And ECMAscript. And CSS. But no more need for Java just to see what is going on with the UCS infrastructure. It does seem to still be needed for the KVM. But this is a major step forward. I very strongly recommend getting to the 3.1 UCS Manager releases.

And now for some server (blade) firmware. Available at the same place as the UCS Manager download, a section called "Related Software" has the server firmware images. Download and copy those to the UCS Manager running on the fabric interconnects, too:

ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # download image scp://adj@172.17.0.18/home/adj/Downloads/ucs-k9-bundle-b-series.3.1.3l.B.bin
ucs-fi-6248up-0-A /firmware # show download-task

Download task:
    File Name Protocol Server                                Userid          State
    --------- -------- ------------------------------------- --------------- -----
    ucs-k9-bundle-b-series.3.1.3l.B.bin
              Scp      172.17.0.18                           adj             Downloading
    ucs-k9-bundle-infra.2.2.8m.A.bin
              Tftp     172.17.0.18                                           Downloaded
    ucs-k9-bundle-infra.2.5.2a.A.bin
              Scp      172.17.0.18                           adj             Downloaded
    ucs-k9-bundle-infra.3.1.3l.A.bin
              Tftp     172.17.0.18                                           Downloaded
    ucs-mini-k9-bundle-infra.3.0.2f.A.bin
              Tftp     172.17.0.18                                           Downloaded
ucs-fi-6248up-0-A /firmware # show download-task ucs-k9-bundle-b-series.3.1.3l.B.bin detail
 
Download task:
    File Name: ucs-k9-bundle-b-series.3.1.3l.B.bin
    Protocol: Scp
    Server: 172.17.0.18
    Userid: adj
    Path: /home/adj/Downloads
    Downloaded Image Size (KB): 517633
    State: Downloading
    Current Task: unpacking image ucs-k9-bundle-b-series.3.1.3l.B.bin on primary(FSM-STAGE:sam:dme:FirmwareDownloaderDownload:UnpackLocal)
ucs-fi-6248up-0-A /firmware # show download-task ucs-k9-bundle-b-series.3.1.3l.B.bin detail

Download task:
    File Name: ucs-k9-bundle-b-series.3.1.3l.B.bin
    Protocol: Scp
    Server: 172.17.0.18
    Userid: adj
    Path: /home/adj/Downloads
    Downloaded Image Size (KB): 517633
    State: Downloaded
    Current Task:
ucs-fi-6248up-0-A /firmware #

Note that the UCS Manager SSH client can now talk to a Debian 12 OpenSSH server. Small wins are still wins!

A new step for firmware infrastructure updates

It seems that one must accept that a fabric interconnect reboot has happened before applying a new infrastructure firmware pack. Something like this:

ucs-fi-6248up-0-A# scope monitoring
ucs-fi-6248up-0-A /monitoring # show new-faults
Severity  Code     Last Transition Time     ID       Description
--------- -------- ------------------------ -------- -----------
Info      F0440    2026-04-02T18:37:14.718    113208 Chassis discovery policy conflict: Link IOM 1/1/2 to peer port A:N/A/1/2 not configured
ucs-fi-6248up-0-A /monitoring # top
ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # scope auto-install
ucs-fi-6248up-0-A /firmware/auto-install # acknowledge primary fabric-interconnect reboot 
ucs-fi-6248up-0-A /firmware/auto-install* # commit-buffer
ucs-fi-6248up-0-A /firmware/auto-install # install infra infra-vers 3.2(3p)A
This operation upgrades firmware on UCS Infrastructure Components
(UCS manager, Fabric Interconnects and IOMs).
Here is the checklist of things that are recommended before starting Auto-Install
(1) Review current critical/major faults
(2) Initiate a configuration backup
(3) Check if Management Interface Monitoring Policy is enabled
(4) Check if there is a pending Fabric Interconnect Reboot activitiy
(5) Ensure NTP is configured
(6) Check if any hardware (fabric interconnects, io-modules, servers or adapters) is unsupported in the target release
(7) Some fabric-interconnect service-pack install/uninstall will do additional FI reboots in order to complete install/uninstall
    To ensure before triggering auto-install, check manual service-pack activation/remove where warning is given if reboots are required
Do you want to proceed? (yes/no):yes

Triggering Install-Infra with:
   Infrastructure Pack Version: 3.2(3p)A
Warning: Any Service Pack installed in the system will be removed

ucs-fi-6248up-0-A /firmware/auto-install # acknowledge primary fabric-interconnect reboot
Warning: Check the outstanding faults (scope monitoring <enter> show new-faults) since last FI reboot. 
Please make sure the data paths are recovered before proceeding with this FI reboot to ensure there is no interruption to the data traffic.

ucs-fi-6248up-0-A /firmware/auto-install* # commit-buffer 
ucs-fi-6248up-0-A /firmware/auto-install # 
Broadcast message from root (Thu Apr  2 23:36:05 2026):

The system is going down for reboot NOW!
Connection to 172.16.10.176 closed by remote host.
Connection to 172.16.10.176 closed.

The UCS Manager SSH server generated itself a new, longer host RSA key as part of the 3.2(3p) upgrade. So expect host key validation failures after the FI reboots and you tell your SSH client to connect again. Maybe we can get back to updates for the blade servers' firmware now?

Blade server firmware. Another attempt.

This is going to assume that the server (blade) firmware has been copied to the UCS manager. Cisco's filename for the B series blade firmware bundles for the 3.2(3p) release is ucs-k9-bundle-b-series.3.2.3p.B.bin.

ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # show image | i b200-m3
ucs-b200-m3-bios.B200M3.2.0.4a.0.080920121557.bin
ucs-b200-m3-bios.B200M3.2.2.1a.0.111220131105.bin
ucs-b200-m3-bios.B200M3.2.2.6h.0.110720191420.bin
ucs-b200-m3-brdprog.11.0.bin                  Board Controller     11.0
ucs-b200-m3-brdprog.15.0.bin                  Board Controller     15.0
ucs-b200-m3-brdprog.16.0.bin                  Board Controller     16.0
ucs-b200-m3-k9-cimc.2.0.4b.bin                CIMC                 2.0(4b)
ucs-b200-m3-k9-cimc.2.2.1c.bin                CIMC                 2.2(1c)
ucs-b200-m3-k9-cimc.3.1.23e.bin               CIMC                 3.1(23e)
ucs-b200-m3-k9-cimc.3.1.26p.bin               CIMC                 3.1(26p)
ucs-b200-m3-mrsasctlr.20.10.1-0100_4.30.00_NA.bin
ucs-b200-m3-mrsasctlr.20.12.1-0160_4.37.00_NA.bin
ucs-b200-m3-mrsasctlr.20.13.1-0255_4.38.02.2_NA.bin
ucs-fi-6248up-0-A /firmware # top
ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # show fw-host-pack

Server Host Pack:
    Name                 Mode
    -------------------- ----
    default              Staged
    test                 Staged
ucs-fi-6248up-0-A /org # create fw-host-pack b200-m3-3.2.3p
ucs-fi-6248up-0-A /org/fw-host-pack* # create pack-image "Cisco Systems Inc" UCSB-B200-M3 board-controller 16.0
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image* # set version 16.0
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image* # commit-buffer
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image # exit
ucs-fi-6248up-0-A /org/fw-host-pack # create pack-image "Cisco Systems Inc" UCSB-B200-M3 cimc "3.1(26p)"
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image* # set version "3.1(26p)"
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image* # commit-buffer
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image # exit
ucs-fi-6248up-0-A /org/fw-host-pack # create pack-image "Cisco Systems, Inc." UCSB-B200-M3 server-bios "B200M3.2.2.6h.0.110720191420"
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image* # set version "B200M3.2.2.6h.0.110720191420"
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image* # commit-buffer
ucs-fi-6248up-0-A /org/fw-host-pack/pack-image # exit
ucs-fi-6248up-0-A /org/fw-host-pack # show

Server Host Pack:
    Name                 Mode
    -------------------- ----
    b200-m3-3.2.3p       Staged
ucs-fi-6248up-0-A /org/fw-host-pack # up
ucs-fi-6248up-0-A /org # show fw-host-pack

Server Host Pack:
    Name                 Mode
    -------------------- ----
    b200-m3-3.2.3p       Staged
    default              Staged
    test                 Staged
ucs-fi-6248up-0-A /org #

VERY IMPORTANT DETAIL: In the above, see how sometimes the vendor string is "Cisco Systems Inc" and other times it is "Cisco Systems, Inc."? This is very stupid. Very easy to miss. And very important to getting a configuration that will validate.

And it still needs to be added to a service profile:

ucs-fi-6248up-0-A /org # scope org /
ucs-fi-6248up-0-A /org # show service-profile

Service Profile:
    Service Profile Name Type              Server  Assignment Association
    -------------------- ----------------- ------- ---------- -----------
    UCS_B200_M3_FCH162871NA
                         Instance          1/1     Assigned   Associating
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # set host-fw-policy b200-m3-3.2.3p
ucs-fi-6248up-0-A /org/service-profile # reapply-config
Warning: Any unapplied configuration changes will now get applied. This potentially will cause a reboot of the associated server.
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile # show assoc
Service Profile Name Association    Server  Server Pool
-------------------- -------------- ------- -----------
UCS_B200_M3_FCH162871NA
                     Associating    1/1
ucs-fi-6248up-0-A /org/service-profile #

It still says Associating. So has not completely worked just yet. Let's see if we can figure out why.

ucs-fi-6248up-0-A /org/service-profile # top
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # show assoc
Association  Service Profile
------------ ---------------
None         UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /chassis/server # 

So not sure what the issue is from that. One more attempt here. We know the service profile name. Let's see if there are any faults listed that mention it:

ucs-fi-6248up-0-A# show fault | i UCS_B200_M3_FCH162871NA
Critical  F1000035 2026-03-30T23:08:37.136    281911 [FSM:FAILED]: Configuring Service Profile UCS_B200_M3_FCH162871NA(FSM:sam:dme:LsServerConfigure)
Major     F0327    2026-03-23T20:49:35.442    124543 Service profile UCS_B200_M3_FCH162871NA configuration failed due to insufficient-resources,mac-address-assignment
ucs-fi-6248up-0-A# show fault 124543 detail
Fault Instance

Severity: Major
Code: F0327
Last Transition Time: 2026-03-23T20:49:35.442
ID: 124543
Status: Pinned
Description: Service profile UCS_B200_M3_FCH162871NA configuration failed due to insufficient-resources,mac-address-assignment
Affected Object: org-root/ls-UCS_B200_M3_FCH162871NA
Name: Ls Server Config Failure
Cause: Configuration Failure
Type: Server
Acknowledged: No
Occurrences: 1
Creation Time: 2026-03-23T20:49:35.442
Original Severity: Major
Previous Severity: Major
Highest Severity: Major
ucs-fi-6248up-0-A# 

That seems like it is likely the culprit. We can't even get a server firmware update without giving it synthetic MAC addresses? OMFG!

Let us see what we can see... I see something about a LAN connectivity policy inside our organization. Can we just make a simple one?

ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # show lan-connectivity-policy
ucs-fi-6248up-0-A /org # create lan-connectivity-policy test_lancon_pol
ucs-fi-6248up-0-A /org/lan-connectivity-policy* # create vnic test_vnic eth-if vlan_1000 fabric a-b
ucs-fi-6248up-0-A /org/lan-connectivity-policy/vnic* # commit-buffer
ucs-fi-6248up-0-A /org/lan-connectivity-policy # show expand

vNIC LAN connectivity policy:
    Name: test_lancon_pol
    Description:

    vNIC:
        Name: test_vnic
        Fabric ID: A B
        Dynamic MAC Addr: Derived
        Virtualization Preference: NONE

        Ethernet Interface:
            Name: vlan_1000
            Dynamic MAC Addr: Derived
            Type: Ether
            Owner: Logical
            Default Network: No
            Config Qual: N/A
ucs-fi-6248up-0-A /org/lan-connectivity-policy # up
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # set lan-connectivity-policy-name test_lancon_pol
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer verify-only
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile # reapply-config
Warning: Any unapplied configuration changes will now get applied. This potentially will cause a reboot of the associated server.
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile # top
ucs-fi-6248up-0-A# show fault | i UCS_B200_M3_FCH162871NA
Minor     F0169    2026-04-03T05:03:54.043    518234 Eth vNIC test_vnic, service profile UCS_B200_M3_FCH162871NA failed to apply configuration
Critical  F1000035 2026-03-30T23:08:37.136    281911 [FSM:FAILED]: Configuring Service Profile UCS_B200_M3_FCH162871NA(FSM:sam:dme:LsServerConfigure)
Major     F0327    2026-03-23T20:49:35.442    124543 Service profile UCS_B200_M3_FCH162871NA configuration failed due to insufficient-resources,connection-placement,insufficient-vnic-capacity
ucs-fi-6248up-0-A# show fault 518234
Severity  Code     Last Transition Time     ID       Description
--------- -------- ------------------------ -------- -----------
Minor     F0169    2026-04-03T05:03:54.043    518234 Eth vNIC test_vnic, service profile UCS_B200_M3_FCH162871NA failed to apply configuration
ucs-fi-6248up-0-A# show fault 518234 detail
Fault Instance

Severity: Minor
Code: F0169
Last Transition Time: 2026-04-03T05:03:54.043
ID: 518234
Status: None
Description: Eth vNIC test_vnic, service profile UCS_B200_M3_FCH162871NA failed to apply configuration
Affected Object: org-root/ls-UCS_B200_M3_FCH162871NA/ether-test_vnic
Name: Vnic Ether Config Failed
Cause: Configuration Failed
Type: Configuration
Acknowledged: No
Occurrences: 1
Creation Time: 2026-04-03T05:03:54.043
Original Severity: Minor
Previous Severity: Minor
Highest Severity: Minor
ucs-fi-6248up-0-A# 

Still not there. Inching closer. Hopefully.