UCS Manager

There are a few pieces of Cisco UCS equipment in the data center:

• One UCS 5108 blade enclosure, accommodating up to 8 B200 server blades of various generations
• 2 UCS-FI-6248UP fabric interconnects, which sit between UCS servers or blade chassis and the rest of the network
• 1 UCS C220 M5 server

Cisco UCS Manager gets installed on the Fabric Interconnects and does Cisco proprietary magic things. Like powering on a single blade. This is way more complicated that it needs to be. So this document is being written to capture knowledge of installation, upgrades, and server management tasks.

How to get

Much (all?) of Cisco's UCS software is available directly without a support contract. But a customer account is required to download. Registration is free as in beer. I have had one since July 2021 and have not been troubled with Cisco spam or any sort of "are you a real person at a real company?" validation.

As this is written, UCS Infrastructure and UCS Manager Software is the place to get it. As I have some older, end-of-life, end-of-support fabric interconnects (62xx series) which are not supported on the current 6.0 release train, I have instead selected the 4.2(3p) version for download. This comes as a 1Gbyte-ish ucs-k9-bundle-infra.4.2.3p.A.bin file.

Untested Fabric Interconnect, what do we do?

5ish years ago (summer 2021), I acquired three of these 6248UP FIs. And I think I got one of them configured for use on the management network and updated to then-current software. I'm not sure which of the three that was. And I am sure I did no documentation at the time. So here we are with new notes.

Console and power are at the front?!?!

So, this is weird. Console and power connections are at the front side of the chassis. All of the connectivity is at the rear as it should be. There is sufficient room on the back for a couple of C14 power connectors and 2 more 8P8C modular connectors for management LAN and serial console. But they are at the front. So cabling is kinda sorta extra fun. sigh

Console serial cable is the usual Cisco modular pinout. Signal on pins 3 and 6. 9600 bps, 8-n-1 are the serial port settings. Nothing unusual there.

Mystery solved regarding the connections at front of the chassis. My FIs have 32 ports for traffic, but there is (well, was, back when this was a current product) a 16 port expansion module available. See https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-6200-series-fabric-interconnects/data_sheet_c78-675245.html for some diagrams.

Basic Fabric Interconnect configuration

I might have lucked out on the first one I received, but it booted normally and started the interactive system setup dialog. It is all pretty basic. Set a password for the admin user and an IP address, netmask, IPv4 gateway address, and DNS server address for the management Ethernet interface (again, on the front). That's pretty much it.

After this is done, make sure the management Ethernet interface is connected to a switch on the correct VLAN and subnet. Then do all the things over an SSH connection. At least for version 2.1, the SSH server only knows about RSA host keys, so a modern OpenSSH client will need something like -o HostKeyAlgorithms=+ssh-rsa listed as a command line option.

Clock discipline is good

To set the time in the FI. Or just get it close. NTP will come. Eventually.

ucs-fi-6248up-0-A# scope system
ucs-fi-6248up-0-A /system # scope services
ucs-fi-6248up-0-A /system/services # set clock Mar 22 2026 02 53 35
Sun Mar 22 02:53:35 UTC 2026
ucs-fi-6248up-0-A /system/services #

And now for the NTP:

ucs-fi-6248up-0-A# scope system
ucs-fi-6248up-0-A /system # scope services
ucs-fi-6248up-0-A /system/services # create ntp-server 172.16.10.2
ucs-fi-6248up-0-A /system/services* # commit-buffer
ucs-fi-6248up-0-A /system/services # create ntp-server 172.16.10.3
ucs-fi-6248up-0-A /system/services* # commit-buffer
ucs-fi-6248up-0-A /system/services # show ntp
NTP Servers:
    Name: 172.16.10.2
    Name: 172.16.10.3
ucs-fi-6248up-0-A /system/services #

Software upgrade time!

Again, on the first one I have tried, let's log in as admin and see what's running:

Cisco UCS 6200 Series Fabric Interconnect

ucs-fi-6248up-0-A login: admin
Password: 
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

ucs-fi-6248up-0-A# show version
System version: 2.2(1c)
ucs-fi-6248up-0-A#

And, more verbosely (for reasons that are beyond me, "brief" mode shows more output?):

ucs-fi-6248up-0-A# show version brief
System version: 2.2(1c)
Package-Vers: 2.2(1c)A
ucs-fi-6248up-0-A# 

So that's really very super dooper old. I downloaded a copy of the 4.1(3i) code in August 2022. Off do so some reading to see if I can skip all the in-between versions.

Did not find anything authoritative in the release notes (but I only looked very briefly.) So, we can do it stepwise. Hopefully. Process looks something like this:

ucs-fi-6248up-0-A# scope firmware 
ucs-fi-6248up-0-A /firmware # download image tftp://172.17.0.18/ucs-k9-bundle-infra.2.2.8m.A.bin
ucs-fi-6248up-0-A /firmware # show download-task

Download task:
    File Name Protocol Server          Userid          State
    --------- -------- --------------- --------------- -----
    ucs-k9-bundle-infra.2.2.8m.A.bin
              Tftp     172.17.0.18                     Downloaded
ucs-fi-6248up-0-A /firmware # 

Yes, you read that right. I delivered the software image to the FI over TFTP. SSH (scp and sftp) are supported protocols, but could not connect to an OpenSSH server on Debian 13. This might be better with newer FI firmware, but for now, I already have a TFTP server handy for netbooting other machines in the environment.

And from here, the firmware bundle can actually be installed on the Fabric Interconnect like so:

ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # scope auto-install
ucs-fi-6248up-0-A /firmware/auto-install # install infra infra-vers 2.2(8m)A
This operation upgrades firmware on UCS Infrastructure Components
(UCS manager, Fabric Interconnects and IOMs). Do you want to proceed?
 (yes/no):yes

Triggering Install-Infra with:
   Infrastructure Pack Version: 2.2(8m)A
ucs-fi-6248up-0-A /firmware/auto-install # acknowledge primary fabric-interconnect reboot
ucs-fi-6248up-0-A /firmware/auto-install* # commit-buffer
ucs-fi-6248up-0-A /firmware/auto-install # show fsm status expand


    FSM Status:

        Affected Object: sys/fw-system/fsm
        Current FSM: Deploy
        Status: In Progress
        Completion Time:
        Progress (%): 87

        FSM Stage:

        Order  Stage Name                               Status       Try
        ------ ---------------------------------------- ------------ ---
        1      DeployWaitForDeploy                      Success      0
        2      DeployResolveDistributableNames          Skip         0
        3      DeployResolveDistributable               Skip         0
        4      DeployResolveImages                      Skip         0
        5      DeployActivateUCSM                       Success      0
        6      DeployPollActivateOfUCSM                 In Progress  1
        7      DeployUpdateIOM                          Pending      0
        8      DeployPollUpdateOfIOM                    Pending      0
        9      DeployActivateIOM                        Pending      0
        10     DeployPollActivateOfIOM                  Pending      0
        11     DeployActivateRemoteFI                   Pending      0
        12     DeployPollActivateOfRemoteFI             Pending      0
        13     DeployWaitForUserAck                     Pending      0
        14     DeployActivateLocalFI                    Pending      0
        15     DeployPollActivateOfLocalFI              Pending      0
ucs-fi-6248up-0-A /firmware/auto-install #  Connection to 172.16.10.176 closed.

Somewhere along the way, the SSH server hung up on me. But that seems like the sort of thing that should happen as the FI's OS is being upgraded. Progress can also be tracked on the serial console port like so:

ucs-fi-6248up-0-A# scope firmware
ucs-fi-6248up-0-A /firmware # scope auto-install
ucs-fi-6248up-0-A /firmware/auto-install # show fsm status expand | no-more


    FSM Status:

        Affected Object: sys/fw-system/fsm
        Current FSM: Deploy
        Status: In Progress
        Completion Time:
        Progress (%): 98

        FSM Stage:

        Order  Stage Name                               Status       Try
        ------ ---------------------------------------- ------------ ---
        1      DeployWaitForDeploy                      Success      0
        2      DeployResolveDistributableNames          Skip         0
        3      DeployResolveDistributable               Skip         0
        4      DeployResolveImages                      Skip         0
        5      DeployDownloadImages                     Skip         0
        6      DeployCopyAllImagesToPeer                Skip         0
        7      DeployInternalBackup                     Success      0
        8      DeployPollInternalBackup                 Success      2
        9      DeployActivateUCSM                       Skip         0
        10     DeployPollActivateOfUCSM                 Success      0
        11     DeployUpdateIOM                          Success      0
        12     DeployPollUpdateOfIOM                    Success      0
        13     DeployActivateIOM                        Success      0
        14     DeployPollActivateOfIOM                  Success      0
        15     DeployActivateRemoteFI                   Skip         0
        16     DeployPollActivateOfRemoteFI             Skip         0
        17     DeployWaitForUserAck                     Skip         0
        18     DeployPollWaitForUserAck                 Success      0
        19     DeployActivateLocalFI                    Success      0
        20     DeployPollActivateOfLocalFI              In Progress  2
ucs-fi-6248up-0-A /firmware/auto-install # 

At some point, the FI will reboot itself.

Confirm new code is running like so:

ucs-fi-6248up-0-A# show version
System version: 2.2(8m)
ucs-fi-6248up-0-A# show version brief
System version: 2.2(8m)
Package-Vers: 2.2(8m)A
ucs-fi-6248up-0-A#

STOP! Don't forget the other UCS things!

Um, yeah. So there is this UCS 5108 blade chassis in the data center, too. No idea what its firmware versions are. But they are sure to also need firmware updates. And it would be distressing if those components were left in a state where they are unusable because the FIs are so much newer than the blade enclosure and its servers. So I guess we need to see about warming the ZIP code 5degC by attaching the blade chassis to the mains power and see what we can see.

Correct cabling for single FI to UCS 5108

According to https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/chassis-install-guide/ucs5108_install/ucs5108_install_chapter_010.html, a single Fabric Interconnect can only be connected to a single Fabric Extender on a UCS chassis. So for now, enable 6248UP ports 1 and 3, and cable to ports 1 and 2 on the left side (as seen from the rear) Fabric Extender.

SFPs go into the FI and the UCS 5108 FEXes

Pretty self explanatory, right. Find Cisco 10GbE SR SFP+ modules, insert into port one on each of the FEXes, insert into ports 1 and 3 on the FI. Attach nice aqua colored cable. And don't panic as the lights don't come on.

Telling the FI to discover the blade chassis

This is way more complicated that it needs to be. I'm sure it's totally amazing if you have racks full of Cisco UCS hardware. But I have just the one blade chassis and the one C series server and the 2 fabric interconnects. And the FI management software is all lovely out of date Java stuff. So fumbling around the CLI, hoping to find my way out of the darkness here.

The process for this seems to be something like:

• Configure FI Ethernet ports as server ports
• Ensure the attached chassis has been discovered and acknowledge any faults

ucs-fi-6248up-0-A# scope eth-server
ucs-fi-6248up-0-A /eth-server # show fabric a detail

Fabric:
    Id: A
    Current Task:

ucs-fi-6248up-0-A /eth-server # scope fabric a
ucs-fi-6248up-0-A /eth-server/fabric # create interface 1 1
ucs-fi-6248up-0-A /eth-server/fabric/interface* # enable
ucs-fi-6248up-0-A /eth-server/fabric/interface* # show interface detail
ucs-fi-6248up-0-A /eth-server/fabric/interface* # commit-buffer
ucs-fi-6248up-0-A /eth-server/fabric/interface # exit
ucs-fi-6248up-0-A /eth-server/fabric # create interface 1 3
ucs-fi-6248up-0-A /eth-server/fabric/interface* # enable
ucs-fi-6248up-0-A /eth-server/fabric/interface* # commit-buffer
ucs-fi-6248up-0-A /eth-server/fabric/interface # exit
ucs-fi-6248up-0-A /eth-server/fabric # exit
ucs-fi-6248up-0-A /eth-server # show interface

Interface:

Fabric  Slot  Port  Admin State  Oper State   State Reason                    Chassis  Lic State            Grace Prd
------- ----- ----- ------------ ------------ ------------------------------- -------- -------------------- ---------
A           1     1 Enabled      Up                                           1        License Ok                         0
A           1     3 Enabled      Up                                           1        License Ok                         0
ucs-fi-6248up-0-A /eth-server # exit
ucs-fi-6248up-0-A# show chassis inventory
Chassis    PID             Vendor            Serial (SN) HW Revision
---------- --------------- ----------------- ----------- -----------
         1 N20-C6508       Cisco Systems Inc FOX1808GLJG 0
ucs-fi-6248up-0-A#

Success! Serial number reported in the UCS Manager matches the one I found on the label.

So how do I power on a server?

Still fumbling through here. But: servers (blade servers for sure, not sure about rack servers) belong to UCS "Organizations." These organizations are managed by the UCS Manager, too. And since an organization owns a server (blade or otherwise) the organization can power servers on and off.

What orgs exist?

This is what we get from a factory default UCS Manager:

ucs-fi-6248up-0-A# show org

Organizations:
    Name
    ----
    / (root)
ucs-fi-6248up-0-A# show org detail

Organizations:
    Name: / (root)
ucs-fi-6248up-0-A#

Enter an org, see what we have for basic policies

ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # show bladeserver-disc-policy
ucs-fi-6248up-0-A /org # show chassis-conn-policy

Chassis/FEX Connectivity Policy:
    Chassis/FEX Id Chassis/FEX Name Fabric Id Link Aggregation Preference
    -------------- ---------------- --------- ---------------------------
                 1 sys/chassis-1    A         Global
ucs-fi-6248up-0-A /org # show chassis-disc-policy

Chassis/FEX Discovery Policy:
    Description Qualifier  Action            Rebalance         Link Aggregation Pref Multicast Hw Hash
    ----------- ---------- ----------------- ----------------- --------------------- -----------------
                none       1 Link            User Acknowledged None                  Disabled
ucs-fi-6248up-0-A /org # show chassis-disc-policy detail

Chassis/FEX Discovery Policy:
    Description:
    Qualifier: none
    Action: 1 Link
    Rebalance: User Acknowledged
    Link Aggregation Pref: None
    Multicast Hw Hash: Disabled
ucs-fi-6248up-0-A /org # show rackserver-disc-policy

Rack Server Discovery Policy:
    Action            Scrub Policy
    ----------------- ------------
    Immediate
ucs-fi-6248up-0-A /org # show rackserver-disc-policy detail

Rack Server Discovery Policy:
    Action: Immediate
    Scrub Policy:
    Description:
    Current Task:
ucs-fi-6248up-0-A /org # show rackserver-mgmt-policy

Rack Server Management Policy:
    Action
    ------
    Auto Acknowledged
ucs-fi-6248up-0-A /org # show rackserver-mgmt-policy detail

Rack Server Management Policy:
    Action: Auto Acknowledged
ucs-fi-6248up-0-A /org #

Create a profile and add a (blade) server to it

ucs-fi-6248up-0-A# show chassis inventory 1
Chassis    PID             Vendor            Serial (SN) HW Revision
---------- --------------- ----------------- ----------- -----------
         1 N20-C6508       Cisco Systems Inc FOX1808GLJG 0
ucs-fi-6248up-0-A# show chassis inventory 1 expand 
Chassis 1:
    Servers:
        Server 1/1:
            Equipped Product Name: Cisco UCS B200 M3
            Equipped PID: UCSB-B200-M3
            Equipped VID: V03
            Equipped Serial (SN): FCH162871NA
            Slot Status: Equipped
            Acknowledged Product Name: Cisco UCS B200 M3
            Acknowledged PID: UCSB-B200-M3
            Acknowledged VID: V03
            Acknowledged Serial (SN): FCH162871NA
            Acknowledged Memory (MB): 196608
            Acknowledged Effective Memory (MB): 196608
            Acknowledged Cores: 0
            Acknowledged Adapters: 2

        Server 1/2:
            Equipped Product Name: Cisco UCS B200 M3
            Equipped PID: UCSB-B200-M3
            Equipped VID: V06
            Equipped Serial (SN): FCH1811JELC
            Slot Status: Equipped
            Acknowledged Product Name: Cisco UCS B200 M3
        [...]
ucs-fi-6248up-0-A# top
ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # create service-profile
ucs-fi-6248up-0-A /org # create service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile* # associate server 1/1
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile #

I can power on now?

Well, no. Maybe it should have worked. But for this blade, there was a Power On Self Test (POST) failure. Let's dig in. To power on the server included in the service profile, do this:

ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # power up
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile # top
ucs-fi-6248up-0-A #

And let's have a look at its status:

ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # show

Server:
    Slot    Overall Status        Service Profile      Availability
    ------- --------------------- -------------------- ------------
          1 Compute Failed                             Unavailable
ucs-fi-6248up-0-A /chassis/server # show detail

Server:
    Slot: 1
    Name:
    User Label:
    Overall Status: Compute Failed
    Oper Qualifier: Compute Post Failure
    Service Profile:
    Association: None
    Availability: Unavailable
    Discovery: Failed
    Conn Path: A
    Conn Status: A
    Managing Instance: A
    Admin Power: Policy
    Oper Power: Off
    Admin State: In Service
    Product Name: Cisco UCS B200 M3
    PID: UCSB-B200-M3
    VID: V03
    Vendor: Cisco Systems Inc
    Serial (SN): FCH162871NA
    HW Revision: 0
    Mfg Date: 2012-07-26T00:00:00.000
    Part Number: 73-13217-08
    Memory (MB): 196608
    Effective Memory (MB): 196608
    Operating Memory Speed (MHz): 1333
    Operating Memory Voltage: Regular Voltage
    Cores: 0
    Num Of Cores Enabled: 0
    Adapters: 2
    Eth Host Interfaces: 0
    FC Host Interfaces: 0
    Burned-In UUID: 00000000-0000-0000-0000-000000000000
    Dynamic UUID: 00000000-0000-0000-0000-000000000000
    Current Task 1: Checking hardware configuration server 1/1(FSM-STAGE:sam:dme:ComputeBladeDiscover:Sanitize)
    Current Task 2:
    Current Task 3:
ucs-fi-6248up-0-A /chassis/server # show post

POST:
    Global ID Code      Severity  Affected Object                  Description
    --------- --------- --------- -------------------------------- -----------
    6215      POST-6215 Critical  sys/chassis-1/blade-1            Board Programmable version not valid for Processor Type
ucs-fi-6248up-0-A /chassis/server # show post detail

POST:
    Global ID: 6215
    Code: POST-6215
    Local ID: 1536
    Severity: Critical
    Affected Object: sys/chassis-1/blade-1
    Description: Board Programmable version not valid for Processor Type
    Type: server: Cisco Systems Inc UCSB-B200-M3
    Recoverable: Non Recoverable
    Recovery Action: Refer the CPU upgrade guide.
    Timestamp: 2026-03-22T16:17:06.001
ucs-fi-6248up-0-A /chassis/server #

Making an inferences from "Board Programmable version not valid for Processor Type" I am thinking that it does not like the Xeon E5-2650v2 I installed. Now to find a not v2 E5-26xx CPU and see if that works any better. I found a not obviously damaged E5-2609 in storage (er, piles in the garage) and installed that in the blade. Let's see what we have now:

ucs-fi-6248up-0-A# scope org /
ucs-fi-6248up-0-A /org # scope service-profile UCS_B200_M3_FCH162871NA
ucs-fi-6248up-0-A /org/service-profile # power up
ucs-fi-6248up-0-A /org/service-profile* # commit-buffer
ucs-fi-6248up-0-A /org/service-profile # top
ucs-fi-6248up-0-A# scope server 1/1
ucs-fi-6248up-0-A /chassis/server # show inventory
Server 1/1:
    Name:
    User Label:
    Equipped PID: UCSB-B200-M3
    Equipped VID: V03
    Equipped Serial (SN): FCH162871NA
    Slot Status: Equipped
    Acknowledged Product Name: Cisco UCS B200 M3
    Acknowledged PID: UCSB-B200-M3
    Acknowledged VID: V03
    Acknowledged Serial (SN): FCH162871NA
    Acknowledged Memory (MB): 16384
    Acknowledged Effective Memory (MB): 16384
    Acknowledged Cores: 4
    Acknowledged Adapters: 2
ucs-fi-6248up-0-A /chassis/server # top
ucs-fi-6248up-0-A# 

Great Success! The Xeon E5-2609 is a 4 core CPU. And there are 4x 4GiByte memory DIMMs installed. So inventory looks good. Huzzah! (And what an utter pain.)

Server and Infrastructure issues

So, we just saw the first one of these above. The B200 M3 blades do not like Xeon E5-26xx v2 processors. At least with the firmware installed at present. Let's figure that one out first. https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/hw/blade-servers/B200M3.html#reference_CD3077C1064743F99C0F9A935778BC2F has a little table listing minimum versions for various components to get the Xeon E5-2600 v2 CPUs to run. These are the required versions for E5-26xx V2 CPUs:

Software or Firmware	Minimum Version
Server CIMC	2.1(3)
Server BIOS	2.1(3)
Cisco UCS Manager	2.1(3)
Board controller firmware	8.0

Let's see if we can see what we are running in one of the blades I have and the UCS Manager itself:

ucs-fi-6248up-0-A# scope chassis 1
ucs-fi-6248up-0-A /chassis # scope server 1/1
ucs-fi-6248up-0-A /chassis/server # show firmware cimc
Server  Running-Vers    Package-Vers    Update-Status   Activate-Status
------- --------------- --------------- --------------- ---------------
1/1     3.1(23c)                        Ready           Ready
ucs-fi-6248up-0-A /chassis/server # show firmware bios
Server 1/1:
    BIOS:
        Running-Vers: B200M3.2.2.6d.0.062220160055
        Package-Vers:
        Update-Status: Ready
        Activate-Status: Ready


ucs-fi-6248up-0-A /chassis/server # show firmware boardcontroller

Management Controller:

Server  Running-Vers    Package-Vers    Activate-Status
------- --------------- --------------- ---------------
1/1     15.0                            Ready
ucs-fi-6248up-0-A /chassis/server # top
ucs-fi-6248up-0-A# show version brief
System version: 2.2(8m)
Package-Vers: 2.2(8m)A
ucs-fi-6248up-0-A#

So that is odd. The requirements listed in Cisco's documentation for supporting Xeon E5-26xx v2 CPUs are met. But the blade complains about "Board Programmable version not valid for Processor Type". Maybe the blade server's CIMC could tell us more?