Mellanox Infiniband HCA and/or Ethernet adapters

I have a number of servers with ConnectX-3 cards to connect to various network infrastructure bits and bobs. Recently (today) installed on in a Cisco UCS C220 M5 server and realized that I have made no notes about things that have been done to make the cards work. Ooops.

ConnectX-3 cards

Several of these are deployed around the home network. There are various models (pretty much all are ConnectX-3, though.)

Finding card info

List out all Mellanox cards in a machine. "15b3" is Mellanox's PCI vendor ID.

itops@syadasti:~$ lspci -d 15b3:
d8:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
itops@syadasti:~$ 

First column of output is the PCI bus ID of the card. This is used in the next bunch of commands. Running with elevated privileges (under sudo) allows reading to the capabilities info. That isn't really necessary for this step, but causes no harm.

itops@syadasti:~$ sudo lspci -s d8:00.0 -vv
d8:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
        Subsystem: Mellanox Technologies Device 0003
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 83
        NUMA node: 1
        IOMMU group: 20
        Region 0: Memory at fbe00000 (64-bit, non-prefetchable) [size=1M]
        Region 2: Memory at 4ffff800000 (64-bit, prefetchable) [size=8M]
        Expansion ROM at fbd00000 [disabled] [size=1M]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] Vital Product Data
        Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
        Capabilities: [60] Express Endpoint, IntMsgNum 0
        Capabilities: [c0] Vendor Specific Information: Len=18 <?>
        Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [148] Device Serial Number f4-52-14-03-00-2c-5e-00
        Capabilities: [108] Single Root I/O Virtualization (SR-IOV)
        Capabilities: [154] Advanced Error Reporting
        Capabilities: [18c] Secondary PCI Express
        Kernel driver in use: mlx4_core
        Kernel modules: mlx4_core

itops@syadasti:~$ 

Query the card to find out its code revision, Ethernet MAC address(es), Infiniband GUID(s), and the Mellanox Parameter Set ID (PSID). This does need the elevated privileges provided by sudo:

itops@syadasti:~$ sudo mstflint -d d8:00.0 q full
Image type:            FS2
FW Version:            2.36.5000
FW Release Date:       26.1.2016
MIC Version:           1.5.0
Config Sectors:        2
Product Version:       02.36.50.00
Rom Info:              type=PXE version=3.4.718
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 f4521403002c5e00 f4521403002c5e01 f4521403002c5e02 f4521403002c5e03 
MACs:                                       f452142c5e01     f452142c5e02
VSD:                   
PSID:                  MT_1090111019
itops@syadasti:~$ 

Where to get firmware images

See that PSID in the output from the mstflint query operation? We are running relase 2.36.5000 of the code in the ConnectX-3 chip on this card. There are also code blobs for BIOS/UEFI that can be installed.

nVidia bought Mellanox some years ago. With that purchase, firmware images for switches and adapter cards can be found at https://network.nvidia.com/support/firmware/firmware-downloads/. If it is not there in the future, try Duck-Duck-Going for something like "nVidia adapter card firmware download" until you find the right thing. Today's card is a "ConnectX-3 Pro" (as seen in the output of lspci -d 15b3:). These cards came in Infiniband only and Ethernet+Infiniband (and maybe just Ethernet, too?) varieties. Navigating through nVidia firmware download, I found a card with a matching PSID under the ConnectX-3 Pro Infiniband link. PSID MT_1090111019 looks to go with a Mellanox MCX354A-FCCT (dual port) or MCX353A-FCCT (single port) adapter. Download the corresponding ZIP archive from nVidia's site. As this is written, that is https://www.mellanox.com/downloads/firmware/fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin.zip.

Burning new firmwares

Unpack the just-downloaded ZIP file and find a .bin file inside. And compare what is currently on the card with what is in the .bin file:

itops@syadasti:~/Mlnx-HCA-Firmware$ diff -u --color <(sudo mstflint -d d8:00.0 q full) <(mstflint -i fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin q full) 
--- /dev/fd/63	2026-04-13 19:15:20.421912090 +0000
+++ /dev/fd/62	2026-04-13 19:15:20.421912090 +0000
@@ -1,13 +1,14 @@
 Image type:            FS2
-FW Version:            2.36.5000
-FW Release Date:       26.1.2016
-MIC Version:           1.5.0
+FW Version:            2.42.5000
+FW Release Date:       5.9.2017
+MIC Version:           2.0.0
 Config Sectors:        2
-Product Version:       02.36.50.00
-Rom Info:              type=PXE version=3.4.718
+PRS Name:              cx3pro_MCX354A_fdr_09v.prs
+Product Version:       02.42.50.00
+Rom Info:              type=PXE version=3.4.752
 Device ID:             4103
 Description:           Node             Port1            Port2            Sys image
-GUIDs:                 f4521403002c5e00 f4521403002c5e01 f4521403002c5e02 f4521403002c5e03 
-MACs:                                       f452142c5e01     f452142c5e02
-VSD:                   
+GUIDs:                 0002c9000100d050 0002c9000100d051 0002c9000100d052 0002c9000100d050 
+MACs:                                       0002c9000001     0002c9000002
+VSD:                   n/a
 PSID:                  MT_1090111019
itops@syadasti:~/Mlnx-HCA-Firmware$

Along with the firmware version numbers, we can see that the Infiniband GUIDs and Ethernet MAC addresses are different. It would probably be good to keep those even after the new firmware image is flashed. Note that the PSID stays the same. I believe it is possible to change an adapter card from a ConnectX-3 to a ConnectX-3 Pro by flashing different images into it. But we do not need to do that here.

Now, let's make a backup of what is in the card:

itops@syadasti:~/Mlnx-HCA-Firmware$ sudo mstflint -d d8:00.0 ri backup_$(date -Iseconds)_2.36.5000.bin
itops@syadasti:~/Mlnx-HCA-Firmware$ mstflint -i backup_2026-04-13T19\:58\:45+00\:00_2.36.5000.bin q full
Image type:            FS2
FW Version:            2.36.5000
FW Release Date:       26.1.2016
MIC Version:           1.5.0
Config Sectors:        2
Product Version:       02.36.50.00
Rom Info:              type=PXE version=3.4.718
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 f4521403002c5e00 f4521403002c5e01 f4521403002c5e02 f4521403002c5e03 
MACs:                                       f452142c5e01     f452142c5e02
VSD:                   
PSID:                  MT_1090111019
itops@syadasti:~/Mlnx-HCA-Firmware$

Remove drivers from the running kernel and flash the new firmware image:

itops@syadasti:~/Mlnx-HCA-Firmware$ lsmod | grep mlx
mlx4_ib               262144  0
ib_uverbs             204800  1 mlx4_ib
mlx4_en               167936  0
mlx4_core             442368  2 mlx4_ib,mlx4_en
ib_core               524288  2 mlx4_ib,ib_uverbs
itops@syadasti:~/Mlnx-HCA-Firmware$ sudo modprobe -rv mlx4_ib mlx4_en mlx4_core
rmmod mlx4_ib
rmmod ib_uverbs
rmmod mlx4_en
rmmod mlx4_core
itops@syadasti:~/Mlnx-HCA-Firmware$ sudo mstflint -d d8:00.0 --guids f4521403002c5e00,f4521403002c5e01,f4521403002c5e02,f4521403002c5e03 --macs f452142c5e01,f452142c5e02 --image fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin burn
    You are about to change the Guids/Macs/Uids on the device:

                        New Values              Current Values
        Node  GUID:     f4521403002c5e00        f4521403002c5e00
        Port1 GUID:     f4521403002c5e01        f4521403002c5e01
        Port2 GUID:     f4521403002c5e02        f4521403002c5e02
        Sys.Image GUID: f4521403002c5e03        f4521403002c5e03
        Port1 MAC:          f452142c5e01            f452142c5e01
        Port2 MAC:          f452142c5e02            f452142c5e02

 Do you want to continue ? (y/n) [n] : y

    Current FW version on flash:  2.36.5000
    New FW version:               2.42.5000

Burning FS2 FW image without signatures - OK  
Restoring signature                     - OK
itops@syadasti:~/Mlnx-HCA-Firmware$

And query the card to check on the flashing:

itops@syadasti:~/Mlnx-HCA-Firmware$ sudo mstflint -d d8:00.0 q full
Image type:            FS2
FW Version:            2.42.5000
FW Version(Running):   2.36.5000
FW Release Date:       5.9.2017
MIC Version:           2.0.0
Config Sectors:        2
PRS Name:              cx3pro_MCX354A_fdr_09v.prs
Product Version:       02.42.50.00
Rom Info:              type=PXE version=3.4.752
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 f4521403002c5e00 f4521403002c5e01 f4521403002c5e02 f4521403002c5e03 
MACs:                                       f452142c5e01     f452142c5e02
VSD:                   
PSID:                  MT_1090111019
itops@syadasti:~/Mlnx-HCA-Firmware$

New code isn't running right now. But there is an "image reactivate" operation that could have been performed along with the flashing. Let's give that a go here:

itops@syadasti:~/Mlnx-HCA-Firmware$ sudo mstflint -d d8:00.0 ir
-E- Failed to execute image reactivation on device d8:00.0. Error: Operation not supported..
itops@syadasti:~/Mlnx-HCA-Firmware$ 

Hmmm. A different tool, maybe?

itops@syadasti:~/Mlnx-HCA-Firmware$ sudo mstfwreset -d d8:00.0 query
-E- Unsupported Device: d8:00.0 (ConnectX3Pro).
itops@syadasti:~/Mlnx-HCA-Firmware$ 

Fine, be that way. Halt the OS and power cycle the machine. Grrrr. Fast forward a bit (power cycle, OS start done) and we can check again:

itops@syadasti:~$ sudo mstflint -d d8:00.0 query full
Image type:            FS2
FW Version:            2.42.5000
FW Release Date:       5.9.2017
MIC Version:           2.0.0
Config Sectors:        2
PRS Name:              cx3pro_MCX354A_fdr_09v.prs
Product Version:       02.42.50.00
Rom Info:              type=PXE version=3.4.752
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 f4521403002c5e00 f4521403002c5e01 f4521403002c5e02 f4521403002c5e03 
MACs:                                       f452142c5e01     f452142c5e02
VSD:                   
PSID:                  MT_1090111019
itops@syadasti:~$ 

So that is looking correct. Yay!

Set card mode

This server has a 2 port card installed. And that card only has one Ethernet port right now. At last as listed by ip link list. This seems to be a Linux kernel driver thing more than a hardware configuration thing, though:

itops@syadasti:~$ sudo mstconfig -d d8:00.0 query

Device #1:
----------

Device type:    ConnectX3Pro    
Device:         d8:00.0         

Configurations:                              Next Boot
         SRIOV_EN                            True(1)         
         NUM_OF_VFS                          8               
         LINK_TYPE_P1                        VPI(3)          
         LINK_TYPE_P2                        VPI(3)          
         LOG_BAR_SIZE                        3               
         BOOT_PKEY_P1                        0               
         BOOT_PKEY_P2                        0               
         BOOT_OPTION_ROM_EN_P1               True(1)         
         BOOT_VLAN_EN_P1                     False(0)        
         BOOT_RETRY_CNT_P1                   0               
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)          
         BOOT_VLAN_P1                        1               
         BOOT_OPTION_ROM_EN_P2               True(1)         
         BOOT_VLAN_EN_P2                     False(0)        
         BOOT_RETRY_CNT_P2                   0               
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)          
         BOOT_VLAN_P2                        1               
         IP_VER_P1                           IPv4(0)         
         IP_VER_P2                           IPv4(0)         
         CQ_TIMESTAMP                        True(1)         
itops@syadasti:~$

Note that both LINK_TYPEs are "VPI" which is Mellanox speak for "I'm bilingual. I will speak both Infiniband and Ethernet." But there's only one. Let's set them both to Ethernet mode, reload the kernel drivers, and see what happens:

itops@syadasti:~$ sudo mstconfig -d d8:00.0 --enable_verbosity set LINK_TYPE_P1=2 LINK_TYPE_P2=2

Device #1:
----------

Device type:    ConnectX3Pro    
Device:         d8:00.0         

Configurations:                              Default         Next Boot       New
         LINK_TYPE_P1                        VPI(3)          VPI(3)          ETH(2)          
         LINK_TYPE_P2                        VPI(3)          VPI(3)          ETH(2)          
Device Firmware does not support reading current configurations

 Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
itops@syadasti:~$ sudo modprobe -rv mlx4_ib mlx4_en mlx4_core
rmmod mlx4_ib
rmmod ib_uverbs
rmmod mlx4_en
rmmod mlx4_core
itops@syadasti:~$ sudo modprobe -v mlx4_core
insmod /lib/modules/6.12.73+deb13-amd64/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko.xz 
itops@syadasti:~$ 

And now? Well, we can see two Ethernet adapters in that PCI slot. And their MAC addresses match what was set in the firmware flashing operation above, too:

itops@syadasti:~$ ip link list | grep ^[0-9]
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
2: eno5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
3: eno6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
4: ens1f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
5: ens1f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
6: ens1f2np2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
7: ens1f3np3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
8: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
9: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
12: enp216s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
13: enp216s0d1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
itops@syadasti:~$ ip link list dev enp216s0; ip link list dev enp216s0d1
12: enp216s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether f4:52:14:2c:5e:01 brd ff:ff:ff:ff:ff:ff
    altname enxf452142c5e01
13: enp216s0d1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether f4:52:14:2c:5e:02 brd ff:ff:ff:ff:ff:ff
    altname enxf452142c5e02
itops@syadasti:~$

Set link up, see what's on the other end:

itops@syadasti:~$ sudo ip link set up enp216s0
itops@syadasti:~$ sleep 180
itops@syadasti:~$ sudo lldpcli show neighbors ports enp216s0
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
Interface:    enp216s0, via: LLDP, RID: 2, Time: 0 day, 00:05:43
  Chassis:     
    ChassisID:    mac 00:02:c9:6d:4f:30
    SysName:      mellanox-sx6036-rack-2
    SysDescr:     SX6036,MLNX-OS,SWv3.6.8012
    Capability:   Bridge, on
    Capability:   Router, off
  Port:        
    PortID:       ifname Eth1/4
    PortDescr:     
    TTL:          120
  Unknown TLVs:
    TLV:          OUI: 00,80,C2, SubType: 9, Len: 21 04,00,11,22,33,19,19,19,19,00,00,00,00,02,02,02,02,00,00,00,00
    TLV:          OUI: 00,80,C2, SubType: 10, Len: 21 00,00,11,22,33,19,19,19,19,00,00,00,00,02,02,02,02,00,00,00,00
    TLV:          OUI: 00,80,C2, SubType: 12, Len: 1 00
-------------------------------------------------------------------------------
itops@syadasti:~$

Now what?

Now, it is an Ethernet card. Treat it like any other. Make bond interfaces. And VLANed sub-interfaces. Copy bits across it. Have fun!