Mellanox Infiniband HCA and/or Ethernet adapters - Revision history

Adj: /* Set card mode */

2026-04-13T21:35:30Z

Set card mode

← Older revision		Revision as of 21:35, 13 April 2026
Line 285:		Line 285:
	-------------------------------------------------------------------------------		-------------------------------------------------------------------------------
	itops@syadasti:~$		itops@syadasti:~$
			== Now what? ==
			Now, it is an Ethernet card. Treat it like any other. Make bond interfaces. And VLANed sub-interfaces. Copy bits across it. Have fun!

Adj: /* Set card mode */

2026-04-13T21:32:50Z

Set card mode

@@ Line 260: / Line 260: @@
      link/ether f4:52:14:2c:5e:02 brd ff:ff:ff:ff:ff:ff
      altname enxf452142c5e02
  itops@syadasti:~$

Adj: /* Set card mode */

2026-04-13T21:23:49Z

Set card mode

← Older revision		Revision as of 21:23, 13 April 2026
Line 184:		Line 184:

	=== Set card mode ===		=== Set card mode ===
			This server has a 2 port card installed. And that card only has one Ethernet port right now. At last as listed by <code>ip link list</code>. This seems to be a Linux kernel driver thing more than a hardware configuration thing, though:
			itops@syadasti:~$ '''sudo mstconfig -d d8:00.0 query'''

			Device #1:
			----------

			Device type: ConnectX3Pro
			Device: d8:00.0

			Configurations: Next Boot
			SRIOV_EN True(1)
			NUM_OF_VFS 8
			LINK_TYPE_P1 VPI(3)
			LINK_TYPE_P2 VPI(3)
			LOG_BAR_SIZE 3
			BOOT_PKEY_P1 0
			BOOT_PKEY_P2 0
			BOOT_OPTION_ROM_EN_P1 True(1)
			BOOT_VLAN_EN_P1 False(0)
			BOOT_RETRY_CNT_P1 0
			LEGACY_BOOT_PROTOCOL_P1 PXE(1)
			BOOT_VLAN_P1 1
			BOOT_OPTION_ROM_EN_P2 True(1)
			BOOT_VLAN_EN_P2 False(0)
			BOOT_RETRY_CNT_P2 0
			LEGACY_BOOT_PROTOCOL_P2 PXE(1)
			BOOT_VLAN_P2 1
			IP_VER_P1 IPv4(0)
			IP_VER_P2 IPv4(0)
			CQ_TIMESTAMP True(1)
			itops@syadasti:~$
			Note that both LINK_TYPEs are "VPI" which is Mellanox speak for "I'm bilingual. I will speak both Infiniband '''and''' Ethernet." But there's only one. Let's set them both to Ethernet mode, reload the kernel drivers, and see what happens:
			itops@syadasti:~$ '''sudo mstconfig -d d8:00.0 --enable_verbosity set LINK_TYPE_P1=2 LINK_TYPE_P2=2'''

			Device #1:
			----------

			Device type: ConnectX3Pro
			Device: d8:00.0

			Configurations: Default Next Boot New
			LINK_TYPE_P1 VPI(3) VPI(3) ETH(2)
			LINK_TYPE_P2 VPI(3) VPI(3) ETH(2)
			Device Firmware does not support reading current configurations

			Apply new Configuration? (y/n) [n] : '''y'''
			Applying... Done!
			-I- Please reboot machine to load new configurations.
			itops@syadasti:~$ '''sudo modprobe -rv mlx4_ib mlx4_en mlx4_core'''
			rmmod mlx4_ib
			rmmod ib_uverbs
			rmmod mlx4_en
			rmmod mlx4_core
			itops@syadasti:~$ '''sudo modprobe -v mlx4_core'''
			insmod /lib/modules/6.12.73+deb13-amd64/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko.xz
			itops@syadasti:~$
			And now? Well, we can see two Ethernet adapters in that PCI slot. And their MAC addresses match what was set in the firmware flashing operation above, too:
			itops@syadasti:~$ '''ip link list \| grep ^[0-9]'''
			1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
			2: eno5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
			3: eno6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
			4: ens1f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
			5: ens1f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
			6: ens1f2np2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
			7: ens1f3np3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
			8: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
			9: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
			12: enp216s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
			13: enp216s0d1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
			itops@syadasti:~$ '''ip link list dev enp216s0; ip link list dev enp216s0d1'''
			12: enp216s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
			link/ether f4:52:14:2c:5e:01 brd ff:ff:ff:ff:ff:ff
			altname enxf452142c5e01
			13: enp216s0d1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
			link/ether f4:52:14:2c:5e:02 brd ff:ff:ff:ff:ff:ff
			altname enxf452142c5e02
			itops@syadasti:~$

Adj: /* Burning new firmwares */

2026-04-13T20:50:53Z

Burning new firmwares

← Older revision		Revision as of 20:50, 13 April 2026
Line 59:		Line 59:

	=== Burning new firmwares ===		=== Burning new firmwares ===
			Unpack the just-downloaded ZIP file and find a .bin file inside. And compare what is currently on the card with what is in the .bin file:
			itops@syadasti:~/Mlnx-HCA-Firmware$ '''diff -u --color <(sudo mstflint -d d8:00.0 q full) <(mstflint -i fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin q full) '''
			--- /dev/fd/63 2026-04-13 19:15:20.421912090 +0000
			+++ /dev/fd/62 2026-04-13 19:15:20.421912090 +0000
			@@ -1,13 +1,14 @@
			Image type: FS2
			-FW Version: 2.36.5000
			-FW Release Date: 26.1.2016
			-MIC Version: 1.5.0
			+FW Version: 2.42.5000
			+FW Release Date: 5.9.2017
			+MIC Version: 2.0.0
			Config Sectors: 2
			-Product Version: 02.36.50.00
			-Rom Info: type=PXE version=3.4.718
			+PRS Name: cx3pro_MCX354A_fdr_09v.prs
			+Product Version: 02.42.50.00
			+Rom Info: type=PXE version=3.4.752
			Device ID: 4103
			Description: Node Port1 Port2 Sys image
			-GUIDs: f4521403002c5e00 f4521403002c5e01 f4521403002c5e02 f4521403002c5e03
			-MACs: f452142c5e01 f452142c5e02
			-VSD:
			+GUIDs: 0002c9000100d050 0002c9000100d051 0002c9000100d052 0002c9000100d050
			+MACs: 0002c9000001 0002c9000002
			+VSD: n/a
			PSID: MT_1090111019
			itops@syadasti:~/Mlnx-HCA-Firmware$
			Along with the firmware version numbers, we can see that the Infiniband GUIDs and Ethernet MAC addresses are different. It would probably be good to keep those even after the new firmware image is flashed. Note that the PSID stays the same. I believe it is possible to change an adapter card from a ConnectX-3 to a ConnectX-3 Pro by flashing different images into it. But we do not need to do that here.

			Now, let's make a backup of what is in the card:
			itops@syadasti:~/Mlnx-HCA-Firmware$ '''sudo mstflint -d d8:00.0 ri backup_$(date -Iseconds)_2.36.5000.bin'''
			itops@syadasti:~/Mlnx-HCA-Firmware$ '''mstflint -i backup_2026-04-13T19\:58\:45+00\:00_2.36.5000.bin q full'''
			Image type: FS2
			FW Version: 2.36.5000
			FW Release Date: 26.1.2016
			MIC Version: 1.5.0
			Config Sectors: 2
			Product Version: 02.36.50.00
			Rom Info: type=PXE version=3.4.718
			Device ID: 4103
			Description: Node Port1 Port2 Sys image
			GUIDs: f4521403002c5e00 f4521403002c5e01 f4521403002c5e02 f4521403002c5e03
			MACs: f452142c5e01 f452142c5e02
			VSD:
			PSID: MT_1090111019
			itops@syadasti:~/Mlnx-HCA-Firmware$

			Remove drivers from the running kernel and flash the new firmware image:
			itops@syadasti:~/Mlnx-HCA-Firmware$ '''lsmod \| grep mlx'''
			mlx4_ib 262144 0
			ib_uverbs 204800 1 mlx4_ib
			mlx4_en 167936 0
			mlx4_core 442368 2 mlx4_ib,mlx4_en
			ib_core 524288 2 mlx4_ib,ib_uverbs
			itops@syadasti:~/Mlnx-HCA-Firmware$ '''sudo modprobe -rv mlx4_ib mlx4_en mlx4_core'''
			rmmod mlx4_ib
			rmmod ib_uverbs
			rmmod mlx4_en
			rmmod mlx4_core
			itops@syadasti:~/Mlnx-HCA-Firmware$ '''sudo mstflint -d d8:00.0 --guids f4521403002c5e00,f4521403002c5e01,f4521403002c5e02,f4521403002c5e03 --macs f452142c5e01,f452142c5e02 --image fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin burn'''
			You are about to change the Guids/Macs/Uids on the device:

			New Values Current Values
			Node GUID: f4521403002c5e00 f4521403002c5e00
			Port1 GUID: f4521403002c5e01 f4521403002c5e01
			Port2 GUID: f4521403002c5e02 f4521403002c5e02
			Sys.Image GUID: f4521403002c5e03 f4521403002c5e03
			Port1 MAC: f452142c5e01 f452142c5e01
			Port2 MAC: f452142c5e02 f452142c5e02

			Do you want to continue ? (y/n) [n] : '''y'''

			Current FW version on flash: 2.36.5000
			New FW version: 2.42.5000

			Burning FS2 FW image without signatures - OK
			Restoring signature - OK
			itops@syadasti:~/Mlnx-HCA-Firmware$
			And query the card to check on the flashing:
			itops@syadasti:~/Mlnx-HCA-Firmware$ '''sudo mstflint -d d8:00.0 q full'''
			Image type: FS2
			FW Version: 2.42.5000
			FW Version(Running): 2.36.5000
			FW Release Date: 5.9.2017
			MIC Version: 2.0.0
			Config Sectors: 2
			PRS Name: cx3pro_MCX354A_fdr_09v.prs
			Product Version: 02.42.50.00
			Rom Info: type=PXE version=3.4.752
			Device ID: 4103
			Description: Node Port1 Port2 Sys image
			GUIDs: f4521403002c5e00 f4521403002c5e01 f4521403002c5e02 f4521403002c5e03
			MACs: f452142c5e01 f452142c5e02
			VSD:
			PSID: MT_1090111019
			itops@syadasti:~/Mlnx-HCA-Firmware$
			New code isn't running right now. But there is an "image reactivate" operation that could have been performed along with the flashing. Let's give that a go here:
			itops@syadasti:~/Mlnx-HCA-Firmware$ '''sudo mstflint -d d8:00.0 ir'''
			-E- Failed to execute image reactivation on device d8:00.0. Error: Operation not supported..
			itops@syadasti:~/Mlnx-HCA-Firmware$
			Hmmm. A different tool, maybe?
			itops@syadasti:~/Mlnx-HCA-Firmware$ '''sudo mstfwreset -d d8:00.0 query'''
			-E- Unsupported Device: d8:00.0 (ConnectX3Pro).
			itops@syadasti:~/Mlnx-HCA-Firmware$
			Fine, be that way. Halt the OS and power cycle the machine. Grrrr. Fast forward a bit (power cycle, OS start done) and we can check again:
			itops@syadasti:~$ '''sudo mstflint -d d8:00.0 query full'''
			Image type: FS2
			FW Version: 2.42.5000
			FW Release Date: 5.9.2017
			MIC Version: 2.0.0
			Config Sectors: 2
			PRS Name: cx3pro_MCX354A_fdr_09v.prs
			Product Version: 02.42.50.00
			Rom Info: type=PXE version=3.4.752
			Device ID: 4103
			Description: Node Port1 Port2 Sys image
			GUIDs: f4521403002c5e00 f4521403002c5e01 f4521403002c5e02 f4521403002c5e03
			MACs: f452142c5e01 f452142c5e02
			VSD:
			PSID: MT_1090111019
			itops@syadasti:~$
			So that is looking correct. Yay!

	=== Set card mode ===		=== Set card mode ===

Adj: /* Where to get firmware images */

2026-04-13T19:13:18Z

Where to get firmware images

← Older revision		Revision as of 19:13, 13 April 2026
Line 54:		Line 54:

	=== Where to get firmware images ===		=== Where to get firmware images ===
			See that PSID in the output from the <code>mstflint</code> query operation? We are running relase 2.36.5000 of the code in the ConnectX-3 chip on this card. There are also code blobs for BIOS/UEFI that can be installed.

			nVidia bought Mellanox some years ago. With that purchase, firmware images for switches and adapter cards can be found at https://network.nvidia.com/support/firmware/firmware-downloads/. If it is not there in the future, try Duck-Duck-Going for something like "nVidia adapter card firmware download" until you find the right thing. Today's card is a "ConnectX-3 Pro" (as seen in the output of <code>lspci -d 15b3:</code>). These cards came in Infiniband only and Ethernet+Infiniband (and maybe just Ethernet, too?) varieties. Navigating through nVidia firmware download, I found a card with a matching PSID under the ConnectX-3 Pro Infiniband link. PSID MT_1090111019 looks to go with a Mellanox MCX354A-FCCT (dual port) or MCX353A-FCCT (single port) adapter. Download the corresponding ZIP archive from nVidia's site. As this is written, that is https://www.mellanox.com/downloads/firmware/fw-ConnectX3Pro-rel-2_42_5000-MCX354A-FCC_Ax-FlexBoot-3.4.752.bin.zip.

	=== Burning new firmwares ===		=== Burning new firmwares ===
	=== Set card mode ===		=== Set card mode ===

Adj: /* ConnectX-3 cards */

2026-04-13T18:42:18Z

ConnectX-3 cards

@@ Line 2: / Line 2: @@
 == ConnectX-3 cards ==
 === Finding card info ===
 === Where to get firmware images ===
 === Burning new firmwares ===

Adj: Created page with "I have a number of servers with ConnectX-3 cards to connect to various network infrastructure bits and bobs. Recently (today) installed on in a Cisco UCS C220 M5 server and realized that I have made no notes about things that have been done to make the cards work. Ooops. == ConnectX-3 cards == === Finding card info === === Where to get firmware images === === Burning new firmwares === === Set card mode ==="

2026-04-13T03:15:41Z

Created page with "I have a number of servers with ConnectX-3 cards to connect to various network infrastructure bits and bobs. Recently (today) installed on in a Cisco UCS C220 M5 server and realized that I have made no notes about things that have been done to make the cards work. Ooops. == ConnectX-3 cards == === Finding card info === === Where to get firmware images === === Burning new firmwares === === Set card mode ==="

New page

I have a number of servers with ConnectX-3 cards to connect to various network infrastructure bits and bobs. Recently (today) installed on in a Cisco UCS C220 M5 server and realized that I have made no notes about things that have been done to make the cards work. Ooops.

== ConnectX-3 cards ==

=== Finding card info ===
=== Where to get firmware images ===
=== Burning new firmwares ===
=== Set card mode ===