Mellanox SX6036
Console access
Serial port
There is an 8P8C modular (RJ45) connector on the right side of the port side of the switch. This connector uses the Cisco pinout and a common Cisco console cable works great for driving it.
Serial settings are 9600 bits per second, 8 data bits, no parity bits, 1 stop bit (9600 8N1)
Default credentials
User admin.
Password admin.
Resetting the password
In the lower left corner of the port side of the switch, there is a small hole labelled "RST". Holding the button inside with a paperclip or similar tool for 15 seconds will reset the switch and erase any password set for the admin user.
Firmware updates
Updating firmware on these switches is a protracted process. Firmware images are available on the public internet, but some searching is required. To bring a switch from a 3.2.x release up to the final 3.6.8012 release, locate the following images on the web and save them to an SSHable or HTTP(S)able location near you:
- image-PPC_M460EX-3.4.3002.img
- image-PPC_M460EX-3.5.1006.img
- image-PPC_M460EX-3.6.1002.img
- image-PPC_M460EX-3.6.8012.img
- image-PPC_M460EX-SX_3.3.3400.img
- image-PPC_M460EX-SX_3.3.5006.img
- image-PPC_M460EX-SX_3.4.0012.img
New firmware image flashing procedure
This is a privileged operation, so first off gain elevated privileges...
enable
image fetch http://172.17.0.17/~adj/image-PPC_M460EX-SX_3.3.3400.img
image install image-PPC_M460EX-SX_3.3.3400.img location 2 progress track verify require-sig
reload
Wait for U-Boot to prompt about a menu, and send a Ctrl-B before the countdown reaches 0. Select option 2 (no need for a newline) to boot the newly flashed image. Iterate through each of the newer images, alternating location 1 and location 2 in the image install ... command.
When updates version installations are finally complete, it might be good to ensure that 3.6.8012 is flashed to both locations 1 and 2.
Useful CLI commands
Find MAC address of 1000baseT management interface 0
Good for setting up the DHCP server's config for this switch. For some reason, this command needs elevated privileges. Grrr.
enable
show interfaces mgmt0 | include HW
Find system serial number without reading the physical tag
This is useful for license key generation. And inventory purposes.
show inventory
Find system host ID
Also can be used for license key generation. Looks suspiciously like the mgmt0 MAC address...
show version | include Host
Licensing features
Mellanox's SwitchX2 ASIC is capable of performing Infiniband and Ethernet switching. Mellanox the company likes to lock some of these features behind licensing keys. Which is annoying. But with enough effort, is a surmountable obstacle
The license generation tool
Inside the MLNX-OS distribution images (those files with names like image-PPC_M460EX-3.6.8012.img) there exists a gzipped tarball containing the MLNX-OS filesystem. Inside the tarball is a directory called /opt/tms/bin. And inside that directory, one can find the license key generation program, genlicense. The MLNX-OS filesystem image is a PowerPC Linux distribution that can be run from a Qemu emulated big-endian PowerPC machine, or natively on big-endian PowerPC hardware. We can use this to generate feature licenses until the cows come home.
Our first license, shell access from the switch CLI
There is a hidden CLI command, _shell which will cause the CLI to execve("/bin/bash", ...). This command is privileged (must be run from enable mode) and locked behind a license key. The following is a working CLI session to install a key for this feature:
Mellanox MLNX-OS Switch Management mellanox-sx6036-rack-1 login: admin Password: passwordgoesherebutisnotechoed Last login: Sun Feb 19 23:25:30 on ttyS0 Mellanox Switch mellanox-sx6036-rack-1 [standalone: master] > enable mellanox-sx6036-rack-1 [standalone: master] # configure terminal mellanox-sx6036-rack-1 [standalone: master] (config) # license install LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1 mellanox-sx6036-rack-1 [standalone: master] (config) # exit mellanox-sx6036-rack-1 [standalone: master] # show licenses License 1: LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1 Feature: RESTRICTED_CMDS_GEN2 Description: Access to restricted system functionality Valid: yes Active: yes mellanox-sx6036-rack-1 [standalone: master] #
Ethernet features: Ethernet, Ethernet L2, and Ethernet L3
We will generate these from the switch as we already have shell access and can use the included genlicense tool.
Running the following from the switch's shell will print a key for a non-hardware-locked, non-date-locked license to activate the switch's full Ethernet capabilities:
[admin@mellanox-sx6036-rack-0 ~]# cd /opt/tms/bin [admin@mellanox-sx6036-rack-0 bin]# ./genlicense 2 EFM_SX m2l0n%0x9 -o 53 true -o 51 true -o 58 true LK2-EFM_SX-5M11-5K11-5T11-88A1-BBD0-JP82-X [admin@mellanox-sx6036-rack-0 bin]#
A quick explanatory diversion is in order here. There are 2 types of keys. genlicense's first option above specifies key type 2. EFM_SX is the Generic SX license type. By itself it does nothing, but all of the -o flags specify further features to be activated. A complete list of these can be generated by running /opt/tms/bin/genlicense 2. There are 14 of these additional options, and their meaning is not well documented. The magic word m2l0n%0x9 can be found by watching for strlen() calls in the process flow of /opt/tms/bin/dumplicense when run under ltrace.
And it can be installed in the switch's configuration database by leaving the shell (which will disconnect the terminal session), logging in again, running enable to get elevated privileges, and using the license install configuration command like so:
Mellanox MLNX-OS Switch Management mellanox-sx6036-rack-1 login: admin Password: Last login: Sun Feb 19 23:31:51 on ttyS0 Mellanox Switch mellanox-sx6036-rack-1 [standalone: master] > enable mellanox-sx6036-rack-1 [standalone: master] # configure terminal mellanox-sx6036-rack-1 [standalone: master] (config) # license install LK2-EFM_SX-5M11-5K11-5T11-88A1-BBD0-JP82-X License was installed successfully. Please wait 1 minute before further configurations. mellanox-sx6036-rack-1 [standalone: master] (config) # exit mellanox-sx6036-rack-1 [standalone: master] # write memory mellanox-sx6036-rack-1 [standalone: master] #
Switch configuration commands
Some of these things will need licenses first.
I can haz Ethernet?
Install the Ethernet key as shown above and confirm it is working:
mellanox-sx6036-rack-1 [standalone: master] # show system capabilities IB: Supported, L2, Adaptive Routing Ethernet: Supported, L2, L3 GW: Not supported Max SM nodes: 648 IB Max licensed speed: FDR10 Ethernet Max licensed speed: 56Gb mellanox-sx6036-rack-1 [standalone: master] #
It says we can do Ethernet. Which is cool. But it does not have the Infiniband-Ethernet gateway feature licensed. And this is required to enable the switch to run in VPI mode. VPI is "Virtual Protocol Interconnect" or Mellanox's way of saying "We can do Infiniband and Ethernet on a single switch." So another license is needed. Here's a key for that:
mellanox-sx6036-rack-1 [standalone: master] # configure terminal mellanox-sx6036-rack-1 [standalone: master] (config) # license install LK2-EFM_SX-5N21-488A-182A-UQXB-Y6 mellanox-sx6036-rack-1 [standalone: master] (config) # exit mellanox-sx6036-rack-1 [standalone: master] # show system capabilities IB: Supported, L2, Adaptive Routing Ethernet: Supported, L2, L3 GW: Supported Max SM nodes: 648 IB Max licensed speed: FDR10 Ethernet Max licensed speed: 56Gb mellanox-sx6036-rack-1 [standalone: master] #
We will need to do this next thing to convert the switch from an Infiniband-only switch to an Infiniband-and-Ethernet switch:
mellanox-sx6036-rack-1 [standalone: master] # configure terminal mellanox-sx6036-rack-1 [standalone: master] (config) # system profile vpi-single-switch Warning - confirming will cause system reboot and all configuration will be deleted Type 'yes' to confirm profile change: yes
Allow time for the switch to reboot and log in again. When it next starts up, run the command show system profile to confirm that it is in VPI mode:
mellanox-sx6036-rack-1 [standalone: master] > show system profile Profile: vpi-single-switch mellanox-sx6036-rack-1 [standalone: master] >
And finally, let's set some ports to Ethernet mode:
mellanox-sx6036-rack-1 [standalone: master] > show ports type InfiniBand: 1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10 1/11 1/12 1/13 1/14 1/15 1/16 1/17 1/18 1/19 1/20 1/21 1/22 1/23 1/24 1/25 1/26 1/27 1/28 1/29 1/30 1/31 1/32 1/33 1/34 1/35 1/36 mellanox-sx6036-rack-1 [standalone: master] > enable mellanox-sx6036-rack-1 [standalone: master] # configure terminal mellanox-sx6036-rack-1 [standalone: master] (config) # interface ib 1/19-1/36 mellanox-sx6036-rack-1 [standalone: master] (config interface ib 1/19-1/36) # shutdown mellanox-sx6036-rack-1 [standalone: master] (config interface ib 1/19-1/36) # exit mellanox-sx6036-rack-1 [standalone: master] (config) # port 1/19-1/36 type ethernet mellanox-sx6036-rack-1 [standalone: master] (config) # interface ethernet 1/19-1/36 mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/19-1/36) # no shutdown mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/19-1/36) # exit mellanox-sx6036-rack-1 [standalone: master] (config) # exit mellanox-sx6036-rack-1 [standalone: master] # write memory mellanox-sx6036-rack-1 [standalone: master] # show ports type Ethernet: 1/19 1/20 1/21 1/22 1/23 1/24 1/25 1/26 1/27 1/28 1/29 1/30 1/31 1/32 1/33 1/34 1/35 1/36 InfiniBand: 1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10 1/11 1/12 1/13 1/14 1/15 1/16 1/17 1/18 mellanox-sx6036-rack-1 [standalone: master] #
Switch management items
Password for the admin user
Factory defaults are probably not a good idea, so let's tighten things up a bit.
mellanox-sx6036-rack-1 [standalone: master] > enable mellanox-sx6036-rack-1 [standalone: master] # configure terminal mellanox-sx6036-rack-1 [standalone: master] (config) # username admin password the_password_goes_here mellanox-sx6036-rack-1 [standalone: master] (config) # exit mellanox-sx6036-rack-1 [standalone: master] # write memory mellanox-sx6036-rack-1 [standalone: master] #
IP addressing for the management interface
The following will configure the switch to retrieve its IP address, subnet mask, router IP address, name server info, and hostname from DHCP server on the management interface's local network.
mellanox-sx6036-rack-1 [standalone: master] > enable mellanox-sx6036-rack-1 [standalone: master] # configure terminal mellanox-sx6036-rack-1 [standalone: master] (config) # interface mgmt0 mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # dhcp mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # dhcp hostname mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # exit mellanox-sx6036-rack-1 [standalone: master] (config) # exit mellanox-sx6036-rack-1 [standalone: master] # write memory mellanox-sx6036-rack-1 [standalone: master] #
NTP
Let's add a couple of NTP servers to the config and discipline our poorly behaved battery backed clock:
mellanox-sx6036-rack-1 [standalone: master] > enable
mellanox-sx6036-rack-1 [standalone: master] # configure terminal
mellanox-sx6036-rack-1 [standalone: master] (config) # ntp server 172.16.10.2
mellanox-sx6036-rack-1 [standalone: master] (config) # ntp server 172.17.0.17
mellanox-sx6036-rack-1 [standalone: master] (config) # exit
mellanox-sx6036-rack-1 [standalone: master] # write memory
mellanox-sx6036-rack-1 [standalone: master] # show ntp
NTP is administratively : enabled
NTP Authentication administratively: disabled
Clock is synchronized:
Reference: 172.16.10.2
Offset : -0.006 ms
Active servers and peers:
172.16.10.2:
Conf Type : serv
Status : sys.peer(*)
Stratum : 3
Offset(msec) : -0.006
Ref clock : 45.45.184.14
Poll Interval (sec): 64
Last Response (sec): 8
Auth state : none
172.17.0.17:
Conf Type : serv
Status : pending
Stratum : 2
Offset(msec) : 0.429
Ref clock : 198.60.22.240
Poll Interval (sec): 64
Last Response (sec): 7
Auth state : none
mellanox-sx6036-rack-1 [standalone: master] #
Ethernet goodies
LLDP
Turn it on and test it out like so:
mellanox-sx6036-rack-1 [standalone: master] > enable
mellanox-sx6036-rack-1 [standalone: master] # configure terminal
mellanox-sx6036-rack-1 [standalone: master] (config) # lldp
mellanox-sx6036-rack-1 [standalone: master] (config) # exit
mellanox-sx6036-rack-1 [standalone: master] # write memory
mellanox-sx6036-rack-1 [standalone: master] # show lldp local
LLDP: enabled
Local global configuration
Chassis sub type: Mac Address
Chassis id: 00:02:c9:6d:4d:b0
System Name: mellanox-sx6036-rack-1
System Description: SX6036,MLNX-OS,SWv3.6.8012
Supported capabilities: B,R
Supported capabilities enabled: B
mellanox-sx6036-rack-1 [standalone: master] # show lldp interfaces ethernet 1/36
TLV flags
PD: port-description
SN: sys-name
SD: sys-description
SC: sys-capabilities
MA: management-address
ETS-C: ETS-Configuration
ETS-R: ETS-Recommendation
AP: Application Priority
PFC: Priority Flow Control
CEE: Converged Enhanced Ethernet DCBX version
MED-CAP: Media Capabilities
MED-NWP: MED-Network Policy
-----------------------------------------------------------------------------------
Interface Receive Transmit TLVs
-----------------------------------------------------------------------------------
Eth1/36 Enabled Enabled PD, SN, SD, SC, PFC, AP, ETS-C, ETS-R
mellanox-sx6036-rack-1 [standalone: master] # show lldp interfaces ethernet 1/36 remote
Eth1/36
Remote Index: 1
Remote chassis id: cc:4e:24:83:4b:36 ; chassis id subtype: Mac Address (4)
Remote port-id: cc:4e:24:83:4b:6c ; port id subtype: Mac Address (3)
Remote port description: 40GigabitEthernet1/2/6
Remote system name: brocade-icx6610-48p-rack-1
Remote system description: Not Advertised
Remote system capabilities supported: B,R ; enabled B,R
Remote Management Addresses:
SubType Address
------------------------------------
IPv4 172.16.10.17
No Remote PFC entry
No Remote ETS entry
No Remote Application Priority entry
mellanox-sx6036-rack-1 [standalone: master] #
And enjoy getting to know all the things about your neighbors.
Multiple Spanning Tree Protocol
Wherein we avoid broadcast loops. Because they are bad, even if they do make for lots of blinken lights. Our Brocade switches are running MSTP and, fortunately, MLNX-OS supports that on our SX6036es as well. Setting the switch to MSTP mode is pretty simple:
mellanox-sx6036-rack-1 [standalone: master] > enable
mellanox-sx6036-rack-1 [standalone: master] # configure terminal
mellanox-sx6036-rack-1 [standalone: master] (config) # spanning-tree mode mst
mellanox-sx6036-rack-1 [standalone: master] (config) # exit
mellanox-sx6036-rack-1 [standalone: master] # write memory
mellanox-sx6036-rack-1 [standalone: master] # show spanning-tree
Switch : ethernet-default
Spanning tree protocol mst : enabled
Spanning tree force version: 3
Root ID:
Priority : 32768
Address : 00:02:c9:6d:4c:b0
Cost : 0
Port : Eth1/36
Hello Time (sec) : 2
Max Age (sec) : 20
Forward Delay (sec): 15
MST00:
Bridge is executing the mstp compatible Spanning Tree Protocol
Bridge ID:
Priority : 32768
Address : 00:02:c9:6d:4d:b0
Hello Time (sec) : 2
Max Age (sec) : 20
Forward Delay (sec): 15
L: Loop Inconsistent
R: Root Inconsistent
G: BPDU Guard Inconsistent
--------------------------------------------------------------------------
Interface Role Sts Cost Prio Type
--------------------------------------------------------------------------
Eth1/36 Root Forwarding 500 128 normal
mellanox-sx6036-rack-1 [standalone: master] #
VLANs
Create our standard assortment of VLANs like so:
mellanox-sx6036-rack-1 [standalone: master] > enable
mellanox-sx6036-rack-1 [standalone: master] # configure terminal
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 10 name Management
mellanox-sx6036-rack-1 [standalone: master] (config vlan 10) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 100 name "WAN 0 (Comcast 44th Avenue)"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 100) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 101 name "WAN 1 (Allo)"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 101) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 102 name "WAN 2 (unused)"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 102) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 103 name "WAN 3 (unused)"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 103) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 104 name "WAN 4 (unused)"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 104) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 105 name "WAN 5 (unused)"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 105) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 106 name "WAN 6 (unused)"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 106) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 107 name "WAN 7 (unused)"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 107) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 108 name "WAN 8 (unused)"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 108) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 109 name "WAN 9 (unused)"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 109) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 1000 name "Internal Services"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1000) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 1001 name "Internal clients"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1001) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 1002 name "Ceph front-end"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1002) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 1003 name "Ceph back-end"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 1003) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # vlan 3900 name "Guest network"
mellanox-sx6036-rack-1 [standalone: master] (config vlan 3900) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # exit
mellanox-sx6036-rack-1 [standalone: master] # write memory
mellanox-sx6036-rack-1 [standalone: master] # show vlan
----------------------------------------------------------------------
VLAN Name Ports
----------------------------------------------------------------------
1 default Eth1/19, Eth1/20, Eth1/21, Eth1/22, Eth1/23,
Eth1/24, Eth1/25, Eth1/26, Eth1/27, Eth1/28,
Eth1/29, Eth1/30, Eth1/31, Eth1/32, Eth1/33,
Eth1/34, Eth1/35, Eth1/36
10 Management
100 WAN 0 (Comcast 44th Avenue)
101 WAN 1 (Allo)
102 WAN 2 (unused)
103 WAN 3 (unused)
104 WAN 4 (unused)
105 WAN 5 (unused)
106 WAN 6 (unused)
107 WAN 7 (unused)
108 WAN 8 (unused)
109 WAN 9 (unused)
1000 Internal Services
1001 Internal clients
1002 Ceph front-end
1003 Ceph back-end
3900 Guest network
mellanox-sx6036-rack-1 [standalone: master] #
And now, to assign all those VLANs as tagged on port 1/36:
mellanox-sx6036-rack-1 [standalone: master] > enable
mellanox-sx6036-rack-1 [standalone: master] # configure terminal
mellanox-sx6036-rack-1 [standalone: master] (config) # interface ethernet 1/36
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # switchport mode trunk
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # switchport trunk allowed-vlan 10
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # switchport trunk allowed-vlan add 100-109
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # switchport trunk allowed-vlan add 1000-1003
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # switchport trunk allowed-vlan add 3900
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/36) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # exit
mellanox-sx6036-rack-1 [standalone: master] # write memory
mellanox-sx6036-rack-1 [standalone: master] # show vlan
----------------------------------------------------------------------
VLAN Name Ports
----------------------------------------------------------------------
1 default Eth1/19, Eth1/20, Eth1/21, Eth1/22, Eth1/23,
Eth1/24, Eth1/25, Eth1/26, Eth1/27, Eth1/28,
Eth1/29, Eth1/30, Eth1/31, Eth1/32, Eth1/33,
Eth1/34, Eth1/35
10 Management Eth1/36
100 WAN 0 (Comcast 44th Avenue)Eth1/36
101 WAN 1 (Allo) Eth1/36
102 WAN 2 (unused) Eth1/36
103 WAN 3 (unused) Eth1/36
104 WAN 4 (unused) Eth1/36
105 WAN 5 (unused) Eth1/36
106 WAN 6 (unused) Eth1/36
107 WAN 7 (unused) Eth1/36
108 WAN 8 (unused) Eth1/36
109 WAN 9 (unused) Eth1/36
1000 Internal Services Eth1/36
1001 Internal clients Eth1/36
1002 Ceph front-end Eth1/36
1003 Ceph back-end Eth1/36
3900 Guest network Eth1/36
mellanox-sx6036-rack-1 [standalone: master] #