Mellanox SX6036

Console access

Serial port

There is an 8P8C modular (RJ45) connector on the right side of the port side of the switch. This connector uses the Cisco pinout and a common Cisco console cable works great for driving it.

Serial settings are 9600 bits per second, 8 data bits, no parity bits, 1 stop bit (9600 8N1)

Default credentials

User admin.

Password admin.

Resetting the password

In the lower left corner of the port side of the switch, there is a small hole labelled "RST". Holding the button inside with a paperclip or similar tool for 15 seconds will reset the switch and erase any password set for the admin user.

Firmware updates

Updating firmware on these switches is a protracted process. Firmware images are available on the public internet, but some searching is required. To bring a switch from a 3.2.x release up to the final 3.6.8012 release, locate the following images on the web and save them to an SSHable or HTTP(S)able location near you:

• image-PPC_M460EX-3.4.3002.img
• image-PPC_M460EX-3.5.1006.img
• image-PPC_M460EX-3.6.1002.img
• image-PPC_M460EX-3.6.8012.img
• image-PPC_M460EX-SX_3.3.3400.img
• image-PPC_M460EX-SX_3.3.5006.img
• image-PPC_M460EX-SX_3.4.0012.img

New firmware image flashing procedure

This is a privileged operation, so first off gain elevated privileges...

enable

image fetch http://172.17.0.17/~adj/image-PPC_M460EX-SX_3.3.3400.img

image install image-PPC_M460EX-SX_3.3.3400.img location 2 progress track verify require-sig

reload

Wait for U-Boot to prompt about a menu, and send a Ctrl-B before the countdown reaches 0. Select option 2 (no need for a newline) to boot the newly flashed image. Iterate through each of the newer images, alternating location 1 and location 2 in the image install ... command.

When updates version installations are finally complete, it might be good to ensure that 3.6.8012 is flashed to both locations 1 and 2.

Useful CLI commands

Find MAC address of 1000baseT management interface 0

Good for setting up the DHCP server's config for this switch. For some reason, this command needs elevated privileges. Grrr.

enable

show interfaces mgmt0 | include HW

Find system serial number without reading the physical tag

This is useful for license key generation. And inventory purposes.

show inventory

Find system host ID

Also can be used for license key generation. Looks suspiciously like the mgmt0 MAC address...

show version | include Host

Licensing features

Mellanox's SwitchX2 ASIC is capable of performing Infiniband and Ethernet switching. Mellanox the company likes to lock some of these features behind licensing keys. Which is annoying. But with enough effort, is a surmountable obstacle

The license generation tool

Inside the MLNX-OS distribution images (those files with names like image-PPC_M460EX-3.6.8012.img) there exists a gzipped tarball containing the MLNX-OS filesystem. Inside the tarball is a directory called /opt/tms/bin. And inside that directory, one can find the license key generation program, genlicense. The MLNX-OS filesystem image is a PowerPC Linux distribution that can be run from a Qemu emulated big-endian PowerPC machine, or natively on big-endian PowerPC hardware. We can use this to generate feature licenses until the cows come home.

Our first license, shell access from the switch CLI

There is a hidden CLI command, _shell which will cause the CLI to execve("/bin/bash", ...). This command is privileged (must be run from enable mode) and locked behind a license key. The following is a working CLI session to install a key for this feature:

Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-1 login: admin
Password: passwordgoesherebutisnotechoed
Last login: Sun Feb 19 23:25:30 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-1 [standalone: master] > enable
mellanox-sx6036-rack-1 [standalone: master] # configure terminal 
mellanox-sx6036-rack-1 [standalone: master] (config) # license install LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1
mellanox-sx6036-rack-1 [standalone: master] (config) # exit
mellanox-sx6036-rack-1 [standalone: master] # show licenses
License 1: LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1
   Feature:          RESTRICTED_CMDS_GEN2
   Description:      Access to restricted system functionality
   Valid:            yes
   Active:           yes
mellanox-sx6036-rack-1 [standalone: master] #

Ethernet features: Ethernet, Ethernet L2, and Ethernet L3

We will generate these from the switch as we already have shell access and can use the included genlicense tool.

Running the following from the switch's shell will print a key for a non-hardware-locked, non-date-locked license to activate the switch's full Ethernet capabilities:

[admin@mellanox-sx6036-rack-0 ~]# cd /opt/tms/bin
[admin@mellanox-sx6036-rack-0 bin]# ./genlicense 2 EFM_SX m2l0n%0x9 -o 53 true -o 51 true -o 58 true
LK2-EFM_SX-5M11-5K11-5T11-88A1-BBD0-JP82-X
[admin@mellanox-sx6036-rack-0 bin]#

A quick explanatory diversion is in order here. There are 2 types of keys. genlicense's first option above specifies key type 2. EFM_SX is the Generic SX license type. By itself it does nothing, but all of the -o flags specify further features to be activated. A complete list of these can be generated by running /opt/tms/bin/genlicense 2. There are 14 of these additional options, and their meaning is not well documented. The magic word m2l0n%0x9 can be found by watching for strlen() calls in the process flow of /opt/tms/bin/dumplicense when run under ltrace.

And it can be installed in the switch's configuration database by leaving the shell (which will disconnect the terminal session), logging in again, running enable to get elevated privileges, and using the license install configuration command like so:

Mellanox MLNX-OS Switch Management

mellanox-sx6036-rack-1 login: admin
Password: 
Last login: Sun Feb 19 23:31:51 on ttyS0

Mellanox Switch

mellanox-sx6036-rack-1 [standalone: master] > enable
mellanox-sx6036-rack-1 [standalone: master] # configure terminal
mellanox-sx6036-rack-1 [standalone: master] (config) # license install LK2-EFM_SX-5M11-5K11-5T11-88A1-BBD0-JP82-X
License was installed successfully. Please wait 1 minute before further configurations.
mellanox-sx6036-rack-1 [standalone: master] (config) # exit
mellanox-sx6036-rack-1 [standalone: master] # write memory
mellanox-sx6036-rack-1 [standalone: master] #

Switch configuration commands

Some of these things will need licenses first.

I can haz Ethernet?

Install the Ethernet key as shown above and confirm it is working:

mellanox-sx6036-rack-1 [standalone: master] # show system capabilities
IB: Supported, L2, Adaptive Routing
Ethernet: Supported, L2, L3
GW: Not supported
Max SM nodes: 648
IB Max licensed speed: FDR10
Ethernet Max licensed speed: 56Gb
mellanox-sx6036-rack-1 [standalone: master] #

It says we can do Ethernet. Which is cool. But it does not have the Infiniband-Ethernet gateway feature licensed. And this is required to enable the switch to run in VPI mode. VPI is "Virtual Protocol Interconnect" or Mellanox's way of saying "We can do Infiniband and Ethernet on a single switch." So another license is needed. Here's a key for that:

mellanox-sx6036-rack-1 [standalone: master] # configure terminal
mellanox-sx6036-rack-1 [standalone: master] (config) # license install LK2-EFM_SX-5N21-488A-182A-UQXB-Y6
mellanox-sx6036-rack-1 [standalone: master] (config) # exit
mellanox-sx6036-rack-1 [standalone: master] # show system capabilities 
IB: Supported, L2, Adaptive Routing
Ethernet: Supported, L2, L3
GW: Supported
Max SM nodes: 648
IB Max licensed speed: FDR10
Ethernet Max licensed speed: 56Gb
mellanox-sx6036-rack-1 [standalone: master] # 

We will need to do this next thing to convert the switch from an Infiniband-only switch to an Infiniband-and-Ethernet switch:

mellanox-sx6036-rack-1 [standalone: master] # configure terminal
mellanox-sx6036-rack-1 [standalone: master] (config) # system profile vpi-single-switch
Warning - confirming will cause system reboot and all configuration will be deleted
Type 'yes' to confirm profile change: yes

Allow time for the switch to reboot and log in again. When it next starts up, run the command show system profile to confirm that it is in VPI mode:

mellanox-sx6036-rack-1 [standalone: master] > show system profile

Profile: vpi-single-switch

mellanox-sx6036-rack-1 [standalone: master] >

And finally, let's set some ports to Ethernet mode:

mellanox-sx6036-rack-1 [standalone: master] > show ports type
InfiniBand: 1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10 1/11 1/12 1/13 1/14 1/15 1/16 1/17 1/18 1/19 1/20 1/21 1/22 1/23 1/24 1/25 1/26 1/27 1/28 1/29 1/30 1/31 1/32 1/33 1/34 1/35 1/36 
mellanox-sx6036-rack-1 [standalone: master] > enable
mellanox-sx6036-rack-1 [standalone: master] # configure terminal
mellanox-sx6036-rack-1 [standalone: master] (config) # interface ib 1/19-1/36
mellanox-sx6036-rack-1 [standalone: master] (config interface ib 1/19-1/36) # shutdown
mellanox-sx6036-rack-1 [standalone: master] (config interface ib 1/19-1/36) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # port 1/19-1/36 type ethernet
mellanox-sx6036-rack-1 [standalone: master] (config) # interface ethernet 1/19-1/36
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/19-1/36) # no shutdown
mellanox-sx6036-rack-1 [standalone: master] (config interface ethernet 1/19-1/36) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # exit
mellanox-sx6036-rack-1 [standalone: master] # write memory
mellanox-sx6036-rack-1 [standalone: master] # show ports type
Ethernet:   1/19 1/20 1/21 1/22 1/23 1/24 1/25 1/26 1/27 1/28 1/29 1/30 1/31 1/32 1/33 1/34 1/35 1/36 
InfiniBand: 1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 1/10 1/11 1/12 1/13 1/14 1/15 1/16 1/17 1/18 
mellanox-sx6036-rack-1 [standalone: master] #

Switch management items

Password for the admin user

Factory defaults are probably not a good idea, so let's tighten things up a bit.

mellanox-sx6036-rack-1 [standalone: master] > enable
mellanox-sx6036-rack-1 [standalone: master] # configure terminal
mellanox-sx6036-rack-1 [standalone: master] (config) # username admin password the_password_goes_here 
mellanox-sx6036-rack-1 [standalone: master] (config) # exit
mellanox-sx6036-rack-1 [standalone: master] # write memory
mellanox-sx6036-rack-1 [standalone: master] # 

IP addressing for the management interface

The following will configure the switch to retrieve its IP address, subnet mask, router IP address, name server info, and hostname from DHCP server on the management interface's local network.

mellanox-sx6036-rack-1 [standalone: master] > enable
mellanox-sx6036-rack-1 [standalone: master] # configure terminal
mellanox-sx6036-rack-1 [standalone: master] (config) # interface mgmt0
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # dhcp
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # dhcp hostname
mellanox-sx6036-rack-1 [standalone: master] (config interface mgmt0) # exit
mellanox-sx6036-rack-1 [standalone: master] (config) # exit
mellanox-sx6036-rack-1 [standalone: master] # write memory
mellanox-sx6036-rack-1 [standalone: master] # 

NTP

Let's add a couple of NTP servers to the config and discipline our poorly behaved battery backed clock:

mellanox-sx6036-rack-1 [standalone: master] > enable
mellanox-sx6036-rack-1 [standalone: master] # configure terminal
mellanox-sx6036-rack-1 [standalone: master] (config) # ntp server 172.16.10.2
mellanox-sx6036-rack-1 [standalone: master] (config) # ntp server 172.17.0.17
mellanox-sx6036-rack-1 [standalone: master] (config) # exit
mellanox-sx6036-rack-1 [standalone: master] # write memory
mellanox-sx6036-rack-1 [standalone: master] # show ntp

NTP is administratively            : enabled
NTP Authentication administratively: disabled 

Clock is synchronized:
  Reference: 172.16.10.2
  Offset   : -0.006 ms

Active servers and peers:
  172.16.10.2:
    Conf Type          : serv
    Status             : sys.peer(*)
    Stratum            : 3  
    Offset(msec)       : -0.006
    Ref clock          : 45.45.184.14   
    Poll Interval (sec): 64  
    Last Response (sec): 8   
    Auth state         : none    

  172.17.0.17:
    Conf Type          : serv
    Status             : pending    
    Stratum            : 2  
    Offset(msec)       : 0.429
    Ref clock          : 198.60.22.240  
    Poll Interval (sec): 64  
    Last Response (sec): 7   
    Auth state         : none    

mellanox-sx6036-rack-1 [standalone: master] #