Difference between revisions of "EMC branded Mellanox SX60xx switch"
Line 38: | Line 38: | ||
* some willingness to violate software licenses |
* some willingness to violate software licenses |
||
* attention to detail: breaking the U-Boot image inside the switch is likely to result in a need for I2C tools to make things work again. This is not something I want to deal with. |
* attention to detail: breaking the U-Boot image inside the switch is likely to result in a need for I2C tools to make things work again. This is not something I want to deal with. |
||
+ | |||
+ | === Before breaking it, a diversion into EMC's SwitchOS === |
||
+ | At power on time, a tiny bit of hardware initialization is done and then [https://www.denx.de/wiki/U-Boot U-Boot] (a free as in freedom) bootloader is given control. |
||
+ | |||
+ | Here's what it looks is dumped to the console when power is applied and the automatic boot is interrupted: |
||
+ | |||
+ | |||
+ | |||
+ | U-Boot 2009.01 SX_PPC_M460EX SX_3.2.0330-82-EMC ppc (Feb 27 2013 - 12:13:42) |
||
+ | |||
+ | CPU: AMCC PowerPC 460EX Rev. B at 1000 MHz (PLB=166, OPB=83, EBC=83 MHz) |
||
+ | Security/Kasumi support |
||
+ | Bootstrap Option H - Boot ROM Location I2C (Addr 0x52) |
||
+ | Internal PCI arbiter disabled |
||
+ | 32 kB I-Cache 32 kB D-Cache |
||
+ | Board: Mellanox PPC460EX Board |
||
+ | FDEF: No |
||
+ | I2C: ready |
||
+ | DRAM: 2 GB (ECC enabled, 333 MHz, CL3) |
||
+ | FLASH: 16 MB |
||
+ | NAND: 1024 MiB |
||
+ | PCI: Bus Dev VenId DevId Class Int |
||
+ | PCIE0: link is not up. |
||
+ | PCIE1: successfully set as root-complex |
||
+ | 01 00 15b3 c738 0c06 00 |
||
+ | Net: ppc_4xx_eth0, ppc_4xx_eth1 |
||
+ | Hit any key to stop autoboot: 0 |
||
+ | => |
||
+ | Nothing too super exciting there, but it does tell us what we're looking at. Note that u-Boot is running and we have its CLI available to use. A little bit more info about the hardware is available using the bdinfo U-Boot command: |
||
+ | => '''bdinfo''' |
||
+ | memstart = 0x00000000 |
||
+ | memsize = 0x80000000 |
||
+ | flashstart = 0xFF000000 |
||
+ | flashsize = 0x01000000 |
||
+ | flashoffset = 0x00000000 |
||
+ | sramstart = 0x00000000 |
||
+ | sramsize = 0x00000000 |
||
+ | bootflags = 0xFFFE0218 |
||
+ | intfreq = 1000 MHz |
||
+ | busfreq = 166.667 MHz |
||
+ | ethaddr = 00:02:C9:63:EF:EE |
||
+ | eth1addr = 00:02:C9:63:EF:EF |
||
+ | IP addr = 172.17.255.120 |
||
+ | baudrate = 9600 bps |
||
+ | => |
||
+ | printenv will give a feel for what EMC (before Dell bought them all up) has done pre-boot software wise: |
||
+ | => '''printenv''' |
||
+ | bootcmd=run flash_jffs2 |
||
+ | baudrate=9600 |
||
+ | loads_echo= |
||
+ | autoload=n |
||
+ | hostname=mlnx460ex |
||
+ | netdev=eth0 |
||
+ | nfsargs=setenv bootargs root=/dev/nfs rw nfsroot=${serverip}:${rootpath} |
||
+ | ramargs=setenv bootargs root=/dev/ram rw |
||
+ | addip=setenv bootargs ${bootargs} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}:${netdev}:off panic=1 |
||
+ | addtty=setenv bootargs ${bootargs} console=ttyS0,${baudrate} |
||
+ | addmisc=setenv bootargs ${bootargs} |
||
+ | initrd_high=30000000 |
||
+ | kernel_addr_r=400000 |
||
+ | fdt_addr_r=800000 |
||
+ | ramdisk_addr_r=C00000 |
||
+ | hostname=mlnx460ex |
||
+ | ramdisk_file=mlnx460ex/uRamdisk |
||
+ | rootpath=/opt/eldk/ppc_4xxFP |
||
+ | flash_self=run ramargs addip addtty addmisc;bootm ${kernel_addr} ${ramdisk_addr} ${fdt_addr} |
||
+ | flash_nfs=run nfsargs addip addtty addmisc;bootm ${kernel_addr} - ${fdt_addr} |
||
+ | net_nfs=tftp ${kernel_addr_r} ${bootfile}; tftp ${fdt_addr_r} ${fdt_file}; run nfsargs addip addtty addmisc;bootm ${kernel_addr_r} - ${fdt_addr_r} |
||
+ | net_self_load=tftp ${kernel_addr_r} ${bootfile};tftp ${fdt_addr_r} ${fdt_file};tftp ${ramdisk_addr_r} ${ramdisk_file}; |
||
+ | net_self=run net_self_load;run ramargs addip addtty addmisc;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} |
||
+ | fdt_file=mlnx460ex/mlnx460ex.dtb |
||
+ | flash_self_old=run ramargs addip addtty addmisc;bootm ${kernel_addr} ${ramdisk_addr} |
||
+ | flash_nfs_old=run nfsargs addip addtty addmisc;bootm ${kernel_addr} |
||
+ | net_nfs_old=tftp ${kernel_addr_r} ${bootfile};run nfsargs addip addtty addmisc;bootm ${kernel_addr_r} |
||
+ | load=tftp 200000 mlnx460ex/u-boot.bin |
||
+ | update=protect off 0xFFFA0000 FFFFFFFF;era 0xFFFA0000 FFFFFFFF;cp.b ${fileaddr} 0xFFFA0000 ${filesize};setenv filesize;saveenv |
||
+ | upd=run load update |
||
+ | dhcp_vendor-class-identifier=bootmfg:hwname:mlnx460ex: |
||
+ | clear_filesize=setenv filesize |
||
+ | mfg_dir=mlnx460ex |
||
+ | mfg_args=setenv bootargs root=/dev/ram rw ramdisk_size=${mfg_ramdisk_size} ${mfg_extra_args} |
||
+ | mfg_common_args=run addtty addmisc |
||
+ | mfg_load=tftp ${kernel_addr_r} ${mfg_root}${mfg_dir}/${mfg_kernel_file};tftp ${fdt_addr_r} ${mfg_root}${mfg_dir}/${mfg_fdt_file};tftp ${ramdisk_addr_r} ${mfg_root}${mfg_dir}/${mfg_ramdisk_file} |
||
+ | mfg_nodhcp=echo "Manufacture will TFTP from directory ${mfg_root}${mfg_dir}, and boot";echo; run clear_filesize ; run mfg_load;if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} ; else ; echo Failed mfg load ; fi |
||
+ | mfg=echo "Manufacture will DHCP, TFTP from directory ${mfg_root}${mfg_dir}, and boot";echo; dhcp; run clear_filesize ; run mfg_load;if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} |
||
+ | ${fdt_addr_r} ; else ; echo Failed mfg load ; fi |
||
+ | menu_file=menu.img |
||
+ | menu_load=tftp ${menu_addr_r} ${mfg_root}${mfg_dir}/${menu_file}; if test $? -ne 0; then run clear_filesize ; echo Download failed ;fi |
||
+ | menu_usb_load_ext2=usb start; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${menu_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${menu_file}; |
||
+ | menu_usb_load_fat=usb start; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${menu_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${menu_file}; |
||
+ | menu_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run menu_usb_load_ext2 ; else ; run menu_usb_load_fat ; fi |
||
+ | menu_nodhcp=run clear_filesize ; run menu_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi |
||
+ | menu=dhcp ; run clear_filesize ; run menu_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi |
||
+ | fw_file=u-boot.bin |
||
+ | fw_load=tftp ${fw_addr_r} ${mfg_root}${mfg_dir}/${fw_file}; if test $? -ne 0; then run clear_filesize ; echo Download failed ;fi |
||
+ | fw_usb_load_ext2=usb start; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${fw_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${fw_file}; |
||
+ | fw_usb_load_fat=usb start; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${fw_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${fw_file}; |
||
+ | fw_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run fw_usb_load_ext2 ; else ; run fw_usb_load_fat ; fi |
||
+ | fw_update_raw=protect off 0xFFFA0000 FFFFFFFF;erase 0xFFFA0000 FFFFFFFF;cp.b ${fw_addr_r} 0xFFFA0000 ${filesize};cmp.b ${fw_addr_r} 0xFFFA0000 ${filesize};setenv filesize; saveenv |
||
+ | fw_usb_update=run clear_filesize ; run fw_usb_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi |
||
+ | fw_update_nodhcp=run clear_filesize ; run fw_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi |
||
+ | fw_update=dhcp ; run clear_filesize ; run fw_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi |
||
+ | boot_common_args=run addtty addmisc |
||
+ | mfg_usb_dir=mlnx460ex |
||
+ | mfg_usb_load_ext2=usb start; echo "Loading ${mfg_kernel_file}";ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${kernel_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_kernel_file};echo "Loading ${mfg_fdt_file}"; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${fdt_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_fdt_file};echo "Loading ${mfg_ramdisk_file}"; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${ramdisk_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_ramdisk_file} |
||
+ | mfg_usb_load_fat=usb start; echo "Loading ${mfg_kernel_file}";fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${kernel_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_kernel_file};echo "Loading ${mfg_fdt_file}"; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${fdt_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_fdt_file};echo "Loading ${mfg_ramdisk_file}"; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${ramdisk_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_ramdisk_file} |
||
+ | mfg_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run mfg_usb_load_ext2 ; else ; run mfg_usb_load_fat ; fi |
||
+ | mfg_usb=echo "Manufacture will load from USB directory ${mfg_usb_root}${mfg_usb_dir}, and boot"; echo; run clear_filesize ; run mfg_usb_load; if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} |
||
+ | ${fdt_addr_r} ; else ; echo Failed mfg load ; fi |
||
+ | kernel_addr=ff000000 |
||
+ | fdt_addr=ff1e0000 |
||
+ | ramdisk_addr=ff200000 |
||
+ | fw_addr_r=400000 |
||
+ | menu_addr_r=B00000 |
||
+ | pciconfighost=1 |
||
+ | pcie_mode=RP:RP |
||
+ | autoload=no |
||
+ | rootdev=/dev/mtdblock6 |
||
+ | boot_usb_ext2_loc_1=run usb_args_loc_1 boot_common_args;echo "Loading ${boot_kernel_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r} |
||
+ | boot_usb_ext2_loc_2=run usb_args_loc_2 boot_common_args;echo "Loading ${boot_kernel_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r} |
||
+ | boot_usb_fat_loc_1=run usb_args_loc_1 boot_common_args;echo "Loading ${boot_kernel_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r} |
||
+ | boot_usb_fat_loc_2=run usb_args_loc_2 boot_common_args;echo "Loading ${boot_kernel_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r} |
||
+ | mfg_kernel_file=vmlinuz |
||
+ | mfg_ramdisk_file=rootfs |
||
+ | mfg_ramdisk_size=180224 |
||
+ | mfg_fdt_file=fdt |
||
+ | mfg_usb_dev=0 |
||
+ | mfg_usb_part=1 |
||
+ | mfg_usb_fstype=fat |
||
+ | mfg_usb_root=/ |
||
+ | boot_kernel_file=vmlinuz |
||
+ | boot_fdt_file=fdt |
||
+ | boot_usb_dev=0 |
||
+ | boot_usb_part_loc_1=2 |
||
+ | boot_usb_part_loc_2=3 |
||
+ | boot_usb_root_loc_1=/dev/sda5 |
||
+ | boot_usb_root_loc_2=/dev/sda6 |
||
+ | usb_args_loc_1=setenv bootargs root=${boot_usb_root_loc_1} ro reset_button=${reset_button} rootdelay=8 ${image_kernel_args} ${extra_args} |
||
+ | usb_args_loc_2=setenv bootargs root=${boot_usb_root_loc_2} ro reset_button=${reset_button} rootdelay=8 ${image_kernel_args} ${extra_args} |
||
+ | jffs2_args=setenv bootargs root=${rootdev} rootfstype=jffs2 ro reset_button=${reset_button} ${image_kernel_args} ${extra_args} |
||
+ | mfg_extra_args=ramdisk=65536 |
||
+ | ethaddr=00:02:C9:63:EF:EE |
||
+ | eth1addr=00:02:C9:63:EF:EF |
||
+ | ethact=ppc_4xx_eth0 |
||
+ | bootfile=pxelinux.0 |
||
+ | emc_fabric=B |
||
+ | autoscr=no |
||
+ | filesize=175B |
||
+ | fileaddr=400000 |
||
+ | gatewayip=10.10.4.1 |
||
+ | netmask=255.255.252.0 |
||
+ | ipaddr=172.17.255.120 |
||
+ | serverip=172.17.255.252 |
||
+ | script_rev=3.3.0 |
||
+ | script_date=7.12.13 |
||
+ | autostart=yes |
||
+ | bootdelay=5 |
||
+ | emcram_addr=400000 |
||
+ | emcfl_one_start=ff400000 |
||
+ | emcfl_one_end=ff5fffff |
||
+ | emcfl_two_start=ff600000 |
||
+ | emcfl_two_end=ff7fffff |
||
+ | emcload_addr=ff400000 |
||
+ | load_dir=. |
||
+ | binary_name=ibsw.bin.load |
||
+ | restore_defaults=setenv load_dir .; setenv binary_name ibsw.bin.load; saveenv; |
||
+ | setip_176=setenv ipaddr 192.168.176.190; setenv serverip 192.168.176.253; saveenv |
||
+ | setip_177=setenv ipaddr 192.168.177.191; setenv serverip 192.168.177.253; saveenv |
||
+ | setip_linux=setenv ipaddr 192.168.10.10; setenv serverip 192.168.10.1; saveenv |
||
+ | setip_16=setenv ipaddr 172.16.255.120; setenv serverip 172.16.255.252; saveenv |
||
+ | setip_17=setenv ipaddr 172.17.255.120; setenv serverip 172.17.255.252; saveenv |
||
+ | set_fab_a=setenv emc_fabric A; saveenv |
||
+ | set_fab_b=setenv emc_fabric B; saveenv |
||
+ | emcpxe=tftp ${emcram_addr} ${load_dir}/${binary_name} |
||
+ | emcflash=ping $serverip; bootm ${emcload_addr} |
||
+ | mlxlinux=run jffs2_args boot_common_args;bootm ${kernel_addr} - ${fdt_addr} |
||
+ | flash_jffs2=run emcflash |
||
+ | menu_usb=run emcflash |
||
+ | emcburn_one=prot off ${emcfl_one_start} ${emcfl_one_end}; erase ${emcfl_one_start} ${emcfl_one_end}; setenv autostart no; tftp ${emcram_addr} ${load_dir}/${binary_name}; iminfo ${emcram_addr}; cp.b ${emcram_addr} ${emcfl_one_start} ${filesize}; iminfo |
||
+ | ${emcfl_one_start}; setenv autostart yes; prot on ${emcfl_one_start} ${emcfl_one_end}; |
||
+ | emcburn_two=prot off ${emcfl_two_start} ${emcfl_two_end}; erase ${emcfl_two_start} ${emcfl_two_end}; setenv autostart no; tftp ${emcram_addr} ${load_dir}/${binary_name}; iminfo ${emcram_addr}; cp.b ${emcram_addr} ${emcfl_two_start} ${filesize}; iminfo |
||
+ | ${emcfl_two_start}; setenv autostart yes; prot on ${emcfl_two_start} ${emcfl_two_end}; |
||
+ | emcload_one=setenv emcload_addr ${emcfl_one_start}; saveenv |
||
+ | emcload_two=setenv emcload_addr ${emcfl_two_start}; saveenv |
||
+ | script_get=setenv autostart no; tftp ${emcram_addr} uboot_start_script.img; setenv autostart yes; imi ${emcram_addr}; |
||
+ | script_exe=autoscr ${emcram_addr} |
||
+ | boot_emcpxe=setenv flash_jffs2 run emcpxe; setenv menu_usb run emcpxe; saveenv |
||
+ | boot_emcflash=setenv flash_jffs2 run emcflash; setenv menu_usb run emcflash; saveenv |
||
+ | boot_mlxlinux=setenv flash_jffs2 run mlxlinux; setenv menu_usb run mlxlinux; saveenv |
||
+ | emchelp=echo All commands are preceeded by \"run \"; echo setip_176 ...... set ip and server ip to 192.176 subnet; echo setip_177 ...... set ip and server ip to 192.177 subnet; echo setip_16 ....... set ip and server ip to 172.16 subnet; echo setip_17 ....... set |
||
+ | ip and server ip to 172.17 subnet;echo setip_linux .... set ip and server ip to linux default (10.10); echo emcburn_one .... burn ibsw.bin.load to the first symmk flash location;echo emcburn_two .... burn ibsw.bin.load to the second symmk flash location;echo |
||
+ | emcload_one .... set emcflash to point to the first symmk flash location;echo emcload_two .... set emcflash to point to the second symmk flash location;echo script_get ..... Load a new version of the uboot_start_script.img;echo script_exe ..... Execute the script |
||
+ | loaded by script_get (check CRC first!);echo script_status .. Dump revision and build date of last run script; echo boot_emcpxe .... Set PXE boot to be the default boot;echo emcpxe ......... Run PXE boot (just this time); echo boot_emcflash .. Set the Symmk in |
||
+ | flash to be the default boot;echo emcflash ....... Boot (just this time) from flash;echo boot_mlxlinux .. Set the MLX linux version to be the default boot;echo mlxlinux ....... Run MLX linux (just this time);echo set_fab_a ...... Set switch to be fabric A; echo |
||
+ | set_fab_b ...... Set switch to be fabric B |
||
+ | script_status=echo Last Script Executed; echo Script Revision: 3.3.0; echo Script Built on: 7.12.13; |
||
+ | stdin=serial |
||
+ | stdout=serial |
||
+ | stderr=serial |
||
+ | reset_button=0 |
||
+ | ver=U-Boot 2009.01 SX_PPC_M460EX SX_3.2.0330-82-EMC ppc (Feb 27 2013 - 12:13:42) |
||
+ | |||
+ | Environment size: 12502/16380 bytes |
||
+ | => |
Revision as of 17:28, 16 December 2017
Used enterprise IT stuff is stupid cheap in the US. Stupid cheap like 18 port FDR (56Gbits/sec) Infiniband switches shipped to your door for 6.67USD per port. Only a couple of issues with these:
- They run a very limited EMC OS instead of the somewhat more open Linux-based Mellanox MLNX-OS
- No VPI (Ethernet over Infiniband, more or less)
- No subnet manager (facilitates IP over Infiniband)
- Management by telnet ("SSH? That's so not going to happen!")
- It's not Linux
The devices at FnordNet are Mellanox SX6018s wearing EMC colors. (Mellanox ships switches with blue fronts and EMC switches are black.) On the connector side are:
- 18 QSFP 56Gbits/sec Infiniband connectors
- 2 8P8C modular 1000baseT Ethernet connectors (Labelled MGT)
- 1 USB connector
- 1 8P8C modular connector for the serial console (labelled CONSOLE)
- a small recessed reset button switch (small hole on the left side of the connector panel)
Mellanox makes a number of similar devices with the same generation of silicon:
- SX6005 and SX6012 -- 1U, half width, 12 QSFP connectors arranged in two rows of six. SX6005 is the unmanaged version. SX6012 is the managed switch with the subnet manager and single port Ethernet connectivity
- SX6015 and SX6018 -- 1U, full width, 18 QSFP connectors in one row of 18. SX6018 is the managed version with a subnet manager and dual Ethernet management ports
- SX6025 and SX6036 -- 1U, full width, 36 QSFP connectors in two rows of 18. SX6025 is unmanaged. SX6036 is the managed version.
All of these are built on the same Infiniband silicon -- Mellanox SwitchX-2. The management functions on the SX6012, SX6018, and SX6036 are done by an embedded Linux system running on a PowerPC M460EX series CPU attached to the SwitchX-2 silicon over a PCI interface of some sort.
Putting MLNX-OS on it
As mentioned above, I desire to have a real OS on these switches. Found some leads over on https://forums.servethehome.com/ which said conversion is possible, but "PM me for more details." The following is a translation of the nice gentleman's "Mellanox Switch Conversion Guide" document into the procedure I have used.
Pre-requisiteses
(Yes. I know I misspelled that.)
- console cable for switch (Cisco cables work fine)
- serial comms program (C-Kermit and I get along just fine. Others may use screen. Or minicom. Or Teraterm. Or HyperTerminal. Or for people with long memories, even {COMMO} or Telix or Qmodem.)
- Willingness to root around typing stuff into a text console fiddle with unsupported softwares
- A MLNX-OS binary distribution for the switch in question. Try:
- a hex editor (we'll be picking apart some binary archives at some point)
- some time
- some willingness to violate software licenses
- attention to detail: breaking the U-Boot image inside the switch is likely to result in a need for I2C tools to make things work again. This is not something I want to deal with.
Before breaking it, a diversion into EMC's SwitchOS
At power on time, a tiny bit of hardware initialization is done and then U-Boot (a free as in freedom) bootloader is given control.
Here's what it looks is dumped to the console when power is applied and the automatic boot is interrupted:
U-Boot 2009.01 SX_PPC_M460EX SX_3.2.0330-82-EMC ppc (Feb 27 2013 - 12:13:42) CPU: AMCC PowerPC 460EX Rev. B at 1000 MHz (PLB=166, OPB=83, EBC=83 MHz) Security/Kasumi support Bootstrap Option H - Boot ROM Location I2C (Addr 0x52) Internal PCI arbiter disabled 32 kB I-Cache 32 kB D-Cache Board: Mellanox PPC460EX Board FDEF: No I2C: ready DRAM: 2 GB (ECC enabled, 333 MHz, CL3) FLASH: 16 MB NAND: 1024 MiB PCI: Bus Dev VenId DevId Class Int PCIE0: link is not up. PCIE1: successfully set as root-complex 01 00 15b3 c738 0c06 00 Net: ppc_4xx_eth0, ppc_4xx_eth1 Hit any key to stop autoboot: 0 =>
Nothing too super exciting there, but it does tell us what we're looking at. Note that u-Boot is running and we have its CLI available to use. A little bit more info about the hardware is available using the bdinfo U-Boot command:
=> bdinfo memstart = 0x00000000 memsize = 0x80000000 flashstart = 0xFF000000 flashsize = 0x01000000 flashoffset = 0x00000000 sramstart = 0x00000000 sramsize = 0x00000000 bootflags = 0xFFFE0218 intfreq = 1000 MHz busfreq = 166.667 MHz ethaddr = 00:02:C9:63:EF:EE eth1addr = 00:02:C9:63:EF:EF IP addr = 172.17.255.120 baudrate = 9600 bps =>
printenv will give a feel for what EMC (before Dell bought them all up) has done pre-boot software wise:
=> printenv bootcmd=run flash_jffs2 baudrate=9600 loads_echo= autoload=n hostname=mlnx460ex netdev=eth0 nfsargs=setenv bootargs root=/dev/nfs rw nfsroot=${serverip}:${rootpath} ramargs=setenv bootargs root=/dev/ram rw addip=setenv bootargs ${bootargs} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}:${netdev}:off panic=1 addtty=setenv bootargs ${bootargs} console=ttyS0,${baudrate} addmisc=setenv bootargs ${bootargs} initrd_high=30000000 kernel_addr_r=400000 fdt_addr_r=800000 ramdisk_addr_r=C00000 hostname=mlnx460ex ramdisk_file=mlnx460ex/uRamdisk rootpath=/opt/eldk/ppc_4xxFP flash_self=run ramargs addip addtty addmisc;bootm ${kernel_addr} ${ramdisk_addr} ${fdt_addr} flash_nfs=run nfsargs addip addtty addmisc;bootm ${kernel_addr} - ${fdt_addr} net_nfs=tftp ${kernel_addr_r} ${bootfile}; tftp ${fdt_addr_r} ${fdt_file}; run nfsargs addip addtty addmisc;bootm ${kernel_addr_r} - ${fdt_addr_r} net_self_load=tftp ${kernel_addr_r} ${bootfile};tftp ${fdt_addr_r} ${fdt_file};tftp ${ramdisk_addr_r} ${ramdisk_file}; net_self=run net_self_load;run ramargs addip addtty addmisc;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} fdt_file=mlnx460ex/mlnx460ex.dtb flash_self_old=run ramargs addip addtty addmisc;bootm ${kernel_addr} ${ramdisk_addr} flash_nfs_old=run nfsargs addip addtty addmisc;bootm ${kernel_addr} net_nfs_old=tftp ${kernel_addr_r} ${bootfile};run nfsargs addip addtty addmisc;bootm ${kernel_addr_r} load=tftp 200000 mlnx460ex/u-boot.bin update=protect off 0xFFFA0000 FFFFFFFF;era 0xFFFA0000 FFFFFFFF;cp.b ${fileaddr} 0xFFFA0000 ${filesize};setenv filesize;saveenv upd=run load update dhcp_vendor-class-identifier=bootmfg:hwname:mlnx460ex: clear_filesize=setenv filesize mfg_dir=mlnx460ex mfg_args=setenv bootargs root=/dev/ram rw ramdisk_size=${mfg_ramdisk_size} ${mfg_extra_args} mfg_common_args=run addtty addmisc mfg_load=tftp ${kernel_addr_r} ${mfg_root}${mfg_dir}/${mfg_kernel_file};tftp ${fdt_addr_r} ${mfg_root}${mfg_dir}/${mfg_fdt_file};tftp ${ramdisk_addr_r} ${mfg_root}${mfg_dir}/${mfg_ramdisk_file} mfg_nodhcp=echo "Manufacture will TFTP from directory ${mfg_root}${mfg_dir}, and boot";echo; run clear_filesize ; run mfg_load;if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} ; else ; echo Failed mfg load ; fi mfg=echo "Manufacture will DHCP, TFTP from directory ${mfg_root}${mfg_dir}, and boot";echo; dhcp; run clear_filesize ; run mfg_load;if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} ; else ; echo Failed mfg load ; fi menu_file=menu.img menu_load=tftp ${menu_addr_r} ${mfg_root}${mfg_dir}/${menu_file}; if test $? -ne 0; then run clear_filesize ; echo Download failed ;fi menu_usb_load_ext2=usb start; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${menu_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${menu_file}; menu_usb_load_fat=usb start; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${menu_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${menu_file}; menu_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run menu_usb_load_ext2 ; else ; run menu_usb_load_fat ; fi menu_nodhcp=run clear_filesize ; run menu_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi menu=dhcp ; run clear_filesize ; run menu_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi fw_file=u-boot.bin fw_load=tftp ${fw_addr_r} ${mfg_root}${mfg_dir}/${fw_file}; if test $? -ne 0; then run clear_filesize ; echo Download failed ;fi fw_usb_load_ext2=usb start; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${fw_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${fw_file}; fw_usb_load_fat=usb start; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${fw_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${fw_file}; fw_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run fw_usb_load_ext2 ; else ; run fw_usb_load_fat ; fi fw_update_raw=protect off 0xFFFA0000 FFFFFFFF;erase 0xFFFA0000 FFFFFFFF;cp.b ${fw_addr_r} 0xFFFA0000 ${filesize};cmp.b ${fw_addr_r} 0xFFFA0000 ${filesize};setenv filesize; saveenv fw_usb_update=run clear_filesize ; run fw_usb_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi fw_update_nodhcp=run clear_filesize ; run fw_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi fw_update=dhcp ; run clear_filesize ; run fw_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi boot_common_args=run addtty addmisc mfg_usb_dir=mlnx460ex mfg_usb_load_ext2=usb start; echo "Loading ${mfg_kernel_file}";ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${kernel_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_kernel_file};echo "Loading ${mfg_fdt_file}"; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${fdt_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_fdt_file};echo "Loading ${mfg_ramdisk_file}"; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${ramdisk_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_ramdisk_file} mfg_usb_load_fat=usb start; echo "Loading ${mfg_kernel_file}";fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${kernel_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_kernel_file};echo "Loading ${mfg_fdt_file}"; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${fdt_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_fdt_file};echo "Loading ${mfg_ramdisk_file}"; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${ramdisk_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_ramdisk_file} mfg_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run mfg_usb_load_ext2 ; else ; run mfg_usb_load_fat ; fi mfg_usb=echo "Manufacture will load from USB directory ${mfg_usb_root}${mfg_usb_dir}, and boot"; echo; run clear_filesize ; run mfg_usb_load; if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} ; else ; echo Failed mfg load ; fi kernel_addr=ff000000 fdt_addr=ff1e0000 ramdisk_addr=ff200000 fw_addr_r=400000 menu_addr_r=B00000 pciconfighost=1 pcie_mode=RP:RP autoload=no rootdev=/dev/mtdblock6 boot_usb_ext2_loc_1=run usb_args_loc_1 boot_common_args;echo "Loading ${boot_kernel_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r} boot_usb_ext2_loc_2=run usb_args_loc_2 boot_common_args;echo "Loading ${boot_kernel_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r} boot_usb_fat_loc_1=run usb_args_loc_1 boot_common_args;echo "Loading ${boot_kernel_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r} boot_usb_fat_loc_2=run usb_args_loc_2 boot_common_args;echo "Loading ${boot_kernel_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r} mfg_kernel_file=vmlinuz mfg_ramdisk_file=rootfs mfg_ramdisk_size=180224 mfg_fdt_file=fdt mfg_usb_dev=0 mfg_usb_part=1 mfg_usb_fstype=fat mfg_usb_root=/ boot_kernel_file=vmlinuz boot_fdt_file=fdt boot_usb_dev=0 boot_usb_part_loc_1=2 boot_usb_part_loc_2=3 boot_usb_root_loc_1=/dev/sda5 boot_usb_root_loc_2=/dev/sda6 usb_args_loc_1=setenv bootargs root=${boot_usb_root_loc_1} ro reset_button=${reset_button} rootdelay=8 ${image_kernel_args} ${extra_args} usb_args_loc_2=setenv bootargs root=${boot_usb_root_loc_2} ro reset_button=${reset_button} rootdelay=8 ${image_kernel_args} ${extra_args} jffs2_args=setenv bootargs root=${rootdev} rootfstype=jffs2 ro reset_button=${reset_button} ${image_kernel_args} ${extra_args} mfg_extra_args=ramdisk=65536 ethaddr=00:02:C9:63:EF:EE eth1addr=00:02:C9:63:EF:EF ethact=ppc_4xx_eth0 bootfile=pxelinux.0 emc_fabric=B autoscr=no filesize=175B fileaddr=400000 gatewayip=10.10.4.1 netmask=255.255.252.0 ipaddr=172.17.255.120 serverip=172.17.255.252 script_rev=3.3.0 script_date=7.12.13 autostart=yes bootdelay=5 emcram_addr=400000 emcfl_one_start=ff400000 emcfl_one_end=ff5fffff emcfl_two_start=ff600000 emcfl_two_end=ff7fffff emcload_addr=ff400000 load_dir=. binary_name=ibsw.bin.load restore_defaults=setenv load_dir .; setenv binary_name ibsw.bin.load; saveenv; setip_176=setenv ipaddr 192.168.176.190; setenv serverip 192.168.176.253; saveenv setip_177=setenv ipaddr 192.168.177.191; setenv serverip 192.168.177.253; saveenv setip_linux=setenv ipaddr 192.168.10.10; setenv serverip 192.168.10.1; saveenv setip_16=setenv ipaddr 172.16.255.120; setenv serverip 172.16.255.252; saveenv setip_17=setenv ipaddr 172.17.255.120; setenv serverip 172.17.255.252; saveenv set_fab_a=setenv emc_fabric A; saveenv set_fab_b=setenv emc_fabric B; saveenv emcpxe=tftp ${emcram_addr} ${load_dir}/${binary_name} emcflash=ping $serverip; bootm ${emcload_addr} mlxlinux=run jffs2_args boot_common_args;bootm ${kernel_addr} - ${fdt_addr} flash_jffs2=run emcflash menu_usb=run emcflash emcburn_one=prot off ${emcfl_one_start} ${emcfl_one_end}; erase ${emcfl_one_start} ${emcfl_one_end}; setenv autostart no; tftp ${emcram_addr} ${load_dir}/${binary_name}; iminfo ${emcram_addr}; cp.b ${emcram_addr} ${emcfl_one_start} ${filesize}; iminfo ${emcfl_one_start}; setenv autostart yes; prot on ${emcfl_one_start} ${emcfl_one_end}; emcburn_two=prot off ${emcfl_two_start} ${emcfl_two_end}; erase ${emcfl_two_start} ${emcfl_two_end}; setenv autostart no; tftp ${emcram_addr} ${load_dir}/${binary_name}; iminfo ${emcram_addr}; cp.b ${emcram_addr} ${emcfl_two_start} ${filesize}; iminfo ${emcfl_two_start}; setenv autostart yes; prot on ${emcfl_two_start} ${emcfl_two_end}; emcload_one=setenv emcload_addr ${emcfl_one_start}; saveenv emcload_two=setenv emcload_addr ${emcfl_two_start}; saveenv script_get=setenv autostart no; tftp ${emcram_addr} uboot_start_script.img; setenv autostart yes; imi ${emcram_addr}; script_exe=autoscr ${emcram_addr} boot_emcpxe=setenv flash_jffs2 run emcpxe; setenv menu_usb run emcpxe; saveenv boot_emcflash=setenv flash_jffs2 run emcflash; setenv menu_usb run emcflash; saveenv boot_mlxlinux=setenv flash_jffs2 run mlxlinux; setenv menu_usb run mlxlinux; saveenv emchelp=echo All commands are preceeded by \"run \"; echo setip_176 ...... set ip and server ip to 192.176 subnet; echo setip_177 ...... set ip and server ip to 192.177 subnet; echo setip_16 ....... set ip and server ip to 172.16 subnet; echo setip_17 ....... set ip and server ip to 172.17 subnet;echo setip_linux .... set ip and server ip to linux default (10.10); echo emcburn_one .... burn ibsw.bin.load to the first symmk flash location;echo emcburn_two .... burn ibsw.bin.load to the second symmk flash location;echo emcload_one .... set emcflash to point to the first symmk flash location;echo emcload_two .... set emcflash to point to the second symmk flash location;echo script_get ..... Load a new version of the uboot_start_script.img;echo script_exe ..... Execute the script loaded by script_get (check CRC first!);echo script_status .. Dump revision and build date of last run script; echo boot_emcpxe .... Set PXE boot to be the default boot;echo emcpxe ......... Run PXE boot (just this time); echo boot_emcflash .. Set the Symmk in flash to be the default boot;echo emcflash ....... Boot (just this time) from flash;echo boot_mlxlinux .. Set the MLX linux version to be the default boot;echo mlxlinux ....... Run MLX linux (just this time);echo set_fab_a ...... Set switch to be fabric A; echo set_fab_b ...... Set switch to be fabric B script_status=echo Last Script Executed; echo Script Revision: 3.3.0; echo Script Built on: 7.12.13; stdin=serial stdout=serial stderr=serial reset_button=0 ver=U-Boot 2009.01 SX_PPC_M460EX SX_3.2.0330-82-EMC ppc (Feb 27 2013 - 12:13:42) Environment size: 12502/16380 bytes =>