Difference between revisions of "EMC branded Mellanox SX60xx switch"

From FnordWiki
Jump to navigation Jump to search
Line 652: Line 652:
 
chmod 644 etc/inittab
  
chmod 644 etc/inittab
 
 
−
# /etc/passwd listing for 'root' The password a a classic UNIX hash of "LetMeIn" Note that we don't actually check the password when logging a user in. But still....
 +
# /etc/passwd listing for 'root' The password a classic UNIX (DES based) hash of "changeme" a common default root password on Sun gear.
 
cp /dev/null etc/passwd
  
cp /dev/null etc/passwd
−
echo 'root::0:0::/root:/bin/sh' > etc/passwd
 +
echo 'root:Xxxgg2TS4senE:0:0::/root:/bin/sh' > etc/passwd
 
chmod 644 etc/passwd
  
chmod 644 etc/passwd
 
 

Revision as of 19:18, 27 January 2019

Used enterprise IT stuff is stupid cheap in the US. Stupid cheap like 18 port FDR (56Gbits/sec) Infiniband switches shipped to your door for 6.67USD per port. Only a couple of issues with these:

  • They run a very limited EMC OS instead of the somewhat more open Linux-based Mellanox MLNX-OS
  • No VPI (Ethernet over Infiniband, more or less)
  • No subnet manager (facilitates IP over Infiniband)
  • Management by telnet ("SSH? That's so not going to happen!")
  • It's not Linux

The devices at FnordNet are Mellanox SX6018s wearing EMC colors. (Mellanox ships switches with blue fronts and EMC switches are black.) On the connector side are:

  • 18 QSFP 56Gbits/sec Infiniband connectors
  • 2 8P8C modular 1000baseT Ethernet connectors (Labelled MGT)
  • 1 USB connector
  • 1 8P8C modular connector for the serial console (labelled CONSOLE)
  • a small recessed reset button switch (small hole on the left side of the connector panel)

Mellanox makes a number of similar devices with the same generation of silicon:

  • SX6005 and SX6012 -- 1U, half width, 12 QSFP connectors arranged in two rows of six. SX6005 is the unmanaged version. SX6012 is the managed switch with the subnet manager and single port Ethernet connectivity
  • SX6015 and SX6018 -- 1U, full width, 18 QSFP connectors in one row of 18. SX6018 is the managed version with a subnet manager and dual Ethernet management ports
  • SX6025 and SX6036 -- 1U, full width, 36 QSFP connectors in two rows of 18. SX6025 is unmanaged. SX6036 is the managed version.

All of these are built on the same Infiniband silicon -- Mellanox SwitchX-2. The management functions on the SX6012, SX6018, and SX6036 are done by an embedded Linux system running on a PowerPC M460EX series CPU attached to the SwitchX-2 silicon over a PCI interface of some sort.

Before breaking it, a diversion into EMC's SwitchOS

At power on time, a tiny bit of hardware initialization is done and then U-Boot (a free as in freedom) bootloader is given control.

Here's what it looks is dumped to the console when power is applied and the automatic boot is interrupted: 


U-Boot 2009.01 SX_PPC_M460EX SX_3.2.0330-82-EMC ppc (Feb 27 2013 - 12:13:42)

CPU:   AMCC PowerPC 460EX Rev. B at 1000 MHz (PLB=166, OPB=83, EBC=83 MHz)
       Security/Kasumi support
       Bootstrap Option H - Boot ROM Location I2C (Addr 0x52)
       Internal PCI arbiter disabled
       32 kB I-Cache 32 kB D-Cache
Board: Mellanox PPC460EX Board
FDEF:  No
I2C:   ready
DRAM:   2 GB (ECC enabled, 333 MHz, CL3)
FLASH: 16 MB
NAND:  1024 MiB
PCI:   Bus Dev VenId DevId Class Int
PCIE0: link is not up.
PCIE1: successfully set as root-complex
        01  00  15b3  c738  0c06  00
Net:   ppc_4xx_eth0, ppc_4xx_eth1
Hit any key to stop autoboot:  0 
=>

Nothing too super exciting there, but it does tell us what we're looking at. Note that u-Boot is running and we have its CLI available to use. A little bit more info about the hardware is available using the bdinfo U-Boot command: 

=> bdinfo
memstart    = 0x00000000
memsize     = 0x80000000
flashstart  = 0xFF000000
flashsize   = 0x01000000
flashoffset = 0x00000000
sramstart   = 0x00000000
sramsize    = 0x00000000
bootflags   = 0xFFFE0218
intfreq     =   1000 MHz
busfreq     = 166.667 MHz
ethaddr     = 00:02:C9:63:EF:EE
eth1addr    = 00:02:C9:63:EF:EF
IP addr     = 172.17.255.120
baudrate    =   9600 bps
=>

printenv will give a feel for what EMC (before Dell bought them all up) has done pre-boot software wise: 

=> printenv
bootcmd=run flash_jffs2
baudrate=9600
loads_echo=
autoload=n
hostname=mlnx460ex
netdev=eth0
nfsargs=setenv bootargs root=/dev/nfs rw nfsroot=${serverip}:${rootpath}
ramargs=setenv bootargs root=/dev/ram rw
addip=setenv bootargs ${bootargs} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}:${netdev}:off panic=1
addtty=setenv bootargs ${bootargs} console=ttyS0,${baudrate}
addmisc=setenv bootargs ${bootargs}
initrd_high=30000000
kernel_addr_r=400000
fdt_addr_r=800000
ramdisk_addr_r=C00000
hostname=mlnx460ex
ramdisk_file=mlnx460ex/uRamdisk
rootpath=/opt/eldk/ppc_4xxFP
flash_self=run ramargs addip addtty addmisc;bootm ${kernel_addr} ${ramdisk_addr} ${fdt_addr}
flash_nfs=run nfsargs addip addtty addmisc;bootm ${kernel_addr} - ${fdt_addr}
net_nfs=tftp ${kernel_addr_r} ${bootfile}; tftp ${fdt_addr_r} ${fdt_file}; run nfsargs addip addtty addmisc;bootm ${kernel_addr_r} - ${fdt_addr_r}
net_self_load=tftp ${kernel_addr_r} ${bootfile};tftp ${fdt_addr_r} ${fdt_file};tftp ${ramdisk_addr_r} ${ramdisk_file};
net_self=run net_self_load;run ramargs addip addtty addmisc;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
fdt_file=mlnx460ex/mlnx460ex.dtb
flash_self_old=run ramargs addip addtty addmisc;bootm ${kernel_addr} ${ramdisk_addr}
flash_nfs_old=run nfsargs addip addtty addmisc;bootm ${kernel_addr}
net_nfs_old=tftp ${kernel_addr_r} ${bootfile};run nfsargs addip addtty addmisc;bootm ${kernel_addr_r}
load=tftp 200000 mlnx460ex/u-boot.bin
update=protect off 0xFFFA0000 FFFFFFFF;era 0xFFFA0000 FFFFFFFF;cp.b ${fileaddr} 0xFFFA0000 ${filesize};setenv filesize;saveenv
upd=run load update
dhcp_vendor-class-identifier=bootmfg:hwname:mlnx460ex:
clear_filesize=setenv filesize
mfg_dir=mlnx460ex
mfg_args=setenv bootargs root=/dev/ram rw ramdisk_size=${mfg_ramdisk_size} ${mfg_extra_args}
mfg_common_args=run addtty addmisc
mfg_load=tftp ${kernel_addr_r} ${mfg_root}${mfg_dir}/${mfg_kernel_file};tftp ${fdt_addr_r} ${mfg_root}${mfg_dir}/${mfg_fdt_file};tftp ${ramdisk_addr_r} ${mfg_root}${mfg_dir}/${mfg_ramdisk_file}
mfg_nodhcp=echo "Manufacture will TFTP from directory ${mfg_root}${mfg_dir}, and boot";echo; run clear_filesize ; run mfg_load;if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r} ; else ; echo Failed mfg load ; fi
mfg=echo "Manufacture will DHCP, TFTP from directory ${mfg_root}${mfg_dir}, and boot";echo; dhcp; run clear_filesize ; run mfg_load;if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} 
${fdt_addr_r} ; else ; echo Failed mfg load ; fi
menu_file=menu.img
menu_load=tftp ${menu_addr_r} ${mfg_root}${mfg_dir}/${menu_file}; if test $? -ne 0; then run clear_filesize ; echo Download failed ;fi
menu_usb_load_ext2=usb start; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${menu_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${menu_file};
menu_usb_load_fat=usb start; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${menu_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${menu_file};
menu_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run menu_usb_load_ext2 ; else ; run menu_usb_load_fat ; fi
menu_nodhcp=run clear_filesize ; run menu_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi
menu=dhcp ; run clear_filesize ; run menu_load ; if test 0${filesize} -gt 0; then autoscr ${menu_addr_r}; else ; echo Failed menu load ; fi
fw_file=u-boot.bin
fw_load=tftp ${fw_addr_r} ${mfg_root}${mfg_dir}/${fw_file}; if test $? -ne 0; then run clear_filesize ; echo Download failed ;fi
fw_usb_load_ext2=usb start; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${fw_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${fw_file};
fw_usb_load_fat=usb start; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${fw_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${fw_file};
fw_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run fw_usb_load_ext2 ; else ; run fw_usb_load_fat ; fi
fw_update_raw=protect off 0xFFFA0000 FFFFFFFF;erase 0xFFFA0000 FFFFFFFF;cp.b ${fw_addr_r} 0xFFFA0000 ${filesize};cmp.b ${fw_addr_r} 0xFFFA0000 ${filesize};setenv filesize; saveenv
fw_usb_update=run clear_filesize ; run fw_usb_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi
fw_update_nodhcp=run clear_filesize ; run fw_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi
fw_update=dhcp ; run clear_filesize ; run fw_load ; if test 0${filesize} -gt 0; then run fw_update_raw ; else ; echo Failed update load ; fi
boot_common_args=run addtty addmisc
mfg_usb_dir=mlnx460ex
mfg_usb_load_ext2=usb start; echo "Loading ${mfg_kernel_file}";ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${kernel_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_kernel_file};echo "Loading ${mfg_fdt_file}"; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${fdt_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_fdt_file};echo "Loading ${mfg_ramdisk_file}"; ext2load usb ${mfg_usb_dev}:${mfg_usb_part} ${ramdisk_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_ramdisk_file}
mfg_usb_load_fat=usb start; echo "Loading ${mfg_kernel_file}";fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${kernel_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_kernel_file};echo "Loading ${mfg_fdt_file}"; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${fdt_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_fdt_file};echo "Loading ${mfg_ramdisk_file}"; fatload usb ${mfg_usb_dev}:${mfg_usb_part} ${ramdisk_addr_r} ${mfg_usb_root}${mfg_usb_dir}/${mfg_ramdisk_file}
mfg_usb_load=if test "x${mfg_usb_fstype}" = "xext2"; then run mfg_usb_load_ext2 ; else ; run mfg_usb_load_fat ; fi
mfg_usb=echo "Manufacture will load from USB directory ${mfg_usb_root}${mfg_usb_dir}, and boot"; echo; run clear_filesize ; run mfg_usb_load; if test 0${filesize} -gt 0; then echo Booting mfg ; run mfg_args mfg_common_args;bootm ${kernel_addr_r} ${ramdisk_addr_r} 
${fdt_addr_r} ; else ; echo Failed mfg load ; fi
kernel_addr=ff000000
fdt_addr=ff1e0000
ramdisk_addr=ff200000
fw_addr_r=400000
menu_addr_r=B00000
pciconfighost=1
pcie_mode=RP:RP
autoload=no
rootdev=/dev/mtdblock6
boot_usb_ext2_loc_1=run usb_args_loc_1 boot_common_args;echo "Loading ${boot_kernel_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
boot_usb_ext2_loc_2=run usb_args_loc_2 boot_common_args;echo "Loading ${boot_kernel_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";ext2load usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
boot_usb_fat_loc_1=run usb_args_loc_1 boot_common_args;echo "Loading ${boot_kernel_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_1} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
boot_usb_fat_loc_2=run usb_args_loc_2 boot_common_args;echo "Loading ${boot_kernel_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${kernel_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_kernel_file};echo "Loading ${boot_fdt_file}";fatload usb ${boot_usb_dev}:${boot_usb_part_loc_2} ${fdt_addr_r} ${boot_usb_root}${boot_usb_dir}/${boot_fdt_file};bootm ${kernel_addr_r} - ${fdt_addr_r}
mfg_kernel_file=vmlinuz
mfg_ramdisk_file=rootfs
mfg_ramdisk_size=180224
mfg_fdt_file=fdt
mfg_usb_dev=0
mfg_usb_part=1
mfg_usb_fstype=fat
mfg_usb_root=/
boot_kernel_file=vmlinuz
boot_fdt_file=fdt
boot_usb_dev=0
boot_usb_part_loc_1=2
boot_usb_part_loc_2=3
boot_usb_root_loc_1=/dev/sda5
boot_usb_root_loc_2=/dev/sda6
usb_args_loc_1=setenv bootargs root=${boot_usb_root_loc_1} ro reset_button=${reset_button} rootdelay=8 ${image_kernel_args} ${extra_args}
usb_args_loc_2=setenv bootargs root=${boot_usb_root_loc_2} ro reset_button=${reset_button} rootdelay=8 ${image_kernel_args} ${extra_args}
jffs2_args=setenv bootargs root=${rootdev} rootfstype=jffs2 ro reset_button=${reset_button} ${image_kernel_args} ${extra_args}
mfg_extra_args=ramdisk=65536
ethaddr=00:02:C9:63:EF:EE
eth1addr=00:02:C9:63:EF:EF
ethact=ppc_4xx_eth0
bootfile=pxelinux.0
emc_fabric=B
autoscr=no
filesize=175B
fileaddr=400000
gatewayip=10.10.4.1
netmask=255.255.252.0
ipaddr=172.17.255.120
serverip=172.17.255.252
script_rev=3.3.0
script_date=7.12.13
autostart=yes
bootdelay=5
emcram_addr=400000
emcfl_one_start=ff400000
emcfl_one_end=ff5fffff
emcfl_two_start=ff600000
emcfl_two_end=ff7fffff
emcload_addr=ff400000
load_dir=.
binary_name=ibsw.bin.load
restore_defaults=setenv load_dir .; setenv binary_name ibsw.bin.load; saveenv;
setip_176=setenv ipaddr 192.168.176.190; setenv serverip 192.168.176.253; saveenv
setip_177=setenv ipaddr 192.168.177.191; setenv serverip 192.168.177.253; saveenv
setip_linux=setenv ipaddr 192.168.10.10; setenv serverip 192.168.10.1; saveenv
setip_16=setenv ipaddr 172.16.255.120; setenv serverip 172.16.255.252; saveenv
setip_17=setenv ipaddr 172.17.255.120; setenv serverip 172.17.255.252; saveenv
set_fab_a=setenv emc_fabric A; saveenv
set_fab_b=setenv emc_fabric B; saveenv
emcpxe=tftp ${emcram_addr} ${load_dir}/${binary_name}
emcflash=ping $serverip; bootm ${emcload_addr}
mlxlinux=run jffs2_args boot_common_args;bootm ${kernel_addr} - ${fdt_addr}
flash_jffs2=run emcflash
menu_usb=run emcflash
emcburn_one=prot off ${emcfl_one_start} ${emcfl_one_end}; erase ${emcfl_one_start} ${emcfl_one_end}; setenv autostart no; tftp ${emcram_addr} ${load_dir}/${binary_name}; iminfo ${emcram_addr}; cp.b ${emcram_addr} ${emcfl_one_start} ${filesize}; iminfo 
${emcfl_one_start}; setenv autostart yes; prot on ${emcfl_one_start} ${emcfl_one_end};
emcburn_two=prot off ${emcfl_two_start} ${emcfl_two_end}; erase ${emcfl_two_start} ${emcfl_two_end}; setenv autostart no; tftp ${emcram_addr} ${load_dir}/${binary_name}; iminfo ${emcram_addr}; cp.b ${emcram_addr} ${emcfl_two_start} ${filesize}; iminfo 
${emcfl_two_start}; setenv autostart yes; prot on ${emcfl_two_start} ${emcfl_two_end};
emcload_one=setenv emcload_addr ${emcfl_one_start}; saveenv
emcload_two=setenv emcload_addr ${emcfl_two_start}; saveenv
script_get=setenv autostart no; tftp ${emcram_addr} uboot_start_script.img; setenv autostart yes; imi ${emcram_addr};
script_exe=autoscr ${emcram_addr}
boot_emcpxe=setenv flash_jffs2 run emcpxe; setenv menu_usb run emcpxe; saveenv
boot_emcflash=setenv flash_jffs2 run emcflash; setenv menu_usb run emcflash; saveenv
boot_mlxlinux=setenv flash_jffs2 run mlxlinux; setenv menu_usb run mlxlinux; saveenv
emchelp=echo All commands are preceeded by \"run \"; echo setip_176 ...... set ip and server ip to 192.176 subnet; echo setip_177 ...... set ip and server ip to 192.177 subnet; echo setip_16 ....... set ip and server ip to 172.16 subnet; echo setip_17 ....... set 
ip and server ip to 172.17 subnet;echo setip_linux .... set ip and server ip to linux default (10.10); echo emcburn_one .... burn ibsw.bin.load to the first symmk flash location;echo emcburn_two .... burn ibsw.bin.load to the second symmk flash location;echo 
emcload_one .... set emcflash to point to the first symmk flash location;echo emcload_two .... set emcflash to point to the second symmk flash location;echo script_get ..... Load a new version of the uboot_start_script.img;echo script_exe ..... Execute the script 
loaded by script_get (check CRC first!);echo script_status .. Dump revision and build date of last run script; echo boot_emcpxe .... Set PXE boot to be the default boot;echo emcpxe ......... Run PXE boot (just this time); echo boot_emcflash .. Set the Symmk in 
flash to be the default boot;echo emcflash ....... Boot (just this time) from flash;echo boot_mlxlinux .. Set the MLX linux version to be the default boot;echo mlxlinux ....... Run MLX linux (just this time);echo set_fab_a ...... Set switch to be fabric A; echo 
set_fab_b ...... Set switch to be fabric B
script_status=echo Last Script Executed; echo Script Revision: 3.3.0; echo Script Built on: 7.12.13;
stdin=serial
stdout=serial
stderr=serial
reset_button=0
ver=U-Boot 2009.01 SX_PPC_M460EX SX_3.2.0330-82-EMC ppc (Feb 27 2013 - 12:13:42)

Environment size: 12502/16380 bytes
=>

All that was just the bootloader. Where's SwitchOS?

So, yeah. Last section is poorly named. Remember how the autoboot process was stopped last time? Well, here's what we get without interrupting it... 

[...]
Hit any key to stop autoboot:  0 
Waiting for PHY auto negotiation to complete... done
ENET Speed is 1000 Mbps - FULL duplex connection (EMAC0)
Using ppc_4xx_eth0 device
ping failed; host 172.17.255.252 is not alive
   Loading Kernel Image ... OK
ABCDEFdrv_table_install:         syslog at 0x00033d84 with minor  0 (DRV_SETUP) (DRV_INIT)
drv_table_install:         isrlog at 0x00033d84 with minor  1 (DRV_INIT)
drv_table_install:  userinterface at 0x00004db0 with minor  0 (DRV_SETUP) (DRV_INIT)
drv_table_install:          stty0 at 0x0006e804 with minor  0 (DRV_SETUP) (DRV_INIT)
drv_table_install:           eth0 at 0x0006f194 with minor  0 (DRV_SETUP) (DRV_INIT)
drv_table_install:           i2c0 at 0x00079d94 with minor  0 (DRV_SETUP) (DRV_INIT)
drv_table_install:           i2c1 at 0x00079d94 with minor  1 (DRV_INIT)
drv_table_install:         itcpip at 0x0003c05c with minor  0tcpipInit: Starting internal TCP/IP stack.
 (DRV_SETUP) (DRV_INIT)
kernel_main: Drivers installed, installing INIT process with stack size = 8192.
Bar [ 0 ] is the first memory bar
sk_init_main: Starting process based initialization - 8590.
02:60:48:10:ff:78 UDP socket 3 created
TCP socket 4 created
sk_init_main: Process based initialization complete - 8590.
sk_init_main: Installing task table.
task_table_install:        console at 0x00015c24 stack 0x00452410/26624 : 4
task_table_install:          inetd at 0x000359a0 stack 0x00470c10/8192 : 5
task_table_install:       poll_cqs at 0x000aa44c stack 0x00458c10/8192 : 6
task_table_install:     poll_ports at 0x000aa584 stack 0x0045ac10/8192 : 7
task_table_install:        env_mon at 0x00082f94 stack 0x0045ec10/8192 : 8
task_table_install:    env_bin_api at 0x000143a0 stack 0x00460c10/8192 : 9
task_table_install: incoming_fw_files at 0x00013e84 stack 0x00462c10/8192 : 10
sk_init_main: Task table installed.  Starting tasks and exiting.
----------------------------- Board Info -----------------------------
* Chasis Type         : STINGRAY
* Number of Ports     : 18
* U-Boot Revision     : 3.2.330
* Firmware Revision   : 9.9.1030
* INI file Revision   : 0x21010009
* SwitchOS Revision   : 1.282
* SwitchOS Build Date : 2013-07-23
* SwitchOS Build Time : 15:40:30
* SwitchOS Build Path : /emc/tdowning/ppc460_release/july_23_2013
----------------------------------------------------------------------
15:33:09 12/16/2017
Switch-B(4)>

It almost looks like it's going to netboot if it can find a server to give it an image. But we don't have one and that's totally OK. The Switch-B(4)> is the CLI prompt. There's a help command that lists available commands, but it does not provide a lot of detail. But here are a few potentially useful commands:

  • env show inventory lists out vital product data (VPD) for the replaceable components: switch chassis, power supplies, and fan boards
  • env show fans how fast are they spinning? 0 is probably not good
  • env set uid led blink will blink the blue LED on the left side on the switch's connector panel -- useful for finding the switch you want to work on
  • env set port bad led on is much the same, but activates an orange LED just above the blue UID indicator
  • baz swportinfo 1 will provide port statistics and the like. Change 1 to the number of the switch port of interest
  • build info about the software build
  • board a little bit about the hardware
  • info a bit about the hardware and SwitchOS
  • env show alarms pretty much self explanatory

Putting MLNX-OS on it

As mentioned above, I desire to have a real OS on these switches. Found some leads over on https://forums.servethehome.com/ which said conversion is possible, but "PM me for more details." The following is a translation of the nice gentleman's "Mellanox Switch Conversion Guide" document into the procedure I have used.

Pre-requisiteses

(Yes. I know I misspelled that.)

Make switch not boot automatically

Apply power, interrupt autoboot, run 

=> setenv autostart off
=> setenv autoload off
=> saveenv

We'll revert that by running 

=> setenv autostart yes
=> setenv autoload n
=> saveenv

at some point later on. (Those are the settings as the switch arrived.)

Get bootable bits from MLNX-OS

The image-PPC_M460EX-3.6.1002.img OS images are actually ZIP archives containing a compressed tar file image-PPC_M460EX-ppc-m460ex-20160609-202426.tgz and some package meta-data that we do not need. From image-PPC_M460EX-ppc-m460ex-20160609-202426.tgz, extract the kernel image and device tree files: 

$ gzip -dc image-PPC_M460EX-ppc-m460ex-20160609-202426.tgz | tar xvvf - ./boot/vmlinuz-uni ./boot/fdt-uni

And copy boot/vmlinuz-uni and boot/fdt-uni to a TFTP server that the switch will be able to retrieve these files from. Since my switch has an IP address of 172.17.255.120 as it was delivered, I'm simply adding a secondary IP address to the existing TFTP server instead of changing the switch's IP address.

Let's run Linux!

"Linux" in this section is just the kernel, not an OS -- it will panic the first time through and really not do anything useful.

One these SX6018 switches, the on-board flash has areas for 2 OS images. We're going to overwrite the first one (using U-Boot to TFTP and then flash it) and then boot it. The kernel will not have a root filesystem yet and it will not do anything useful. But it's a step toward having a working MLNX-OS.

We need both the kernel and the matching device tree file from the MLNX-OS distro. The strategy is "use TFTP to transfer kernel image to RAM, save it to flash; use TFTP to transfer the device tree, save it to flash." Here we go: 

=> tftp 400000 172.17.255.118:MLNX-OS_PPC_M460EX_3.6.1002/boot/vmlinuz-uni
Using ppc_4xx_eth0 device
TFTP from server 172.17.255.118; our IP address is 172.17.255.120
Filename 'MLNX-OS_PPC_M460EX_3.6.1002/boot/vmlinuz-uni'.
Load address: 0x400000
Loading: #################################################################
         #################################################################
         ##
done
Bytes transferred = 1927366 (1d68c6 hex)
=> protect off FF000000 FF1FFFFF
................ done
Un-Protected 16 sectors
=> erase FF000000 FF1FFFFF

................ done
Erased 16 sectors
=> cp.b ${fileaddr} FF000000 ${filesize}
Copy to Flash... done
=> tftp 800000 172.17.255.118:MLNX-OS_PPC_M460EX_3.6.1002/boot/fdt-uni
Using ppc_4xx_eth0 device
TFTP from server 172.17.255.118; our IP address is 172.17.255.120
Filename 'MLNX-OS_PPC_M460EX_3.6.1002/boot/fdt-uni'.
Load address: 0x800000
Loading: #
done
Bytes transferred = 10017 (2721 hex)
=> cp.b ${fileaddr} FF1E0000 ${filesize}
Copy to Flash... done
=> protect on FF000000 FF1FFFFF
................ done
Protected 16 sectors
=> imls
Legacy Image at FF000000:
   Verifying Checksum ... OK
Legacy Image at FF400000:
   Verifying Checksum ... OK
Legacy Image at FF600000:
   Verifying Checksum ... OK
=>

File transfers have been finished. The Linux kernel and device tree files have been saved to flash. U-Boot has found the newly transferred image and thinks it's valid (this is what the imls was about.) Let us see what happens when it is run... 

=> run mlxlinux
Using mlnx460ex machine description
Linux version 3.10.27-MELLANOXuni-m460ex (@) (gcc version 4.7.2 (GCC) ) PPC_M460EX 3.6.1002 #1 2016-06-09 20:24:26
Zone ranges:
  DMA      [mem 0x00000000-0x2fffffff]
  Normal   empty
  HighMem  [mem 0x30000000-0x7fffffff]
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x00000000-0x7fffffff]
MMU: Allocated 1088 bytes of context maps for 255 contexts
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 522752
Kernel command line: root=/dev/mtdblock6 rootfstype=jffs2 ro reset_button=0 console=ttyS0,9600
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Sorting __ex_table...
Memory: 2075976k/2097152k available (3556k kernel code, 21176k reserved, 196k data, 109k bss, 148k init)
Kernel virtual memory layout:
  * 0xfffcf000..0xfffff000  : fixmap
  * 0xffc00000..0xffe00000  : highmem PTEs
  * 0xffa00000..0xffc00000  : consistent mem
  * 0xffa00000..0xffa00000  : early ioremap
  * 0xf1000000..0xffa00000  : vmalloc & ioremap
SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Preemptible hierarchical RCU implementation.
NR_IRQS:512 nr_irqs:512 16
UIC0 (32 IRQ sources) at DCR 0xc0
UIC1 (32 IRQ sources) at DCR 0xd0
UIC2 (32 IRQ sources) at DCR 0xe0
UIC3 (32 IRQ sources) at DCR 0xf0
clocksource: timebase mult[800000] shift[23] registered
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
devtmpfs: initialized
NET: Registered protocol family 16
256k L2-cache enabled
PCIE0: Checking link...
PCIE0: No device detected.
PCI host bridge /plb/pciex@d00000000 (primary) ranges:
 MEM 0x0000000e00000000..0x0000000e7fffffff -> 0x0000000080000000 
 MEM 0x0000000f00000000..0x0000000f000fffff -> 0x0000000000000000 
  IO 0x0000000f80000000..0x0000000f8000ffff -> 0x0000000000000000
4xx PCI DMA offset set to 0x00000000
4xx PCI DMA window base to 0x0000000000000000
DMA window size 0x0000000080000000
PCIE0: successfully set as root-complex
PCIE1: Checking link...
PCIE1: Device detected, waiting for link...
PCIE1: link is up !
PCI host bridge /plb/pciex@d20000000 (primary) ranges:
 MEM 0x0000000e80000000..0x0000000effffffff -> 0x0000000080000000 
 MEM 0x0000000f00100000..0x0000000f001fffff -> 0x0000000000000000 
  IO 0x0000000f80010000..0x0000000f8001ffff -> 0x0000000000000000
4xx PCI DMA offset set to 0x00000000
4xx PCI DMA window base to 0x0000000000000000
DMA window size 0x0000000080000000
PCIE1: successfully set as root-complex
PCI: Probing PCI hardware
PCI host bridge to bus 0000:40
pci_bus 0000:40: root bus resource [io  0xfffe0000-0xfffeffff] (bus address [0x0000-0xffff])
pci_bus 0000:40: root bus resource [mem 0xe00000000-0xe7fffffff] (bus address [0x80000000-0xffffffff])
pci_bus 0000:40: root bus resource [mem 0xf00000000-0xf000fffff] (bus address [0x00000000-0x000fffff])
pci_bus 0000:40: root bus resource [bus 40-ff]
PCI: Hiding 4xx host bridge resources 0000:40:00.0
pci 0000:40:00.0: PCI bridge to [bus 41-7f]
PCI host bridge to bus 0001:80
pci_bus 0001:80: root bus resource [io  0x0000-0xffff]
pci_bus 0001:80: root bus resource [mem 0xe80000000-0xeffffffff] (bus address [0x80000000-0xffffffff])
pci_bus 0001:80: root bus resource [mem 0xf00100000-0xf001fffff] (bus address [0x00000000-0x000fffff])
pci_bus 0001:80: root bus resource [bus 80-ff]
PCI: Hiding 4xx host bridge resources 0001:80:00.0
pci 0001:80:00.0: PCI bridge to [bus 81-bf]
pci 0000:40:00.0: disabling bridge window [io  0x0000-0xffffffffffffffff] to [bus 41-7f] (unused)
pci 0000:40:00.0: disabling bridge window [mem 0x00000000-0xffffffffffffffff pref] to [bus 41-7f] (unused)
pci 0000:40:00.0: disabling bridge window [mem 0x00000000-0xffffffffffffffff] to [bus 41-7f] (unused)
pci 0001:80:00.0: disabling bridge window [io  0x0000-0xffffffffffffffff] to [bus 81-bf] (unused)
pci 0000:40:00.0: PCI bridge to [bus 41-7f]
pci 0001:80:00.0: BAR 8: assigned [mem 0xe80000000-0xe800fffff]
pci 0001:81:00.0: BAR 0: assigned [mem 0xe80000000-0xe800fffff 64bit]
pci 0001:80:00.0: PCI bridge to [bus 81-bf]
pci 0001:80:00.0:   bridge window [mem 0xe80000000-0xe800fffff]
bio: create slab <bio-0> at 0
Switching to clocksource timebase
NET: Registered protocol family 2
TCP established hash table entries: 8192 (order: 4, 65536 bytes)
TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP: reno registered
UDP hash table entries: 512 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 512 (order: 1, 8192 bytes)
Configuring USB GPIO 16 + 19
bounce pool size: 64 pages
jffs2: version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
msgmni has been set to 1494
alg: No test for stdrng (krng)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
serial8250.0: ttyS0 at MMIO 0x4ef600300 (irq = 20) is a U6_16550A
console [ttyS0] enabled
4ef600300.serial: ttyS0 at MMIO 0x4ef600300 (irq = 20) is a 16550
brd: module loaded
4ff000000.nor_flash: Found 1 x16 devices at 0x0 in 16-bit bank. Manufacturer ID 0x000089 Chip ID 0x00881e
Intel/Sharp Extended Query Table at 0x010A
Intel/Sharp Extended Query Table at 0x010A
Intel/Sharp Extended Query Table at 0x010A
Intel/Sharp Extended Query Table at 0x010A
Intel/Sharp Extended Query Table at 0x010A
Using buffer write method
Using auto-unlock on power-up/resume
cfi_cmdset_0001: Erase suspend on write enabled
6 ofpart partitions found on MTD device 4ff000000.nor_flash
Creating 6 MTD partitions on "4ff000000.nor_flash":
0x000000000000-0x0000001e0000 : "KERNEL_1"
0x0000001e0000-0x000000200000 : "FDT_1"
0x000000200000-0x0000003e0000 : "KERNEL_2"
0x0000003e0000-0x000000400000 : "FDT_2"
0x000000f80000-0x000000fa0000 : "UBOOTENV"
0x000000fa0000-0x000001000000 : "UBOOT"
NAND device: Manufacturer ID: 0xec, Chip ID: 0xd3 (Samsung NAND 1GiB 3,3V 8-bit), 1024MiB, page size: 2048, OOB size: 64
2 NAND chips detected
Scanning device for bad blocks
Bad eraseblock 898 at 0x000007040000
Bad eraseblock 11548 at 0x00005a380000
Bad eraseblock 14499 at 0x000071460000
4 ofpart partitions found on MTD device 4e0000000.ndfc.nand
Creating 4 MTD partitions on "4e0000000.ndfc.nand":
0x000000000000-0x000020000000 : "ROOT_1"
0x000020000000-0x000040000000 : "ROOT_2"
0x000040000000-0x000046400000 : "CONFIG"
0x000046400000-0x00007c000000 : "VAR"
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
PPC 4xx OCP EMAC driver, version 3.54
MAL v2 /plb/mcmal, 2 TX channels, 16 RX channels
ZMII /plb/opb/emac-zmii@ef600d00 initialized
RGMII /plb/opb/emac-rgmii@ef601500 initialized with MDIO support
TAH /plb/opb/emac-tah@ef601350 initialized
TAH /plb/opb/emac-tah@ef601450 initialized
/plb/opb/emac-rgmii@ef601500: input 0 in RGMII mode
eth0: EMAC-0 /plb/opb/ethernet@ef600e00, MAC 00:02:c9:63:ef:ee
eth0: found Generic MII PHY (0x00)
/plb/opb/emac-rgmii@ef601500: input 1 in RGMII mode
eth1: EMAC-1 /plb/opb/ethernet@ef600f00, MAC 00:02:c9:63:ef:ef
eth1: found Generic MII PHY (0x01)
i2c /dev entries driver
ibm-iic 4ef600700.i2c: using standard (100 kHz) mode
rtc-ds1307 0-0068: SET TIME!
rtc-ds1307 0-0068: rtc core: registered ds1338 as rtc0
rtc-ds1307 0-0068: 56 bytes nvram
ibm-iic 4ef600800.i2c: using standard (100 kHz) mode
TCP: cubic registered
rtc-ds1307 0-0068: setting system clock to 2017-12-17 15:10:43 UTC (1513523443)
jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x00000000: 0xaa55 instead
jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x00000004: 0xaa55 instead
jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x00000008: 0xaa55 instead
jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x0000000c: 0xaa55 instead
jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x00000010: 0xaa55 instead
jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x00000014: 0xaa55 instead
jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x00000018: 0xaa55 instead
jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x0000001c: 0xaa55 instead
jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x00000020: 0xaa55 instead
jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x00000024: 0xaa55 instead
jffs2: Further such events for this erase block will not be printed
jffs2: Cowardly refusing to erase blocks on filesystem with no valid JFFS2 nodes
jffs2: empty_blocks 4094, bad_blocks 1, c->nr_blocks 4096
VFS: Cannot open root device "mtdblock6" or unknown-block(31,6): error -5
Please append a correct "root=" boot option; here are the available partitions:
1f00            1920 mtdblock0  (driver?)
1f01             128 mtdblock1  (driver?)
1f02            1920 mtdblock2  (driver?)
1f03             128 mtdblock3  (driver?)
1f04             128 mtdblock4  (driver?)
1f05             384 mtdblock5  (driver?)
1f06          524288 mtdblock6  (driver?)
1f07          524288 mtdblock7  (driver?)
1f08          102400 mtdblock8  (driver?)
1f09          880640 mtdblock9  (driver?)
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(31,6)
CPU: 0 PID: 1 Comm: swapper Not tainted 3.10.27-MELLANOXuni-m460ex PPC_M460EX
Call Trace(ef840000): name=swapper, state=0
[ef845e00] [c0005614] show_stack+0x54/0x150 (unreliable)
[ef845e40] [c02758d8] panic+0xcc/0x204
[ef845e90] [c0354d00] mount_block_root+0x274/0x284
[ef845ee0] [c0355048] prepare_namespace+0x1a0/0x1e8
[ef845ef0] [c0354930] kernel_init_freeable+0x1b0/0x1c4
[ef845f30] [c0001ac0] kernel_init+0x18/0xf8
[ef845f40] [c000afcc] ret_from_kernel_thread+0x5c/0x64
Rebooting in 180 seconds..

after a 3 minute delay, the switch resets (very noticeably -- the fans spin up to max RPM briefly)

Some interesting tidbits gleaned from this:

  • It's Linux 3.10.27 plus Mellanox modifications
  • 2Gibytes of RAM on board
  • U-Boot eventually passed root=/dev/mtdblock6 rootfstype=jffs2 ro reset_button=0 console=ttyS0,9600 as kernel command line arguments
  • There's a real time clock which seems to be functioning
  • And finally, Linux doesn't do much if it doesn't have a root filesystem -- if it can't start the first user-land process (init/upstart/systemd/whatever) it will panic.

User-mode stuffs (making a root filesystem for MLNX-OS)

Above we've noted that the kernel is being told to look for a JFFS2 image. We'll need to build one now. The JFFS2 creation utility, mkfs.jffs2 takes a directory containing the desired files as input and creates a JFFS2 image as output. (Much like creating an ISO9660 image.) On Debian, mkfs.jffs2 is contained in the mtd-utils package.

Remember that MLNX-OS image we downloaded from Mellanox's web site? And how we pulled a kernel and flattened device tree from it? Well now we're going to build something for the kernel to run.

Bits of shell script here documenting the contents of the JFFS2 directory 

# TAKE NOTE:
#   $jffs2root is the destination directory -- the jffs2 image file will be created from its contents
#   $mlnxosroot is the unpacked MLNX-OS distribution -- some files from here will be copied into ${jffs2root}
#
# Some of these items will need to be run with elevated privileges -- make use of sudo as needed.

cd ${jffs2root}
mkdir -p -m 775 bin dev etc/init.d etc/rc.d lib sbin var/run var/lib
mkdir -p -m 0 proc sys
mkdir -p -m 1777 tmp var/tmp
mkdir -p -m 700 root
cp -p ${mlnxosroot}/sbin/busybox bin/

# busybox decides what it should act as based on its argv[0], so all these symlinks to busybox let us use a single binary to do many different things
cd ${jffs2root}/bin
ln -s busybox bash
ln -s busybox cat
ln -s busybox chmod
ln -s busybox cp
ln -s busybox dmesg
ln -s busybox echo
ln -s busybox free
ln -s busybox getty
ln -s busybox grep
ln -s busybox gunzip
ln -s busybox gzip
ln -s busybox head
ln -s busybox hostname
ln -s busybox ifconfig
ln -s busybox login
ln -s busybox ln
ln -s busybox ls
ln -s busybox lsmod
ln -s busybox mkdir
ln -s busybox modinfo
ln -s busybox more
ln -s busybox mount
ln -s busybox mv
ln -s busybox ping
ln -s busybox poweroff
ln -s busybox ps
ln -s busybox reboot
ln -s busybox rm
ln -s busybox rmdir
ln -s busybox sh
ln -s busybox sleep
ln -s busybox sync
ln -s busybox tail
ln -s busybox tftp
ln -s busybox touch
ln -s busybox tty
ln -s busybox umount
ln -s busybox vi

# Linux kernel execs /sbin/init as process 0
cd ${jffs2root}/sbin
ln -s ../bin/busybox getty
ln -s ../bin/busybox init

# GNU tar is here instead of the busybox version
cd ${jffs2root}
cp -p ${mlnxosroot}/bin/tar bin/
cp -p ${mlnxosroot}/usr/sbin/flash_eraseall sbin/

# make an /etc/init.d/rcS that gets /proc and /sys mounted
cp /dev/null etc/init.d/rcS
(echo '#!/bin/sh'; echo; echo 'mount -t proc none /proc'; echo 'mount -t sysfs none /sys') | tee etc/init.d/rcS
chmod 755 etc/init.d/rcS
ln -s ../init.d/rcS etc/rc.d/rc.sysinit

# make an inittab file -- this will get us a working terminal
cp /dev/null etc/inittab
echo '::sysinit:/etc/init.d/rcS' >> etc/inittab
echo '#::askfirst:/bin/sh' >> etc/inittab
echo 'ttyS0:2345:respawn:/sbin/getty -L 9600 ttyS0 vt102' >> etc/inittab
chmod 644 etc/inittab

# /etc/passwd listing for 'root'  The password a classic UNIX (DES based) hash of "changeme" a common default root password on Sun gear.
cp /dev/null etc/passwd
echo 'root:Xxxgg2TS4senE:0:0::/root:/bin/sh' > etc/passwd
chmod 644 etc/passwd

# Copy name make symlinks for shared libraries (ldconfig doesn't want to help here.  :(  )
cd ${mlnxosroot}/lib
cp -p ld-2.17.so libc-2.17.so libcrypt-2.17.so libm-2.17.so libpthread-2.17.so librt-2.17.so ${jffs2root}/lib/
cd ${jffs2root}/lib
ln -s ld-2.17.so ld.so.1
ln -s ld-2.17.so ld-linux-so.2
ln -s libc-2.17.so libc.so.6
ln -s libcrypt-2.17.so libcrypt.so.1
ln -s libm-2.17.so libm.so.6
ln -s libpthread-2.17.so libpthread.so.0
ln -s librt-2.17.so librt.so.1

# device files (we don't need no steeenking udev)  These will definitely need root privileges to create.
cd ${jffs2root}/dev
mknod -m 622 console c 5 1
mknod -m 666 null c 1 3
mknod -m 444 random c 1 8
mknod -m 644 ttyS0 c 4 64
mknod -m 444 urandom c 1 9
mknod -m 666 zero c 1 5

And now, let's create the actual JFFS2 image file. 

cd ${jffs2root}/..
sudo /usr/sbin/mkfs.jffs2 --verbose --eraseblock=0x20000 --pagesize=2048 --pad \
    --no-cleanmarkers --big-endian --squash-uids --root=jffs2-root \
    --output=bespoke-mlnx-os-image-ppc-m460ex-20160609-202426.jffs2

Copy it to the TFTP server (172.17.255.118, remember?) And then, on the switch, see about pulling it down, flashing it, and trying to get the MLNX-OS kernel to run it. U-Boot conversation with the switch follows: 

U-Boot 2009.01 SX_PPC_M460EX SX_3.2.0330-82-EMC ppc (Feb 27 2013 - 12:13:42)

CPU:   AMCC PowerPC 460EX Rev. B at 1000 MHz (PLB=166, OPB=83, EBC=83 MHz)
       Security/Kasumi support
       Bootstrap Option H - Boot ROM Location I2C (Addr 0x52)
       Internal PCI arbiter disabled
       32 kB I-Cache 32 kB D-Cache
Board: Mellanox PPC460EX Board
FDEF:  No
I2C:   ready
DRAM:   2 GB (ECC enabled, 333 MHz, CL3)
FLASH: 16 MB
NAND:  1024 MiB
PCI:   Bus Dev VenId DevId Class Int
PCIE0: link is not up.
PCIE1: successfully set as root-complex
        01  00  15b3  c738  0c06  00
Net:   ppc_4xx_eth0, ppc_4xx_eth1
Hit any key to stop autoboot:  0 
=> ping 172.17.255.118
Waiting for PHY auto negotiation to complete... done
ENET Speed is 1000 Mbps - FULL duplex connection (EMAC0)
Using ppc_4xx_eth0 device
host 172.17.255.118 is alive
=> tftp 400000 172.17.255.118:MLNX-OS_PPC_M460EX_3.6.1002/bespoke-mlnx-os-image-ppc-m460ex-20160609-202426.jffs2
Using ppc_4xx_eth0 device
TFTP from server 172.17.255.118; our IP address is 172.17.255.120
Filename 'MLNX-OS_PPC_M460EX_3.6.1002/bespoke-mlnx-os-image-ppc-m460ex-20160609-202426.jffs2'.
Load address: 0x400000
Loading: ################################################################# 
         #################################################################
         ###############################
done
Bytes transferred = 2359296 (240000 hex)
=> nand erase clean 0 1FFFFFFF

NAND erase: device 0 offset 0x0, size 0x1fffffff
Skipping bad block at  0x07040000                                            
Erasing at 0x1ffe0000 -- 100% complete. Cleanmarker written at 0x1ffe0000.
OK
=> nand write ${fileaddr} 0 ${filesize}

NAND write: device 0 offset 0x0, size 0x240000
 2359296 bytes written: OK
=>

There was a single bad block of NAND reported on this switch. Small numbers of bad blocks in the NAND area are OK. The U-Boot tftp command filled out the fileaddr and filesize variables for us. Note that we cleared 512Mibytes of the flash storage with the nand erase ... command, but the file we transferred was only about 2.25Mibytes in size. Next up are 2 minor changes to the kernel command lines: 

=> printenv image_kernel_args jffs2_args
## Error: "image_kernel_args" not defined
jffs2_args=setenv bootargs root=${rootdev} rootfstype=jffs2 ro reset_button=${reset_button} ${image_kernel_args} ${extra_args}
=> setenv image_kernel_args loglevel=2
=> setenv jffs2_args "setenv bootargs root=${rootdev} rootfstype=jffs2 rw reset_button=${reset_button} ${image_kernel_args} ${extra_args}"
=>

And, to boot it up: 

=> run mlxlinux

#

And there's a root shell prompt in a very basic Linux OS running on the switch. The jiggering with getty in the inittab lets us have a fully configured terminal to play around with. Job control and having Ctrl-C are both very nice things.

Putting MLNX-OS on the switch

There is now a very basic Linux OS installed and bootable (with a little manual jiggering) Next up is getting MLNX-OS copied to one of the flash areas and some tweaking done.

We're going to put the MLNX-OS root filesystem in the "ROOT_2" flash partition -- cat /proc/mtd will list these out for us. We'll also be taking over the CONFIG and VAR partitions. Note that the flash layout information comes from the flattened device tree structure we pulled from the MLNX-OS distribution a long time ago.

Erase the MTD devices: 

# flash_eraseall -j /dev/mtd7
Erasing 128 Kibyte @ 1ffe0000 -- 99 % complete. Cleanmarker written at 1ffe0000.
# flash_eraseall -j /dev/mtd8
Erasing 128 Kibyte @ 63e0000 -- 99 % complete. Cleanmarker written at 63e0000.
# flash_eraseall -j /dev/mtd9
Erasing 128 Kibyte @ 13f60000 -- 37 % complete. Cleanmarker written at 13f60000.
Skipping bad block at 0x13f80000
Erasing 128 Kibyte @ 2b040000 -- 80 % complete. Cleanmarker written at 2b040000.
Skipping bad block at 0x2b060000
Erasing 128 Kibyte @ 35be0000 -- 99 % complete. Cleanmarker written at 35be0000.
#

Small numbers of bad blocks are expected and nothing to worry about.

Make some directories where we can deposit mount the soon-to-be-create filesystems: 

# mkdir -m 755 /mnt
# mkdir -m 0 /mnt/root2 /mnt/config /mnt/var

And mount the flash there. (JFFS2 magic: No mkfs step is needed) 

# mount -t jffs2 /dev/mtdblock7 /mnt/root2
# mount -f jffs2 /dev/mtdblock8 /mnt/config
# mount -t jffs2 /dev/mtdblock9 /mnt/var
# busybox df -m /mnt/*
Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/mtdblock8             100         2        98   2% /mnt/config
/dev/mtdblock7             512        11       501   2% /mnt/root2
/dev/mtdblock9             860        18       842   2% /mnt/var
#
Retrieved from "https://wiki.fnord.greeley.co.us/mediawiki/index.php?title=EMC_branded_Mellanox_SX60xx_switch&oldid=1074"