A10 AX3000 firewall conversion

From FnordWiki
Revision as of 19:32, 11 January 2019 by Adj (talk | contribs) (→‎Ethernet bypass port frustrations)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

console

Serial console with 8P8C modular connector on front of box. Works with Cisco console cables. (Yay for that.) 9600bps, 8n1.

Hardware inside

Processor 1x Intel Xeon X5670
RAM 24 Gibytes ECC RAM

It wants to boot ACOS!

Not much chance to hit the keyboard escape. If you're not having any luck with that, pop the top cover (3 screws on each side, and one at the rear edge of the top side, near the center) and pull the CMOS battery, wait, and reverse the procedure. The BIOS settings seem to be considered good if the machine boots at least as far as finding the boot loader from the hard drive.

Base OS install

debootstrap to a drive using another system. There is room for a pair of thin 2.5 inch SATA disks, so plan ahead and make a RAID. Firmware is BIOS without an UEFI option, but GNU GRUB can boot an GPT (EFI usual) partitioned disk on a BIOS system just fine.

put random bits onto the disk

Assuming the disk appears as sdg on the host system: 

$ sudo shred -n 1 -v /dev/sdg
[wait however long is needed]

disk partition table

Now that random bits are all over the disk, let's partition it. I like GNU parted for this. 

$ sudo parted /dev/sdg
(parted) unit s
(parted) mktable gpt
(parted) mkpart bios_grub 2048 2099199
(parted) set 1 bios_grub on
(parted) mkpart mdraid_boot 2099200 4196351
(parted) mkpart mdraid_vg00_pv_0 4196352 167772160
(parted) p
Model: WDC WDBN CE2500PNC-WRSN (scsi)
Disk /dev/sdg: 488397168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start     End         Size        File system  Name              Flags
 1      2048s     2099199s    2097152s                 bios_grub         bios_grub
 2      2099200s  4196351s    2097152s                 mdraid_boot
 3      4196352s  167772160s  163575809s               mdraid_vg00_pv_0

(parted) q

RAID!

Well, half a RAID. The new drive has 3 partitions on it. First will be used to keep some GNU Grub bootloader bits handy. Next is a Linux software RAID (MD) device for /boot. And at the end comes another RAID partition that will hold a Linux LVM physical volume which will be the backing store for the vg00 volume group. So here we're making 2 RAID devices each missing their second halves. (Maybe we'll get another disk in there eventually.) 

$ sudo mdadm --create warthog-0-md_boot --metadata=1.2 --name=warthog-0-md_boot --bitmap=internal --level=1 --raid-devices=2 /dev/sdg2 missing
mdadm: array /dev/md/warthog-0-md_boot started.
$ sudo mdadm --create warthog-0-vg00_pv_0 --metadata=1.2 --name=warthog-0-vg00-pv-0 --bitmap=internal --level=1 --raid-devices=2 /dev/sdg3 missing
mdadm: array /dev/md/warthog-0-vg00_pv_0 started.

Logical volume manager bits and bobs

I'm all old school. I like LVM. I really like LVM. 

$ sudo pvcreate --verbose --metadatatype 2 --pvmetadatacopies 2 /dev/md/warthog-0-vg00_pv_0
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Wiping signatures on new PV /dev/md/warthog-0-vg00_pv_0.
    Set up physical volume for "/dev/md/warthog-0-vg00_pv_0" with 163444736 available sectors.
    Zeroing start of device /dev/md/warthog-0-vg00_pv_0.
    Writing physical volume data to disk "/dev/md/warthog-0-vg00_pv_0".
  Physical volume "/dev/md/warthog-0-vg00_pv_0" successfully created.
$ sudo vgcreate --verbose --metadatatype 2 --vgmetadatacopies 2 --physicalextentsize 256M warthog-0-vg00 /dev/md/warthog-0-vg00_pv_0
    Wiping internal VG cache
    Wiping cache of LVM-capable devices
    Wiping signatures on new PV /dev/md/warthog-0-vg00_pv_0.
    Adding physical volume '/dev/md126' to volume group 'warthog-0-vg00'
    Archiving volume group "warthog-0-vg00" metadata (seqno 0).
    Creating volume group backup "/etc/lvm/backup/warthog-0-vg00" (seqno 1).
  Volume group "warthog-0-vg00" successfully created
$

Volume group created. On to some logical volumes to hold filesystems to hold things: 

$ sudo lvcreate --verbose --name root --size 1024M warthog-0-vg00
    Archiving volume group "warthog-0-vg00" metadata (seqno 1).
    Creating logical volume root
    Creating volume group backup "/etc/lvm/backup/warthog-0-vg00" (seqno 2).
    Activating logical volume warthog-0-vg00/root.
    activation/volume_list configuration setting not defined: Checking only host tags for warthog-0-vg00/root.
    Creating warthog--0--vg00-root
    Loading warthog--0--vg00-root table (254:8)
    Resuming warthog--0--vg00-root (254:8)
    Wiping known signatures on logical volume "warthog-0-vg00/root"
    Initializing 4.00 KiB of logical volume "warthog-0-vg00/root" with value 0.
  Logical volume "root" created.
$ sudo lvcreate --verbose --name swap --size 4096M warthog-0-vg00
    Archiving volume group "warthog-0-vg00" metadata (seqno 2).
    Creating logical volume swap
    Creating volume group backup "/etc/lvm/backup/warthog-0-vg00" (seqno 3).
    Activating logical volume warthog-0-vg00/swap.
    activation/volume_list configuration setting not defined: Checking only host tags for warthog-0-vg00/swap.
    Creating warthog--0--vg00-swap
    Loading warthog--0--vg00-swap table (254:9)
    Resuming warthog--0--vg00-swap (254:9)
    Wiping known signatures on logical volume "warthog-0-vg00/swap"
    Initializing 4.00 KiB of logical volume "warthog-0-vg00/swap" with value 0.
  Logical volume "swap" created.
$ sudo lvcreate --verbose --name tmp --size 4096M warthog-0-vg00
    Archiving volume group "warthog-0-vg00" metadata (seqno 3).
    Creating logical volume tmp
    Creating volume group backup "/etc/lvm/backup/warthog-0-vg00" (seqno 4).
    Activating logical volume warthog-0-vg00/tmp.
    activation/volume_list configuration setting not defined: Checking only host tags for warthog-0-vg00/tmp.
    Creating warthog--0--vg00-tmp
    Loading warthog--0--vg00-tmp table (254:10)
    Resuming warthog--0--vg00-tmp (254:10)
    Wiping known signatures on logical volume "warthog-0-vg00/tmp"
    Initializing 4.00 KiB of logical volume "warthog-0-vg00/tmp" with value 0.
  Logical volume "tmp" created.
$ sudo lvcreate --verbose --name var --size 8192M warthog-0-vg00
    Archiving volume group "warthog-0-vg00" metadata (seqno 4).
    Creating logical volume var
    Creating volume group backup "/etc/lvm/backup/warthog-0-vg00" (seqno 5).
    Activating logical volume warthog-0-vg00/var.
    activation/volume_list configuration setting not defined: Checking only host tags for warthog-0-vg00/var.
    Creating warthog--0--vg00-var
    Loading warthog--0--vg00-var table (254:11)
    Resuming warthog--0--vg00-var (254:11)
    Wiping known signatures on logical volume "warthog-0-vg00/var"
    Initializing 4.00 KiB of logical volume "warthog-0-vg00/var" with value 0.
  Logical volume "var" created.
$ sudo lvcreate --verbose --name usr --size 4096M warthog-0-vg00
    Archiving volume group "warthog-0-vg00" metadata (seqno 5).
    Creating logical volume usr
    Creating volume group backup "/etc/lvm/backup/warthog-0-vg00" (seqno 6).
    Activating logical volume warthog-0-vg00/usr.
    activation/volume_list configuration setting not defined: Checking only host tags for warthog-0-vg00/usr.
    Creating warthog--0--vg00-usr
    Loading warthog--0--vg00-usr table (254:12)
    Resuming warthog--0--vg00-usr (254:12)
    Wiping known signatures on logical volume "warthog-0-vg00/usr"
    Initializing 4.00 KiB of logical volume "warthog-0-vg00/usr" with value 0.
  Logical volume "usr" created.
$ sudo lvcreate --verbose --name home --size 4096M warthog-0-vg00
    Archiving volume group "warthog-0-vg00" metadata (seqno 6).
    Creating logical volume home
    Creating volume group backup "/etc/lvm/backup/warthog-0-vg00" (seqno 7).
    Activating logical volume warthog-0-vg00/home.
    activation/volume_list configuration setting not defined: Checking only host tags for warthog-0-vg00/home.
    Creating warthog--0--vg00-home
    Loading warthog--0--vg00-home table (254:13)
    Resuming warthog--0--vg00-home (254:13)
    Wiping known signatures on logical volume "warthog-0-vg00/home"
    Initializing 4.00 KiB of logical volume "warthog-0-vg00/home" with value 0.
  Logical volume "home" created.
$ sudo lvcreate --verbose --name opt --size 1024M warthog-0-vg00
    Archiving volume group "warthog-0-vg00" metadata (seqno 7).
    Creating logical volume opt
    Creating volume group backup "/etc/lvm/backup/warthog-0-vg00" (seqno 8).
    Activating logical volume warthog-0-vg00/opt.
    activation/volume_list configuration setting not defined: Checking only host tags for warthog-0-vg00/opt.
    Creating warthog--0--vg00-opt
    Loading warthog--0--vg00-opt table (254:14)
    Resuming warthog--0--vg00-opt (254:14)
    Wiping known signatures on logical volume "warthog-0-vg00/opt"
    Initializing 4.00 KiB of logical volume "warthog-0-vg00/opt" with value 0.
  Logical volume "opt" created.
$

And some filesystems now that we have logical volumes: 

$ for dev in /dev/warthog-0-vg00/*; do sudo mkfs.ext4 -Fq ${dev}; done
$

"Aha!" you exclaim. "You just put an ext4 FS on the swap device." Yes, I did. To fix that... 

$ sudo mkswap /dev/warthog-0-vg00/swap
mkswap: /dev/warthog-0-vg00/swap: warning: wiping old ext4 signature.
Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
no label, UUID=4443088b-d2dd-4cbb-8b78-cd85b06115fc
$

Almost ready for the install

We need a place to put all the newly created filesystems so that debootstrap can write to them. Let's do that and mount the things... 

$ sudo mkdir -m 0 /media/warthog-0
$ sudo mount /dev/warthog-0-vg00/root /media/warthog-0
$ sudo mkdir -v -m 0 /media/warthog-0/{boot,dev,home,opt,proc,run,sys,tmp,usr,var}
mkdir: created directory '/media/warthog-0/boot'
mkdir: created directory '/media/warthog-0/dev'
mkdir: created directory '/media/warthog-0/home'
mkdir: created directory '/media/warthog-0/opt'
mkdir: created directory '/media/warthog-0/proc'
mkdir: created directory '/media/warthog-0/run'
mkdir: created directory '/media/warthog-0/sys'
mkdir: created directory '/media/warthog-0/tmp'
mkdir: created directory '/media/warthog-0/usr'
mkdir: created directory '/media/warthog-0/var'
$ sudo mount /dev/md/warthog-0-md_boot /media/warthog-0/boot
$ sudo mount /dev/warthog-0-vg00/home /media/warthog-0/home
$ sudo mount /dev/warthog-0-vg00/opt /media/warthog-0/opt
$ sudo mount /dev/warthog-0-vg00/tmp /media/warthog-0/tmp
$ sudo chmod 1777 /media/warthog-0/tmp
$ sudo mount /dev/warthog-0-vg00/tmp /media/warthog-0/tmp
$ sudo mount /dev/warthog-0-vg00/usr /media/warthog-0/usr
$ sudo mount /dev/warthog-0-vg00/var /media/warthog-0/var
$ sudo mount -t proc none /media/warthog-0/proc 
$ sudo mount -t sysfs none /media/warthog-0/sys

Put an OS in it

Wherein debootstrap does its thing... 

adj@new-optiplex:/media$ sudo debootstrap --include=grub-pc,lvm2,mdadm,ssh,sudo,vim-nox,linux-image-4.9.0-6-amd64,man-db,less stretch warthog-0/
I: Retrieving InRelease 
I: Retrieving Release 
I: Retrieving Release.gpg 
I: Checking Release signature
I: Valid Release signature (key id 067E3C456BAE240ACEE88F6FEF0F382A1A7B6500)
I: Retrieving Packages 
I: Validating Packages 
[... many things elided ...]
I: Configuring systemd...
I: Configuring ca-certificates...
I: Configuring initramfs-tools...
I: Base system installed successfully.
$

Additional necessary tweaks

Make sure /dev, /proc, and /sys are available inside the target root directory: 

$ sudo mount --bind /dev /media/warthog-0/dev
$ sudo mount -t proc /media/warthog-0/proc
$ sudo mount -t sys none /media/warthog-0/sys

Link /etc/mtab to /proc/self/mounts so that df and other things wanting to know about mounted filesystems work: 

$ sudo ln -s ../proc/self/mounts /media/warthog-0/etc/mtab

Enable serial console for the boot loader and the Linux kernel: 

$ sudo sed -i.orig -e 's/^GRUB_TIMEOUT=5$/GRUB_TIMEOUT=20/g' /media/warthog-0/etc/default/grub
$ sudo sed -i -e 's/^GRUB_CMDLINE_LINUX_DEFAULT="quiet"$/GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,9600"/g' /media/warthog-0/etc/default/grub
$ echo 'GRUB_TERMINAL=serial' | sudo tee -a /media/warthog-0/etc/default/grub
$ echo 'GRUB_SERIAL_COMMAND="serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1"' | sudo tee -a /media/warthog-0/etc/default/grub

set its host name

$ echo warthog-0 | sudo tee /media/warthog-0/etc/hostname

a list of filesystems is important, too

Else we get a panic at boot. 

$ cat << _EOF_
/dev/warthog-0-vg00/root					/		ext4		errors=remount-ro		0	0
/dev/warthog-0-vg00/home					/home		ext4		defaults			0	0
/dev/warthog-0-vg00/opt					/opt		ext4		defaults			0	0
/dev/warthog-0-vg00/tmp					/tmp		ext4		defaults			0	0
/dev/warthog-0-vg00/usr					/usr		ext4		defaults			0	0
/dev/warthog-0-vg00/var					/var		ext4		defaults			0	0
/dev/warthog-0-vg00/swap					none		swap		sw				0	0
UUID=75dfffb8-1ab6-494d-a1bc-d70d30a135ba	/boot		ext4		defaults			0	0
__EOF__ | sudo tee -a /media/warthog-0/etc/fstab
$

make it bootable

Enter the chroot, put grub onto the new drive: 

$ sudo chroot /media/warthog-0 /bin/bash --login
root@new-optiplex:/# grub-install /dev/sdg
Installing for i386-pc platform.
device node not found
device node not found
device node not found
device node not found
device node not found
device node not found
device node not found
device node not found
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
device node not found
Installation finished. No error reported.
root@new-optiplex:/# update-grub

make a user with admin privileges

Because it will be nice to say "root is locked" and still be able to log in, right? 

root@new-optiplex:/# adduser --home /home/itops --shell /bin/bash --uid 1000 --ingroup itops --gecos "IT operations account" itops
Adding user `itops' ...
Adding new user `itops' (1000) with group `itops' ...
Creating home directory `/home/itops' ...
Copying files from `/etc/skel' ...
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
root@new-optiplex:/# adduser itops sudo
Adding user `itops' to group `sudo' ...
Adding user itops to group sudo
Done.
root@new-optiplex:/#

Clean up /etc/passwd, /etc/group, and related files: 

root@new-optiplex:/# pwck -s
root@new-optiplex:/# grpck -s

un-mount all the stuffs

root@new-optiplex:/# umount -av
root@new-optiplex:/# exit
$ sudo umount /media/warthog-0
$ sudo vgchange -a n warthog-0-vg00
$ sudo mdadm --stop /dev/md/warthog-0-md_boot
$ sudo mdadm --stop /dev/md/warthog-0-vg00-pv-0

Make it go!

Remove drive from installation host system. Install in A10 Networks chassis Re-install CMOS battery Close up the box Attach serial console (Cisco console cables work for this) Configure terminal program (Kermit or similar) Apply power Watch for GRUB menu after Let Linux boot If all goes well, there will be a friendly getty asking for a user name (log-in with account created above)

Make it network

Create an /etc/network/interfaces.d/enp21s0 file with these contents: 

allow-hotplug enp21s0
auto enp21s0
iface enp21s0 inet dhcp

(enp21s0 is just what the new network interface name scheme calls the "MGMT" port on the front the chassis)

Make it less loud

It really is quite loud. Obnoxiously so. Let us see about making it less so. 

$ sudo apt-get install lm-sensors fancontrol
$ sudo sensors-detect
[...]
$ sudo pwmconfig
[go through the all the menus]
$

No luck with this. Will have to see if ACOS does anything to moderate the fan speeds and if it can be pulled into a less specialised Linux distro.

There is a BIOS setting that allows control of the fans by the OS. This needs to be set appropriately. Afterwards pwmconfig kinda works.

Lots of network interfaces, I need a map!

Front panel name Linux name
MGMT enp21s0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17 enp18s0f0 (10Gbits/sec SFP+)
18 enp18s0f1 (10Gbits/sec SFP+)
19 enp19s0f0 (10Gbits/sec SFP+)
20 enp19s0f1 (10Gbits/sec SFP+)

This looks like it's missing some stuff, doesn't it? Well, need to get the drivers for the Ethernet bypass ports sorted out. (Because things can't be too easy.)

Important Note: The stock Debian stretch 4.9 Linux kernel was unable to get the links on the 10GbE ports to come up. Upgrading to a 4.16 backports kernel had good results.

Other Debian packages you will want to install

  • less
  • pciutils
  • iucode-tool intel-microcode (these come from contrib and non-free)

Ethernet bypass port frustrations

Along with the fans... The 1000baseT ports numbered 1 through 8 are being recalcitrant. In the ACOS image, there is a custom driver for the chips called e1000_a10e. As far as I can tell, the A10 ACOS software doesn't really care about LAN bypass stuff at all and pretty much just turns on all the ports as separate units. (The LAN bypass idea here is that if power to the system goes away, a pair of ports can be electrically bound together and let traffic flow as though it were just a cable. A possibly nice feature in a firewall. But the hardware isn't standard and has to be dealt with.)

But... Lanner (who makes the system for A10) does provide source for working with the LAN bypass modules. Watch this space for updates.

Lanner provides a hopefully useful manual here: https://www.lannerinc.com/category/1202-network-appliances?download=1720

Update 2019-01-11: No forward progress on this, unfortunately. I have verified that the 1000baseT cards have Lanner Gen 2 bypass modules and should be controllable with the Lanner wdbp or wd_bp program. I have not had success getting them to leave bypass mode. :( I **do** have an email correspondence with a Lanner rep and have asked for a stock (not A10 custom) BIOS image for the machine, which **should** be able to tweak the bypass settings at POST time.

In the mean time, the eight 1000baseT ports and the eight GigE SFP ports are unused, the Comcast/Xfinity cable modem is connected to the switch and the Comcast/Xfinity WAN VLAN is passed to the A10 device on one of its 10G SFP+ ports.

Retrieved from "https://wiki.fnord.greeley.co.us/mediawiki/index.php?title=A10_AX3000_firewall_conversion&oldid=1068"