Difference between revisions of "Watchguard XTM 5"

From FnordWiki
Jump to navigation Jump to search
Line 130: Line 130:
 
address '''''tunnelclientIPv6address'''''
  
address '''''tunnelclientIPv6address'''''
 
endpoint '''''tunnelserverIPv4address''''' # There are many possible tunnel servers
  
endpoint '''''tunnelserverIPv4address''''' # There are many possible tunnel servers
−
local '''''tunnelclientIPv4address''''' # internet facing address of the local system (may not actually be required)
 +
# local '''''tunnelclientIPv4address''''' # internet facing address of the local system (may not actually be required)
 
ttl 255
  
ttl 255
 
gateway '''''tunnelserverIPv6address''''' # this will be the next hop for the IPv6 default route
  
gateway '''''tunnelserverIPv6address''''' # this will be the next hop for the IPv6 default route

Revision as of 19:44, 30 December 2017

Sticker on back says "WatchGuard XTM 5 series XTM 520"

hardware description

Fairly nifty little box:

  • More or less normal x86 Wintel board inside the box
  • 1x 10/100baseT port on the front (Labeled "0")
  • 6x 1000baseT ports on the front (Labeled "1" through "6")
  • 8P8C modular connector serial console port -- The BIOS is configured for serial port redirection. And a normal Cisco console cable works well.
  • 2x USB ports inside
  • motherboard has a single Socket 775 Celeron 440 CPU (not very exciting, but this will turn exciting later on)
  • 2x DDR2 DIMM sockets (non-ECC)
  • Compact Flash port (not sure largest capacity suppported -- The WatchGuard OS is delivered on a 1Gbyte CF card.
  • 2x SATA (SATA 2, probably, based on age) ports on motherboard.
  • 2x SATA power connectors included on the power supply
  • 1x DDR2 PC6400 1Gbyte DIMM

AFAICT, ***all*** Watchguard XTM 505, 510, 520, 530 have the same hardware inside. Differences are licensed capacity in the Watchguard OS. (But 515, 525, 535 are different in some way)

Modifications so far

  • Replaced DIMM with 2x 1Gbyte PC6400 DIMMs from a Dell Optiplex 740 2Gbytes of RAM
  • Removed CF card and make a backup image of its contents.
  • Completed the Xeon L5430 swap on 2016-08-28
  • 4Gbytes of RAM installed on 2016-09-12. Testing with memtester (user space, protected mode) looking good as of 2016-09-13.

Stuff on order

  • Xeon L5430 processor (4 cores, 2.66GHz clock, 12Mbytes of L2 cache) This is not a socket 775 part, but it is a Socket 771 package. And there are readily available kits for placing a Socket 771 Xeon in a Socket 775 system board. (Installed 2016-08-28)
  • Socket 771 CPU -> Socket 775 socket adapter ("Xeon inside" stickers included) (Installed 2016-08-28)
  • 4 and 8Gbyte CompactFlash cards (arrived 2016-08-27. Not in use yet)
  • 2x 2Gbyte PC2-6400 DDR2 800MHz DIMMs (bump the RAM from 2 -> Gbytes)

Progress toward Linuxification

  • Pulled 1Gbyte Transcend CompactFlash card out, made backup copy of its contents on desktop machine 2016-08-23
  • Located a working 160Gbyte 2.5inch SATA hard drive and commenced to wiping it and ...
  • wiped (with bits from /dev/urandom) said hard drive
  • partitioned and LVMed said hard drive
  • debootstraped Debian Jessie onto said hard drive: sudo debootstrap --components=main,contrib,non-free --verbose jessie target
  • chroot target bin/bash --login and run
apt-get update
apt-get install bash-completion vim-nox sudo openssh-server
groupadd -g 1000 adj
useradd -u 1000 -g 1000 -s /bin/bash -m adj
passwd adj
dpkg-reconfigure -p low debconf
apt-get install grub-pc
# make GRUB use serial console instead of VGA
apt-get install linux-image-3.16.0-4-amd64
  • Pulled hard drive, placed in WatchGuard box.
  • Realized totally forgot an /etc/fstab. We'll see what happens without...
  • Worse realization: No LVM tools in initramfs. (Seeing as I didn't install them inside the debootstrapped chroot. D'oh!)
  • DANGER: Voodoo! Creating LVM logical volume device mapper mappings without the LVM tools is not a good idea (but it works if you have a good reference for the tables somewhere)
(initramfs) dmsetup create vg00-root --table "0 2097152 linear 8:2 2048"
(initramfs) dmsetup create vg00-tmp --table "0 8388608 linear 8:2 4196352"
(initramfs) dmsetup create vg00-var --table "0 8388608 linear 8:2 12584960"
(initramfs) dmsetup create vg00-usr --table "0 8388608 linear 8:2 20973568"
mount -t ext4 /dev/dm-0 /root
mount -t ext4 /dev/dm-1 /root/tmp
mount -t ext4 /dev/dm-2 /root/var
mount -t ext4 /mnt/dm-3 /root/usr
chroot /root /bin/bash --login

(yay for built-in journal playback. And for forgetting to vgchange -an vg00 on the machine that ran the debootstrap.

  • Add an entry to /etc/network/interfaces and ifup eth1
  • create a good /etc/fstab
  • Make sure /boot is mounted (update-initramfs needs to write here)
  • apt-get install lvm2
  • rm /etc/mtab; ln -s /proc/mounts /etc/mtab
  • reboot
  • apt-get install locales
  • Fix up network device names in /etc/udev/rules.d/70-persistent-net.rules (my box's ports are numbered 0-6 on the front and have increasing MAC addresses to correspond)
  • apt-get install pciutils flashrom
  • Fix hostname and /etc/hosts file

At this point, Debian is installed and mostly configured.

Success!

BIOS update

WatchGuard's BIOS allows the appliances owner to change only the date and time. This is not cool. Fortunately, there is an awesome guy out there called stephenw10 who has done a bit of hackery on the BIOS file which lets owners own their boxes. :)

Process: 

sudo flashrom --verbose --programmer internal -r stock_bios_image.bin        # save a backup of the stock BIOS bits
wget --no-check-certificate 'https://sites.google.com/site/pfsensefirebox/home/xtm5_83.rom?attredirects=0&d=1'    # the correct cert doesn't exist in Debian 8 for some reason.  Works fine in Chrome/Firefox
mv xtm5_83.rom\?attredirects\=0\&d\=1 xtm5_83.rom                            # a much less stupid filename
sudo flashrom --verbose --programmer internal -w xtm5_83.rom                 # write StephenW10's modded BIOS image to the flash chip
sudo shutdown -h now                                                         # power off system
  • then remove the coin cell battery for a few minutes (or find the "clear CMOS settings" jumber)
  • While we're at it, pull out the (mostly pointless now) Cavium Networks crypto accelerator card
  • Play a couple rounds of Candy Crush waiting for the CMOS RAM to lose its bits
  • Re-install battery (orientation matters here!)
  • Apply power
  • Turn on machine
  • Hit <TAB> on the serial console a couple of times (Not <DEL>)
  • Enjoy happy access to all of the BIOS setup menus

Special thanks to StephenW10 of the PFsense world for the BIOS image, lots of info about the hardware, and general awesomeness.

Other good packages

apt-get install iucode-tool intel-microcode        # patch the CPU
apt-get install iperf                              # network performance testing

Making it a router

IPv6 software needed

apt-get install radvd ndisc6        # IPv6 tools

IPv6 tunnel to Hurricane Electric/Tunnelbroker.net

Sign up for tunnelbroker.net account. Add stanza like the following to /etc/network/interfaces: 

# each tunnelbroker.net tunnel has a unique integer ID
auto ipv6-tun-tunnelid
iface ipv6-tun-tunnelid inet6 v4tunnel
        # Our IPv6 connection, provided by Hurricane Electric
        # The tunnel info can be updated by fecthing (curl or similar)
        #   https://tunnelbrokerusername:tunnelbrokerupdatekey@ipv4.tunnelbroker.net/nic/update?hostname=tunnelid
        address tunnelclientIPv6address
        endpoint tunnelserverIPv4address   # There are many possible tunnel servers
        # local tunnelclientIPv4address    # internet facing address of the local system (this is not actually be required)
        ttl 255
        gateway tunnelserverIPv6address    # this will be the next hop for the IPv6 default route

IPv6 forwarding enablement

Linux kernel settings for IPv6 forwarding a confusingly different than for IPv4 forwarding. :( The net.ipv6.conf.all.forwarding sysctl must be enabled for any forwarding to occur at all. This might be best set as a post-up option on the tunnel interface by adding a line to it like so: 

# each tunnelbroker.net tunnel has a unique integer ID
auto ipv6-tun-tunnelid
iface ipv6-tun-tunnelid inet6 v4tunnel
        # Our IPv6 connection, provided by Hurricane Electric
        # The tunnel info can be updated by fecthing (curl or similar)
        #   https://tunnelbrokerusername:tunnelbrokerupdatekey@ipv4.tunnelbroker.net/nic/update?hostname=tunnelid
        address  tunnelclientIPv6address
        endpoint tunnelserverIPv4address   # There are many possible tunnel servers
        # local    tunnelclientIPv4address # internet facing address of the local system (may not actually be required)
        ttl      255
        gateway  tunnelserverIPv6address   # this will be the next hop for the IPv6 default route
        post-up  /sbin/sysctl -w net.ipv6.conf.all.forwarding=1
        post-up  /sbin/sysctl -w net.ipv6.conf.ipv6-tun-tunnelid.forwarding=1
        pre-down /sbin/sysctl -w net.ipv6.conf.ipv6-tun-tunnelid.forwarding=0

OpenVPN

apt-get install openvpn

Network performance numbers

Installed iperf above. Let's see how many bits we can push through the network...

Over eth0, the 100baseT port: 

adj@debiwall:~$ iperf -c 172.16.0.1 -t 180
------------------------------------------------------------
Client connecting to 172.16.0.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 172.16.0.55 port 35942 connected with 172.16.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-180.0 sec  1.97 GBytes  94.1 Mbits/sec
adj@debiwall:~$

This is over one of the GigE ports: 

adj@debiwall:~$ iperf -c 172.16.0.1 -t 180
------------------------------------------------------------
Client connecting to 172.16.0.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 172.16.0.44 port 43872 connected with 172.16.0.1 port 5001
[ 6059.633499] perf: interrupt took too long (3161 > 3151), lowering kernel.perf_event_max_sample_rate to 63250
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-180.0 sec  19.7 GBytes   941 Mbits/sec
adj@debiwall:~$

So pretty much line rate for IPv4 over both of those. On to IPv6: 

adj@debiwall:~$ iperf -V -c 2001:470:ba93:10::1 -t 180
------------------------------------------------------------
Client connecting to 2001:470:ba93:10::1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 2001:470:ba93:10:290:7fff:fe86:1e77 port 47714 connected with 2001:470:ba93:10::1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-180.0 sec  1.95 GBytes  92.8 Mbits/sec
adj@debiwall:~$

Down by 1.3Mbits/sec for IPv6. Probably just about the overheard of the larger layer 3 headers. Seems perfectly fine still 

adj@debiwall:~$ iperf -V -c 2001:470:ba93:10::1 -t 180
------------------------------------------------------------
Client connecting to 2001:470:ba93:10::1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 2001:470:ba93:10:290:7fff:fe86:1e78 port 59108 connected with 2001:470:ba93:10::1 port 5001
[ 7489.422197] perf: interrupt took too long (3972 > 3951), lowering kernel.perf_event_max_sample_rate to 50250
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-180.0 sec  19.5 GBytes   928 Mbits/sec
adj@debiwall:~$

Line rate again with IPv6. (Much nicer on the power bill than the PE1950, too.)

Firewall beginnings

apt-get install nftables        # it's maybe more better-er that iptables ?
apt-get install vlan            # Yes, we do have lots of Ethernet interfaces, but sometimes one cable is plenty
apt-get install tcpdump         # packet captures are super useful!
apt-get install fail2ban        # wouldn't want a firewall that can't automatically add rules to block abusive systems, would you?
apt-get install screen          # best .. tool .. ever   (Unless you're a tmux person.  Then just s/screen/tmux/)
apt-get install lldpd           # Lets our switches know what is plugged into the port (and which switch port we are plugged in to)
apt-get install smartmontools   # There's a hard drive in the box.  So let's hopefully get some notice before it dies.
apt-get install lm-sensors      # Tell us if we are going to overheat.  (run sensors-detect after this is done)
apt-get install fancontrol      # Ramps fans up when we move .. in .. to .. the .. dangerzone!
apt-get install haproxy/jessie-backports        # This machine is going to be the TLS (SSL) endpoint for a number of services.  The version of HAproxy in jessie-backports includes LUA support and can be used for zero-downtime Let'sEncrypt certificate updates

nftables resources

CPU Upgrade, anyone?

Celeron 440 is not exactly a screaming piece of computing hardware. But it is what's in there now. 35 Watt TDP, one core, no hyperthreading, 2.0GHz clock. There are people out there who have discovered that a Socket 771 Xeon can be modded to fit in the Socket 775 on the board. Nicest available Socket 771 Xeon is the Xeon L5430 -- 50 Watt TDP, 4 cores, no hyperthreading, 2.66GHz clock. (There is a Xeon L3360 CPU in a Socket 775 (no need for adapter) with 4 cores a faster clock and a 65 Watt TDP, but it costs 4 times as much on eBay when it is available.) So L5430 it is...

Credit for this idea goes to people on the pfsense forums.

See http://www.delidded.com/lga-771-to-775-adapter/ for tutorial/references.

Start state

adj@debiwall:~$ hostname; echo; date; echo; lscpu; echo; dmesg | grep -i bogo
debiwall

Sun Aug 28 18:39:56 UTC 2016

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                1
On-line CPU(s) list:   0
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 22
Model name:            Intel(R) Celeron(R) CPU          440  @ 2.00GHz
Stepping:              1
CPU MHz:               1994.993
BogoMIPS:              3989.98
L1d cache:             32K
L1i cache:             32K
L2 cache:              512K
NUMA node0 CPU(s):     0 

[    0.012013] Calibrating delay loop (skipped), value calculated using timer frequency.. 3989.98 BogoMIPS (lpj=7979972)
[    0.220003] smpboot: Total of 1 processors activated (3989.98 BogoMIPS)
adj@debiwall:~$

mumblemumblemumbleperformCPUtranslplant

It's on youtube! Thank you for the assist, kids!

Post upgrade state

adj@debiwall:~$ hostname; echo; date; echo; lscpu; echo; dmesg | grep -i bogo
debiwall

Sun Aug 28 23:07:22 UTC 2016

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 23
Model name:            Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
Stepping:              10
CPU MHz:               2659.953
BogoMIPS:              5320.28
L1d cache:             32K
L1i cache:             32K
L2 cache:              6144K
NUMA node0 CPU(s):     0-3

[    0.012010] Calibrating delay loop (skipped), value calculated using timer frequency.. 5319.90 BogoMIPS (lpj=10639812)
[    0.468002] smpboot: Total of 4 processors activated (21280.82 BogoMIPS)
adj@debiwall:~$
Retrieved from "https://wiki.fnord.greeley.co.us/mediawiki/index.php?title=Watchguard_XTM_5&oldid=963"