Latest revision as of 00:36, 21 December 2014

BIOS Settings

Turn on virtualization, IOMMU
Lots of options regarding console redirection over serial. Need to investigate.

IPMI

Watch for IPMI setup message during boot. Press Ctrl-E when prompted. Settings as follows:

Static IP: 172.16.0.241
Netmask:   255.255.255.0
Gateway:   0.0.0.0

Reset user credentials.

TODO: Investigate crypto key stuff, alerts, other settings.

What is this good for? Well,

ipmitool -H 172.16.0.241 -U Admin power on

will turn on the server from somewhere on the network. A network attached serial console should be possible, too. And

ipmitool -H 172.16.0.241 -U Admin power off

will turn it off.

ipmitool -H 172.16.0.241 -U Admin sol activate

will conect to a serial console where you can interact with GRUB and the Linux kernel console, and log in via a serial connection on /dev/ttyS1.

There are a bunch of other possibilities, too.

Updating firmware

Should be doable using the PXE boot firmware update procedure found elsewhere in this wiki. (Replace with a real link.)

Debian Install image locations

http://cdimage.debian.org/debian-cd/6.0.3/amd64/iso-cd/debian-6.0.3-amd64-netinst.iso

non-Free firmware for Ethernet cards

PowerEdge 1950 machines have 2 on-board Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet adapters. Firmware for these adapters is non-free and must be loaded to perform the Debian installation.

An ISO image is available at http://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-firmware/current/amd64/iso-cd/ which should include the bits necessary to use these adapters during the OS installation. This is the Netinst ISO but it includes the non-free firmware. Saves on getting the non-free firmware installed using USB keys, floppies, or other media.

http://wiki.debian.org/Firmware has a discussion of issues surrounding the non-free firmware needed by these systems and its use in Debian.

Install process

Proceed as normal with keyboard, setting up network, hostname, etc.

Partition disks by using parted in a shell. Here's what sda looks like now:

~ # parted /dev/sdb
GNU Parted 2.3
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit s                                                           
unit s
(parted) print                                                            
print
Model: ATA WDC WD6400AAKS-0 (scsi)
Disk /dev/sda: 1250263728s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start     End       Size      File system  Name                     Flags
 1      2048s     526335s   524288s                gulik_sda_efi_boot       bios_grub
 2      526336s   2623487s  2097152s               gulid_sda_mdraid_boot    raid
 3      2623488s  4720639s  2097152s               gulik_sda_mdraid_root    raid

(parted)

After this, we create and start up two software raid devices (the "missing" bit lets us create a software raid device before having all the actual pieces):

~ # mdadm --create gulik_mdraid_boot --chunk 128 --level 1 --raid-devices=2 --run --name=gulik_mdraid_boot /dev/sda2 /dev/sdb2
~ # mdadm --create gulik_mdraid_root --chunk 128 --level 1 --raid-devices=2 --run --name=gulik_mdraid_root /dev/sda3 /dev/sdb3

Post-install tweaks

Prevent apt-get from automatically installing Recommended packages

Miminal software installation is good! Put the following into /etc/apt/apt.conf:

APT {
        Install-Recommends "0";
}

USB drivers for the initramfs

Add the following to /etc/initramfs-tools/modules:

hid
ehci_hcd
uhci_hcd

Run update-initramfs when done to add these modules to the pre-init runtime environment.

Add backports.org packages

Add the following to /etc/apt/sources.list:

deb http://backports.debian.org/debian-backports squeeze-backports main contrib non-free

Install a recent kernel from backports.org

Not yet ready to build a custom kernel here, so install something current from backports.org instead.

# linux-image-3.2.0-0.bpo.1-amd64 is the most recent kernel package available at backports.org
apt-get install -t squeeze-backports linux-image-3.2.0-0.bpo.1-amd64

Network console

Motivation

It's nice being able for fix problems on the machine without standing in front of it with a keyboard and monitor.

IPMI Serial Over LAN connects a serial port to the network. It can be talked to with ipmitool.

Serial access for the boot loader and tell Linux it has a serial console

Set following in /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS1,57600n8"
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=57600 --unit=1 --word=8 --parity=no --stop=1"

http://www.cyberciti.biz/faq/howto-setup-serial-console-on-debian-linux/ has a good tutorial on this.

After editing /etc/default/grub, run /usr/sbin/update-grub to write a new bootloader configuration file (/boot/grub/grub.cfg for instance).

Note that the IPMI Serial over LAN redirection on the PowerEdge 1950 redirects COM2, which Linux calls /dev/ttyS1 and GRUB calls "serial unit 1".

Also note: Make sure the BIOS console redirection is set to sent to COM2. It's easy to miss this and get nothing on your IPMI serial over LAN until the boot loader starts writing to the serial port.

Allow logins on the serial port

We need a getty to listen for a user on the serial port. Add the following to /etc/inittab:

S1:2345:respawn:/sbin/getty -L ttyS1 57600

And run "/sbin/init Q" to tell init to re-read its configuration file (/etc/inittab).

Disk encryption, more software RAID, and LVM setup

End goal

Two physical drives (sda and sdb)
Boot from unencrypted, software RAID1 (mirrored) /boot partitions
initramfs does "cyptsetup luksOpen" on large partition 4 of each physical drive
mdadm is then run and assembles a software RAID 1 from the encrypted partitions
the assembled RAID 1 device contains a LUKS partition. This LUKS device contains an LVM physical volume.
Logical volumes are LUKS devices containing filesystems
three layers of crypto here:
- LVM lvols are LUKS devices
- LVM physical volume(s) is (are) LUKS devices
- MD software RAID members are LUKS devices

Open questions

Hot to get passphrases to "cryptsetup luksOpen" ?
What's a reasonable set of stacked ciphers?
How much usable space does the md layer metadata consume?
- Appears to be 576 kibytes for a 2 disk RAID-1 array, 1Gibyte in size, with v1.2 metadata, and a 256kibyte chunk size.
How much usable space does the LUKS layer metadata consume?

Procedure

Squeeze's cryptsetup and associated packages are woefully out of date. I have pulled in cryptsetup and its dependencies from unstable and will hope the lack of security updates does not bite me in the ass.
Create partition number 4 on each of the disks. These will be LUKS devices that will be assembled into a software RAID1.

Partition table now looks like so:

Number  Start     End         Size        File system  Name                            Flags
 1      2048s     526335s     524288s                  gulik_sda_efi_boot              bios_grub
 2      526336s   2623487s    2097152s                 gulik_sda_mdraid_boot           raid
 3      2623488s  4720639s    2097152s                 gulik_sda_mdraid_root           raid
 4      4720640s  143374615s  138653976s               gulik_sda_luks_mdraid_member_0

Splat random bits onto these devices like so:

sudo dd if=/dev/urandom of=/dev/sda4 bs=1M
sudo dd if=/dev/urandom of=/dev/sdb4 bs=1M

Feel free to skip this step if desired. The risk here is that old, unencrypted data might be recoverable from sectors that are never written to.

Also note that /dev/urandom can be pretty slow (about 4Mbytes/sec on this machine). Assuming there are 4Mibytes of free space in /tmp, here's a much faster method:

dd if=/dev/urandom bs=1024 count=4096 of=/tmp/randfile
(while true; do cat /tmp/randfile; done) | sudo dd bs=1M of=/dev/sda4
(while true; do cat /tmp/randfile; done) | sudo dd bs=1M of=/dev/sdb4

Generate keys for these devices:

# 64 bytes of random data, printed in hex is 128 characters and 512 bits, which should be a decent sized passphrase.
# openssl might take a long, long, long time to get its RNG seeds from /dev/random, so be patient
# what's with the "printf %s $(...)" funny business?  It prevents a trailing newline being fed to gpg's stdin
printf %s $(openssl rand -hex -rand /dev/random 64) | gpg --symmetric -a -v --s2k-mode 3 --s2k-count 65011712 -o gulik_sda4_passphrase.gpg
printf %s $(openssl rand -hex -rand /dev/random 64) | gpg --symmetric -a -v --s2k-mode 3 --s2k-count 65011712 -o gulik_sdb4_passphrase.gpg

perhaps multiple keys should go into a single GPG encrypted file, one per line, with a key of the block device
Put a LUKS header on the devices:

sudo cryptsetup -c aes-xts-plain64:sha512 -h SHA512 -y -s 256 --iter-time 5000 --align-payload=2048 luksFormat /dev/sda4
sudo cryptsetup -c aes-xts-plain64:sha512 -h SHA512 -y -s 256 --iter-time 5000 --align-payload=2048 luksFormat /dev/sdb4

Add some modules to the initramfs. Add the following lines to the end of /etc/initramfs-tools/modules:

dm_mod
dm_crypt
cbc
aes_generic
aes_x86_64
cryptd
sha256_generic
sha512_generic
gf128mul
xts

Update /etc/crypttab so that the LUKS device is opened at boot time. So that mdadm can create the RAID1 array. So that cryptsetup can open the LVM volume group's physical volume. So that vgchange can activate vg00. So that cryptsetup can then open the LUKS encrypted logical volumes and make those available to the rest of the system. Put the following in /etc/crypttab:

gulik_mdraid_vg00_pv_0_member_0       sda4            none            luks

Run update-initramfs before rebooting so that the needed drivers are available

sudo update-initramfs -v -u

Create a software RAID device on our LUKS volumes:

sudo mdadm --create gulik_softraid_vg00_pv_0 --metadata=1.2 \
    --level=1 --raid-devices=2 --name gulik_softraid_vg00_pv_0 \
    /dev/mapper/gulik_softraid_vg00_pv_member_0 missing
# (We'll add another physical disk, LUKS format it, and add it to the MD array at some later time.)

Put a LUKS volume on top of the software RAID device like so:

sudo cryptsetup -c twofish-xts-plain64:wp512 -h WHIRLPOOL -y -s 512 --iter-time 10000 --align-payload=2048 luksFormat /dev/md/gulik_mdraid_luks_vg00_pv_0

Open the new LUKS volume so we can start putting LVM structures on it:

sudo cryptsetup luksOpen /dev/md/gulik_mdraid_luks_vg00_pv_0 gulik_vg00_pv_0

Add appropriate entry to /etc/crypttab:

gulik_vg00_pv_0                 /dev/md/gulik_mdraid_luks_vg00_pv_0     none            luks

Run pvcreate:

sudo pvcreate --metadatatype 2 --pvmetadatacopies 2 -v -Z y --dataalignment 1M /dev/mapper/gulik_vg00_pv_0

Add two more modules to /etc/initramfs-tools/modules:

twofish_x86_64
wp512

Rebuild initramfs:

sudo update-initramfs -v -v -u

Create a volume group using the physical volume that was just created:

sudo vgcreate -l 255 -M 2 -p 255 -s 256M -v vg00 /dev/mapper/gulik_vg00_pv_0

Create logical volumes, put LUKS on them, open LUKS volumes, mkfs them:

lvcreate -v -L 1024M -n LUKSroot vg00
lvcreate -v -L 4096M -n LUKSswap vg00
lvcreate -v -L 4096M -n LUKStmp vg00
lvcreate -v -L 4096M -n LUKSusr vg00
lvcreate -v -L 4096M -n LUKSvar vg00
lvcreate -v -L 4096M -n LUKShome vg00
cryptsetup -c anubis-cbc-essiv:rmd256 -h tiger -y -s 256 --iter-time 10000 --align-payload=2048 luksFormat /dev/vg00/LUKSroot
cryptsetup -c anubis-cbc-essiv:rmd256 -h tiger -y -s 256 --iter-time 10000 --align-payload=2048 luksFormat /dev/vg00/LUKSswap
cryptsetup -c anubis-cbc-essiv:rmd256 -h tiger -y -s 256 --iter-time 10000 --align-payload=2048 luksFormat /dev/vg00/LUKStmp
cryptsetup -c anubis-cbc-essiv:rmd256 -h tiger -y -s 256 --iter-time 10000 --align-payload=2048 luksFormat /dev/vg00/LUKSusr
cryptsetup -c anubis-cbc-essiv:rmd256 -h tiger -y -s 256 --iter-time 10000 --align-payload=2048 luksFormat /dev/vg00/LUKSvar
cryptsetup -c anubis-cbc-essiv:rmd256 -h tiger -y -s 256 --iter-time 10000 --align-payload=2048 luksFormat /dev/vg00/LUKShome
cryptsetup luksOpen /dev/vg00/LUKSroot vg00-root
cryptsetup luksOpen /dev/vg00/LUKSswap vg00-swap
cryptsetup luksOpen /dev/vg00/LUKStmp vg00-tmp
cryptsetup luksOpen /dev/vg00/LUKSusr vg00-usr
cryptsetup luksOpen /dev/vg00/LUKSvar vg00-var
cryptsetup luksOpen /dev/vg00/LUKShome vg00-home
mkfs.ext3 -c -c -b 4096 -i 8192 -m 10 -j -O has_journal,ext_attr,resize_inode,dir_index,filetype,sparse_super,large_file /dev/mapper/vg00-root
mkfs.ext3 -c -c -b 4096 -i 8192 -m 10 -j -O has_journal,ext_attr,resize_inode,dir_index,filetype,sparse_super,large_file /dev/mapper/vg00-tmp
mkfs.ext3 -c -c -b 4096 -i 8192 -m 10 -j -O has_journal,ext_attr,resize_inode,dir_index,filetype,sparse_super,large_file /dev/mapper/vg00-usr
mkfs.ext3 -c -c -b 4096 -i 8192 -m 10 -j -O has_journal,ext_attr,resize_inode,dir_index,filetype,sparse_super,large_file /dev/mapper/vg00-var
mkfs.ext3 -c -c -b 4096 -i 8192 -m 10 -j -O has_journal,ext_attr,resize_inode,dir_index,filetype,sparse_super,large_file /dev/mapper/vg00-home
mkswap -c /dev/mapper/vg00-swap

Difference between revisions of "PowerEdge 1950 Debian install"