Proliant G8 Servers

From FnordWiki
Jump to navigation Jump to search

June 2017, acquired a pair of HP Proliant DL380e G8 LFF servers.

The plan for these is turn them into Ceph + hypervisor systems.

hardware overview

  • 2U rackmount server
  • 12x 3.5inch HDD slots in the front
  • 2 sockets for Intel Xeon E5-24xx processors (original or v2)
  • 6 DDR3 DIMM sockets per processor (12 total, so 96 Gbytes of RAM if fully populated with 8Gbyte DIMMs)
  • 4x Intel chipset GigE ports on the back
  • ILO 4 system management

cool things

  • Can be found cheaper than Dell R510s now
  • Faster CPUs than an R510 takes (one or 2 generations newer)
  • 8 core, 16 thread, 1.8 GHz E5 CPUs can be had for under $35
  • heat sinks included in the ~ $200 price

uncool things

  • It's not UEFI based
  • Not sure on-motherboard SATA adapter will drive all 12 disk slots in front
  • Gen 8 3.5inch disk trays are hard to come by
  • Only 12 DIMM slots -- these are not the thing for half Tbyte in memory databases
  • E5-24xx CPUs are a little harder to find than E5-26xx

Goals for these

  • After it's racked and cabled, touchless firmware updates
  • Touchless OS install
    • There is no optical drive, so PXE booting and Fully Automated Install to be done.
    • To include cfengine or salt or chef or puppet or similar
  • Encrypt all the bits
  • Encrypt all the bits (on the disks)
  • Encrypt all the bits (on the wires)
  • One Ceph OSD per spindle

Parts!

  • 3.5 inch HDD tray is HP(E) part number 651314-001 or 651320-001. Available from many eBay sellers for < $10, shipped
  • 684900-001 is a kit for putting 2x 2.5inch (SFF) hot swap drives in the back of the chassis
  • Power supplies are the HP "Common Slot" models used in G7 and G8 Proliant servers
  • 684896-001 is a PCIe riser board with an LSI SAS controller and two (SFF 8087, I think) connectors located toward the front of the chassis

Firmware Updates

HP(E) provides these all bundled up together in a package called "Support Pack for Proliant" or "SPP". Annoyingly, these are only available to organizations with a support contract, but intrepid individuals can find them with some Googling.

SPP comes as an ISO that can be put on a USB flash drive (it's too big for a DVD). But it also includes handy instructions inside the ISO filesystem for deploying it via PXE.

Several pieces need doing to get SPP PXE booting:

  • DHCP server running on subnet the server sits in
  • DHCP server configured to tell the client server "Go PXE boot from here"
  • TFTP server configured to serve files to the PXE client
  • NFS to serve files to the PXE-booted Proliant

DHCP setup for PXE clients

  • Find MAC address for Ethernet 1 in ProLiant server.
    • Does not appear to be available from the ILO command line session. :(
    • So instead, connect to ILO SSH session and do this:
</>hpiLO-> start /system1

status=0
status_tag=COMMAND COMPLETED
Wed Jun 21 18:15:43 2017



Server powering on .......



</>hpiLO-> start /system1/oemhp_vsp1

Virtual Serial Port Active: COM2

Starting virtual serial port.
Press 'ESC (' to return to the CLI Session.

[... many things about server booting will scroll by ...]

Press "F9" key for ROM-Based Setup Utility
Press "F10" key for Intelligent Provisioning
Press "F11" key for Default Boot Override Options
Press "F12" key for Network Boot
For access via BIOS Serial Console
Press "ESC+9" for ROM-Based Setup Utility
Press "ESC+0" for Intelligent Provisioning
Press "ESC+!" for Default Boot Override Options
Press "ESC+@" for Network Boot

Esc @ (to get the PXE client running)

Wait for 

CLIENT MAC ADDR: 14 58 D0 5B 12 64  GUID: 32393636 3735 584D 5135 313330335353

or something very like it. The ProLiant's Ethernet 1 interface MAC address is shown. Power server off like so: 

Esc (
</>hpiLO-> stop /system1

status=0
status_tag=COMMAND COMPLETED
Wed Jun 21 18:28:21 2017



Server powering off .......



</>hpiLO->

And a server definition to the DHCP server's config file for it. Something like: 

host proliant-380e-g8-MXQ51303SS {
        # HP ProLiant DL380e Gen 8 s/n MXQ51303SS
        hardware ethernet 14:58:d0:5b:12:64;
        fixed-address 10.20.199.33;
        option host-name "proliant-380e-g8-mxQ51303ss";
        # un-comment the next two lines to PXEboot this system
        next-server 10.20.199.4;
        filename "pxelinux.0";
        # comment the previous two lines disable PXEboot of this system
}

which pretty much says "You should request pxelinux.0 from 10.20.199.4 and then run it." The EFI PXE boot image file has a different name "bootnext64.efi" or something like that. But ProLiants aren't EFI until Gen 9.

Make sure there's a TFTP server installed on the machine listed in the DHCP "next-server" option. And then, if the TFTP server happens to be atftpd as packed in Debian 8, make sure the daemon command line options are correct: --verbose 5 does not do what one might think. Use --verbose=5 instead.

Some fiddling will be needed to get the PXElinux binary files where the TFTP server can get them: 

$ sudo apt-get install pxelinux
$ sudo ln -s /usr/lib/PXELINUX/pxelinux.0 /srv/tftp/
$ sudo ln -s /usr/lib/syslinux/modules/bios/ldlinux.c32 /srv/tftp/
$ sudo ln -s /usr/lib/syslinux/modules/bios/libutil.c32 /srv/tftp/
$ sudo ln -s /usr/lib/syslinux/modules/bios/menu.c32 /srv/tftp/
Retrieved from "https://wiki.fnord.greeley.co.us/mediawiki/index.php?title=Proliant_G8_Servers&oldid=946"