Backups

Backups would be nice. And there is an excess of hardware to do them to at home.

sacredchao to be master server and do backups to USB connected removable drives.

bitkeeper to power on as needed and get replicas of backup sets from sacredchao stored on local disks

honkingbigtapelibrary to power on as needed and get replicas of bitkeeper 's replicas.

Mom's Windows PC to run storage daemon and do backups to locally attached external USB drive. (Should be fun running a storage daemon under Windows, right?)

youngmi's linux box to also run storage daemon, back up to external USB drive and replicate backup sets to Greeley.

overoverkill? You decide.

deciding on bareos

Bareos is a fork of Bacula. Bacula does all kinds of nice things, but the development model seems to have gone the open core route. Which is fine, but I don't feel a great need to spend dollars on a Windows bacula client, when I can get the Bareos Windows client for free. Bareos also has a nifty feature that allows replications from one storage daemon to another, simplifying the off-siting strategy I'd like to implement.

bareos director setup

• Build bareos binaries for needed platforms:
  • source distribution for Debianized sources is http://download.bareos.org/bareos/release/latest/Debian_7.0/
  • Unable to find source packages for the fastlz library this references, I skipped it. Removed references to it from source tree, and made appropriate comment in debian/changelog.
• Install a director somewhere. sacredchao seems like a good choice.
  • The director is in charge of things. It has the database of backups, job definitions, and all that sort of stuff.
  • Support for three different databases is available. I'll take SQLite over PostgreSQL over MySQL, given the option. (No extra layer of database engine needed (just a tiny library), no additional services exposed to the network, very transparent (UNIX filesystem permissions) security model.)
  • director package installation command was: sudo dpkg -i bareos-director_13.2.2-7.1.fnord.0_amd64.deb bareos-database-common_13.2.2-7.1.fnord.0_amd64.deb bareos-database-sqlite3_13.2.2-7.1.fnord.0_amd64.deb bareos-common_13.2.2-7.1.fnord.0_amd64.deb bareos-database-tools_13.2.2-7.1.fnord.0_amd64.deb

Director configuration bits from /etc/bareos/bareos-dir.conf:

 Director {                      # define myself
        Name = fnord-dir        # the organization is "fnord".  The director is an organization-wide resource, name it a
        QueryFile = "/usr/lib/bareos/scripts/query.sql"
        Maximum Concurrent Jobs = 1
        Password = "a big string from 'pwgen -sync 48'"   # Console password ("pwgen -sync 48" made this)
        Messages = Daemon

        # remove comment in next line to load plugins from specified directory
        # Plugin Directory = /usr/lib/bareos/plugins

        DirAddresses = {        # we have a number of these.  Let's not listen to Internet-facing addresses
                ipv4 = {
                        addr = 172.16.0.1;
                }
                ipv4 = {
                        addr = 10.255.248.1;
                }
                ipv6 = {
                        addr = 2001:470:ba93:10::1;
                }
                ipv6 = {
                        addr = 2001:470:1f0f:5be::1;
                }
        }
}

Trying to start the director by running the init script now fails. Some poking around found nothing recorded by syslog. Tracing the init script eventually got me to running the director by hand to look for output...

adj@sacredchao:/var/log$ sudo /usr/sbin/bareos-dir -t -v
bareos-dir: dird.c:1087-0 Could not open Catalog "MyCatalog", database "bareos".
bareos-dir: dird.c:1092-0 sqlite.c:172 Database /var/lib/bareos/bareos.db does not exist, please create it.
13-Jan 16:36 bareos-dir ERROR TERMINATION
Please correct configuration file: bareos-dir.conf
adj@sacredchao:/var/log$ 

That's actually quite instructive. Let's see about sorting that out. I really think I should call the database after the organization (fnord).

Edit the director config file, and fix up the Catalog section as follows:

# Generic catalog service
Catalog {
        Name = FnordCatalog
        # Uncomment the following lines if you want the dbi driver
        # dbdriver = "dbi:postgresql"; dbaddress = 127.0.0.1; dbport =
        #dbdriver = "postgresql"
        dbdriver = "sqlite3"
        dbname = "fnord-catalog"        # was "bareos", but it ought to be named after the organization.   
        dbuser = "bareos"
        dbpassword = ""
}

Also find any other references to "MyCatalog" and replace with "FnordCatalog". Run the director in test mode again, and look for output like this:

adj@sacredchao:/etc/bareos$ sudo /usr/sbin/bareos-dir -t -v
bareos-dir: dird.c:1087-0 Could not open Catalog "FnordCatalog", database "fnord-catalog".
bareos-dir: dird.c:1092-0 sqlite.c:172 Database /var/lib/bareos/fnord-catalog.db does not exist, please create it.
14-Jan 10:32 bareos-dir ERROR TERMINATION
Please correct configuration file: bareos-dir.conf
adj@sacredchao:/etc/bareos$

So, how to create the SQLite catalog database? There's a handy dandy script to be found in /usr/lib/bareos/scripts helpfully called "create_sqlite3_database". Run that while in the /var/lib/bareos directory. (It assumes you're root, but should also work as user bareos. Expect the chown at the end to fail, though.) Follow up by running /usr/lib/bareos/scripts/make_sqlite3_tables. Then rename /var/lib/bareos/bareos.db to /var/lib/bareos/fnord-catalog.db. Running the director daemon in test mode (-t) should now report no issues:

adj@sacredchao:/var/lib/bareos$ sudo /usr/sbin/bareos-dir -t -v
adj@sacredchao:/var/lib/bareos$

storage daemon setup

Bareos's storage daemons accept the clients' files and put them on some sort of (hopefully) reliable long term storage medium.

My Debian package for the storage daemon is bareos-storage_13.2.2-7.1.fnord.0_amd64.deb. Support for tape libraries is in the bareos-storage-tape_13.2.2-7.1.fnord.0_amd64.deb package. Since sacredchao doesn't have tape attached, we'll just install bacula-storage for now and set up some USB attached disk for storage.

Since we renamed the director from "sacredchao-dir" to "fnord-dir", it is important to make a matching change in the storage daemon's config.

Important bits of storage daemon config:

Storage {               # definition of myself
        Name = sacredchao-sd            # this name identifies a storage daemon inside the org.  Name it after the serv
        Maximum Concurrent Jobs = 20

        # remove comment in next line to load plugins from specified directory
        # Plugin Directory = /usr/lib/bareos/plugins
}

Director {
        Name = fnord-dir                # important that this matches the director's name.  See http://doc.bareos.org/master/html/bareos-manual-main-reference.html#AuthorizationErrors
        Password = "BIG_PASSWORD_FROM_pwgen_-sync_48"           # Be sure to stick this in the director's config, too.
}

Device {
        Name = FileStorage
        Media Type = File
        Archive Device = /var/lib/bareos/storage
        LabelMedia = yes;                   # lets Bareos label unlabeled media
        Random Access = Yes;
        AutomaticMount = yes;               # when device opened, read it
        RemovableMedia = no;
        AlwaysOpen = no;
}

This storage daemon has one storage device defined. It's called "FileStorage" and it writes files under /var/lib/bareos/storage. We'll want to get that on a drive that isn't a backup source sometime in the near future.

Clients? That's where all the data live.

We've got a working director to make decisions now. And a storage daemon to write our backups to. How about being able to suck the files out of a system? The piece that does that is called a file daemon. It runs on each client.

Install by installing the bareos-filedaemon package and update bareos-fd.conf to make the director's name and password match. Also update the bareos-dir.conf to put in a correct password for the file daemon.

Let the two jobs the director has currently defined (back up /usr/sbin and back up the catalog) run and come back in the morning.

Setting up file storage

So, sadly, last night's backups failed. The director was able to talk to the file daemon (client) and the storage daemon, but the storage daemon wasn't really ready to write to the FileStorage device. It seems that it must be labelled before being used. One uses the Bareos console to make a labeling operation happen. Install the bareos-bconsole package and edit the director's name amd password in /etc/bareos/bconsole.conf (check them against the director's config file) to enable the console to manage the director.

Fiddle about inside bconsole quite a bit now to create a working, writable backup target.

Add another client

Because that's where the files are. :)

Install bareos-common and bareos-filedaemon (and any prerequisites) on client system. Update /etc/bareos/bareos-fd.conf (on the client) to fix name and password for the director.

Then, on the director machine, add the client's details to bacula-dir.com. Add a Client block to define the let the director know the client exists. Also add a Job block to get an entry into the schedule. This latter will need a distinct name and Client = line.

security

Well, if security is management of risk, keeping copies of one's bits is all about increasing security. We'll concentrate on a few specific areas. First, preventing an adversary with control of our network connections from pretending to be part of the backup system and from siphoning off all our bits while they move between backup system components. Next up will be details on preventing someone who can walk off with our backup media from reading all the files they'd ever want.

for data in motion

Bacula has provisions for TLS-ecapsulated communication between the various components. Bareos has kept these intact. This functionality allows the director, storage daemons, and file daemons to all be sure they are communicating with other components they should be communicating with, that a third party (hello, NSA) is unable to examine the content of the communication, and that the communications have survived the trip over the network without being tampered with.

So let's create a public key infrastructure to for this:

1. openssl genrsa -aes256 -out bareos-ca-root-key.pem 8192 # creates a private key for our root CA, and encrypts the PEM-encoded key with AES256-CBC using a user provided passphrase. Don't lose the passphrase!
2. openssl req -x509 -sha512 -new -days $((365 * 20 + 5)) -key bareos-ca-root-key.pem -out bareos-ca-root-certificate.pem -config openssl-bareos-CA.conf # creates a self-signed CA certificate. You will be asked for the passphrase for bareos-ca-root-key.pem. Ths CA certificate is good for 20 years (20 * 365 + 5 leap days) and is hashed with SHA512.
3. mkdir "Bareos PKI Root Certificate Authority"; cd $_ # this will be the root of our certificate authority's files
4. touch ca.db.index; echo $(($RANDOM % 90 + 10)) > ca.db.rand; echo 01 > ca.db.serial; mkdir ca.db.certs # our CA is going to keep files in here.