Backups

From FnordWiki
Jump to navigation Jump to search

Backups would be nice. And there is an excess of hardware to do them to at home.

sacredchao to be master server and do backups to USB connected removable drives.

bitkeeper to power on as needed and get replicas of backup sets from sacredchao stored on local disks

honkingbigtapelibrary to power on as needed and get replicas of bitkeeper 's replicas.

Mom's Windows PC to run storage daemon and do backups to locally attached external USB drive. (Should be fun running a storage daemon under Windows, right?)

youngmi's linux box to also run storage daemon, back up to external USB drive and replicate backup sets to Greeley.

overoverkill? You decide.

deciding on bareos

Bareos is a fork of Bacula. Bacula does all kinds of nice things, but the development model seems to have gone the open core route. Which is fine, but I don't feel a great need to spend dollars on a Windows bacula client, when I can get the Bareos Windows client for free. Bareos also has a nifty feature that allows replications from one storage daemon to another, simplifying the off-siting strategy I'd like to implement.

bareos director setup

  • Build bareos binaries for needed platforms:
  • Install a director somewhere. sacredchao seems like a good choice.
    • The director is in charge of things. It has the database of backups, job definitions, and all that sort of stuff.
    • Support for three different databases is available. I'll take SQLite over PostgreSQL over MySQL, given the option. (No extra layer of database engine needed (just a tiny library), no additional services exposed to the network, very transparent (UNIX filesystem permissions) security model.)
    • director package installation command was: sudo dpkg -i bareos-director_13.2.2-7.1.fnord.0_amd64.deb bareos-database-common_13.2.2-7.1.fnord.0_amd64.deb bareos-database-sqlite3_13.2.2-7.1.fnord.0_amd64.deb bareos-common_13.2.2-7.1.fnord.0_amd64.deb bareos-database-tools_13.2.2-7.1.fnord.0_amd64.deb

Director configuration bits from /etc/bareos/bareos-dir.conf: 

 Director {                      # define myself
        Name = fnord-dir        # the organization is "fnord".  The director is an organization-wide resource, name it a
        QueryFile = "/usr/lib/bareos/scripts/query.sql"
        Maximum Concurrent Jobs = 1
        Password = "a big string from 'pwgen -sync 48'"   # Console password ("pwgen -sync 48" made this)
        Messages = Daemon

        # remove comment in next line to load plugins from specified directory
        # Plugin Directory = /usr/lib/bareos/plugins

        DirAddresses = {        # we have a number of these.  Let's not listen to Internet-facing addresses
                ipv4 = {
                        addr = 172.16.0.1;
                }
                ipv4 = {
                        addr = 10.255.248.1;
                }
                ipv6 = {
                        addr = 2001:470:ba93:10::1;
                }
                ipv6 = {
                        addr = 2001:470:1f0f:5be::1;
                }
        }
}

Trying to start the director by running the init script now fails. Some poking around found nothing recorded by syslog. Tracing the init script eventually got me to running the director by hand to look for output... 

adj@sacredchao:/var/log$ sudo /usr/sbin/bareos-dir -t -v
bareos-dir: dird.c:1087-0 Could not open Catalog "MyCatalog", database "bareos".
bareos-dir: dird.c:1092-0 sqlite.c:172 Database /var/lib/bareos/bareos.db does not exist, please create it.
13-Jan 16:36 bareos-dir ERROR TERMINATION
Please correct configuration file: bareos-dir.conf
adj@sacredchao:/var/log$

That's actually quite instructive. Let's see about sorting that out. I really think I should call the database after the organization (fnord).

Edit the director config file, and fix up the Catalog section as follows: 

# Generic catalog service
Catalog {
        Name = FnordCatalog
        # Uncomment the following lines if you want the dbi driver
        # dbdriver = "dbi:postgresql"; dbaddress = 127.0.0.1; dbport =
        #dbdriver = "postgresql"
        dbdriver = "sqlite3"
        dbname = "fnord-catalog"        # was "bareos", but it ought to be named after the organization.   
        dbuser = "bareos"
        dbpassword = ""
}

Also find any other references to "MyCatalog" and replace with "FnordCatalog". Run the director in test mode again, and look for output like this: 

adj@sacredchao:/etc/bareos$ sudo /usr/sbin/bareos-dir -t -v
bareos-dir: dird.c:1087-0 Could not open Catalog "FnordCatalog", database "fnord-catalog".
bareos-dir: dird.c:1092-0 sqlite.c:172 Database /var/lib/bareos/fnord-catalog.db does not exist, please create it.
14-Jan 10:32 bareos-dir ERROR TERMINATION
Please correct configuration file: bareos-dir.conf
adj@sacredchao:/etc/bareos$

So, how to create the SQLite catalog database? There's a handy dandy script to be found in /usr/lib/bareos/scripts helpfully called "create_sqlite3_database". Run that while in the /var/lib/bareos directory. (It assumes you're root, but should also work as user bareos. Expect the chown at the end to fail, though.) Follow up by running /usr/lib/bareos/scripts/make_sqlite3_tables. Then rename /var/lib/bareos/bareos.db to /var/lib/bareos/fnord-catalog.db. Running the director daemon in test mode (-t) should now report no issues: 

adj@sacredchao:/var/lib/bareos$ sudo /usr/sbin/bareos-dir -t -v
adj@sacredchao:/var/lib/bareos$

storage daemon setup

Bareos's storage daemons accept the clients' files and put them on some sort of (hopefully) reliable long term storage medium.

My Debian package for the storage daemon is bareos-storage_13.2.2-7.1.fnord.0_amd64.deb. Support for tape libraries is in the bareos-storage-tape_13.2.2-7.1.fnord.0_amd64.deb package. Since sacredchao doesn't have tape attached, we'll just install bacula-storage for now and set up some USB attached disk for storage.

Since we renamed the director from "sacredchao-dir" to "fnord-dir", it is important to make a matching change in the storage daemon's config.

Important bits of storage daemon config: 

Storage {               # definition of myself
        Name = sacredchao-sd            # this name identifies a storage daemon inside the org.  Name it after the serv
        Maximum Concurrent Jobs = 20

        # remove comment in next line to load plugins from specified directory
        # Plugin Directory = /usr/lib/bareos/plugins
}

Director {
        Name = fnord-dir                # important that this matches the director's name.  See http://doc.bareos.org/master/html/bareos-manual-main-reference.html#AuthorizationErrors
        Password = "BIG_PASSWORD_FROM_pwgen_-sync_48"           # Be sure to stick this in the director's config, too.
}

Device {
        Name = FileStorage
        Media Type = File
        Archive Device = /var/lib/bareos/storage
        LabelMedia = yes;                   # lets Bareos label unlabeled media
        Random Access = Yes;
        AutomaticMount = yes;               # when device opened, read it
        RemovableMedia = no;
        AlwaysOpen = no;
}

This storage daemon has one storage device defined. It's called "FileStorage" and it writes files under /var/lib/bareos/storage. We'll want to get that on a drive that isn't a backup source sometime in the near future.

Clients? That's where all the data live.

We've got a working director to make decisions now. And a storage daemon to write our backups to. How about being able to suck the files out of a system? The piece that does that is called a file daemon. It runs on each client.

Install by installing the bareos-filedaemon package and update bareos-fd.conf to make the director's name and password match. Also update the bareos-dir.conf to put in a correct password for the file daemon.

Let the two jobs the director has currently defined (back up /usr/sbin and back up the catalog) run and come back in the morning.

Setting up file storage

So, sadly, last night's backups failed. The director was able to talk to the file daemon (client) and the storage daemon, but the storage daemon wasn't really ready to write to the FileStorage device. It seems that it must be labelled before being used. One uses the Bareos console to make a labeling operation happen. Install the bareos-bconsole package and edit the director's name amd password in /etc/bareos/bconsole.conf (check them against the director's config file) to enable the console to manage the director.

Fiddle about inside bconsole quite a bit now to create a working, writable backup target.

Add another client

Because that's where the files are. :)

Install bareos-common and bareos-filedaemon (and any prerequisites) on client system. Update /etc/bareos/bareos-fd.conf (on the client) to fix name and password for the director.

Then, on the director machine, add the client's details to bacula-dir.com. Add a Client block to define the let the director know the client exists. Also add a Job block to get an entry into the schedule. This latter will need a distinct name and Client = line.

security

Well, if security is management of risk, keeping copies of one's bits is all about increasing security. We'll concentrate on a few specific areas. First, preventing an adversary with control of our network connections from pretending to be part of the backup system and from siphoning off all our bits while they move between backup system components. Next up will be details on preventing someone who can walk off with our backup media from reading all the files they'd ever want.

for data in motion

Bacula has provisions for TLS-ecapsulated communication between the various components. Bareos has kept these intact. This functionality allows the director, storage daemons, and file daemons to all be sure they are communicating with other components they should be communicating with, that a third party (hello, NSA) is unable to examine the content of the communication, and that the communications have survived the trip over the network without being tampered with.

The following section details a PKI using 8192 bit RSA keys. It turns out these don't work at run-time. Errors like 

Feb 19 15:08:38 sacredchao bareos-dir: crypto_openssl.c:1475 Connect failure: ERR=error:1408E098:SSL routines:SSL3_GET_MESSAGE:excessive message size

will be seen in your logs. I'll be dialing back RSA key sizes until by 1024 until I find something that works at runtime. As of 20 Feb 2014, 7168 bit RSA keys do not appear to work, either. Nor do 6144 bit keys. Nor 5120 bit keys. And reducing the size of the DH parameters used to 1024 bytes doesn't help, either.

So let's create a two-level public key infrastructure, two certificate authorities (one root and a subordinate), for this:

  1. openssl genrsa -aes256 -out bareos-ca-root-key.pem 8192 # creates a private key for our root CA, and encrypts the PEM-encoded key with AES256-CBC using a user provided passphrase. Don't lose the passphrase!
  2. openssl req -x509 -sha512 -new -days $((365 * 20 + 5)) -key bareos-ca-root-key.pem -out bareos-ca-root-certificate.pem -config openssl-bareos-CA.conf # creates a self-signed CA certificate. You will be asked for the passphrase for bareos-ca-root-key.pem. Ths CA certificate is good for 20 years (20 * 365 + 5 leap days) and is hashed with SHA512.
  3. mkdir "Bareos PKI Root Certificate Authority"; cd "$_" # this will be the root of our certificate authority's files
  4. touch ca.db.index; dd if=/dev/random bs=128 count=1 of=ca.db.rand; echo 01 > ca.db.serial; mkdir ca.db.certs # our CA is going to keep files in here.
  5. openssl genrsa -aes256 -out bareos-ca-signing-key.pem 8192 # This key will be signing all of our certificates. This allows the CA root to revoke this key (and all those it has signed) in case of a compromise. Treat the passphrase generated just as carefully as that created for the CA root private key.
  6. openssl req -new -days $((15 * 365 + 4)) -key bareos-ca-signing-key.pem -out bareos-ca-signing-csr.pem -config openssl-bareos-CA.conf # generate a Certificate Signing request to be signed by the root.
  7. openssl ca -config openssl-bareos-CA.conf -name Bareos_CA_default -extensions v3_ca -days $((15 * 365 + 4)) -out bareos-ca-signing-certificate.pem -infiles bareos-ca-signing-csr.pem # This uses the root CA's certificate to create the Signing CA's certificate, good for 15 years.
  8. mkdir ../"Bareos PKI Signing Certificate Authority"; cd "$_" # now that the root CA has issued the Signing CA's certificate, we'll move on to the working on the signing CA
  9. cp "$OLDPWD"/*signing*pem "$OLDPWD"/*.conf . # Get our private key, certificate, and OpenSSL config file here.
  10. mkdir certificates crl; echo 01 > signing.db.serial; echo 00 > signing.db.crlnumber; dd if=/dev/urandom if=signing.db.rand bs=128 count=1 # create us some handy directories and bookkeeping files.
  11. edit the conf file appropriately (change default_ca settings and add a section for the new signing certificate authority.

At this point, a second level certificate authority exists, ready to issue certificates requested by bareos directors, storage daemons, file daemons, and any friends they might have. So let's see if we can generate some and get the various components to authenticate each other:

  1. openssl genrsa -out fnord-director-key.pem 8192 # create an 8192 bit RSA private key, no passphrase on the file storing it. Take great care to be sure it doesn't fall into Eve's or Mallory's or Oscar's hands.
  2. openssl req -new -days $((10 * 365 +2)) -key fnord-director-key.pem -out fnord-director-csr.pem -config ~/proj/bareos-PKI/"Bareos PKI Signing Certificate Authority"/openssl-bareos-signing-CA.conf # generate a certificate request with this director's key, good for 10 years.
  3. cd "Bareos PKI Signing Certificate Authority" # go to the Signing certificate authority directory
  4. openssl ca -config openssl-bareos-signing-CA.conf -name Bareos_Signing_CA_default -days $((10 * 365 + 2)) -out fnord-director-certificate.pem -infiles ../fnord-director-csr.pem # issue the director's certificate

We now have an RSA private key and certificate with which we can configure the director. Let's tell the director about those:

  • edit /etc/bareos/bareos-dir.conf, add the following to the Director section:
       # Network communications security
       TLS Enable =                    yes
       TLS Require =                   yes
       TLS Certificate =               /etc/bareos/fnord-director-certificate.pem
       TLS Key =                       /etc/bareos/fnord-director-key.pem
       TLS Verify Peer =               yes
       # TLS Allowed CN =      # not using this
       TLS CA Certificate File =       /etc/bareos/bareos-ca-root-certificate.pem
       # TLS CA Certificate Dir = /path/to/directory/of/trusted/signing/certificates
       TLS DH File =                   /etc/bareos/fnord-director-dhparam4096.pem

This needs to be repeated for the file daemon (client) running on sacredchao:

  1. openssl genrsa -out sacredchao-fd-key.pem 8192 # create an 8192 bit RSA private key, no passphrase on the file storing it. Take great care to be sure it doesn't fall into Mallory's hands.
  2. openssl req -new -days $((10 * 365 + 2)) -key sacrechao-fd-key.pem -out sacredchao-fd-csr.pem -config ~/proj/bareos-PKI/"Bareos PKI Signing Certificate Authority"/openssl-bareos-signing-CA.conf # generate a certificate request with this director's key, good for 10 years.
  3. cd "Bareos PKI Signing Certificate Authority" # go to the Signing certificate authority directory
  4. openssl ca -config openssl-bareos-signing-CA.conf -name Bareos_Signing_CA_default -days $((10 * 365 + 2)) -out sacredchao-fd-certificate.pem -infiles ../sacredchao-fd-csr.pem # issue the file daemon's certificate
  5. openssl dhparam -out /etc/bareos/sacredchao-fd-dhparam4096.pem -5 4096 # generate a really long prime to be used in the creation of Diffie-Hellman ephemeral keys
  • Edit the file daemon's config file (/etc/bareos/bareos-fd.conf) placing the following in the FileDaemon section:
        # Network communications security
        TLS Enable =                    yes
        TLS Require =                   yes
        TLS Certificate =               /etc/bareos/sacredchao-fd-certificate.pem
        TLS Key =                       /etc/bareos/sacredchao-fd-key.pem
        TLS Verify Peer =               yes
        # TLS Allowed CN =              # not using this
        TLS CA Certificate File =       /etc/bareos/bareos-ca-root-certificate.pem
        # TLS CA Certificate Dir =      /path/to/directory/of/trusted/signing/certificates
        # TLS DH File =                 /etc/bareos/sacredchao-fd-dhparam4096.pem

It turns out that all those trillions of CPU cycles used to generate a DH parameters file were wasted. The file daemon does not use make use of it. :(

And the storage daemon running on sacredchao:

  1. openssl genrsa -out sacredchao-sd-key.pem 8192 # create an 8192 bit RSA private key, no passphrase on the file storing it. Take great care to be sure it doesn't fall into Mallory's hands.
  2. openssl req -new -days $((10 * 365 + 2)) -key sacredchao-sd-key.pem -out sacredchao-sd-csr.pem -config ~/proj/bareos-PKI/"Bareos PKI Signing Certificate Authority"/openssl-bareos-signing-CA.conf # generate a certificate request with this director's key, good for 10 years.
  3. cd "Bareos PKI Signing Certificate Authority" # go to the Signing certificate authority directory
  4. openssl ca -config openssl-bareos-signing-CA.conf -name Bareos_Signing_CA_default -days $((10 * 365 + 2)) -out sacredchao-sd-certificate.pem -infiles ../sacredchao-sd-csr.pem # issue the stoage daemon's certificate
  5. openssl dhparam -out /etc/bareos/sacredchao-sd-dhparam4096.pem -5 4096 # generate a really long prime to be used in the creation of Diffie-Hellman ephemeral keys

And add this block to the storage daemon's config file (/etc/bareos/bareos-sd.conf) Storage block: 

        # Network communications security
        TLS Enable =                    yes
        TLS Require =                   yes
        TLS Certificate =               /etc/bareos/sacredchao-sd-certificate.pem
        TLS Key =                       /etc/bareos/sacredchao-sd-key.pem
        TLS Verify Peer =               yes
        # TLS Allowed CN =              # not using this
        TLS CA Certificate File =       /etc/bareos/bareos-ca-root-certificate.pem
        # TLS CA Certificate Dir =      /path/to/directory/of/trusted/signing/certificates
        TLS DH File =                   /etc/bareos/sacredchao-sd-dhparam4096.pem

Further edits to daemon config files:

  • bareos-dir.conf needs to be told to do TLS authentication, privacy, and integrity with the storage and file daemons. Put the following in the Storage and Client definitions:
        TLS Enable =                    yes
        TLS Require =                   yes
        TLS CA Certificate File =       /etc/bareos/bareos-ca-root-certificate.pem
        TLS Certificate =               /etc/bareos/fnord-director-certificate.pem
        TLS Key =                       /etc/bareos/fnord-director-key.pem

bareos-sd.conf needs to be told to check on who is talking to the storage daemon with the following in the "Director" stanzas:

for data at rest

Retrieved from "https://wiki.fnord.greeley.co.us/mediawiki/index.php?title=Backups&oldid=600"