Debian Etch, which is the current release, has the Bacula 2.x packages. I needed to upgrade to the Bacula 3.x packages, which are located in debian/testing, also known as the forthcoming Debian Squeeze release. In addition, since PostgreSQL 8.3 is packaged in Etch, and PostgreSQL 8.4 is packaged in Squeeze/testing, a database migration is also required.
Friday, January 29. 2010
Migrating Bacula 2.x on Debian Etch to 3.x on Squeeze
I had attempted updating my sources.list file to testing and then running the apt-get dist-upgrade process. This broke some dependences, and also broke on a udev migration. I guess testing has more testing to do on the distribution upgrade process.
In the end, I built a new Bacula service on a freshly installed Debian testing server.
The special consideration for this configuration is that it needs to handle backing up servers across a WAN. As such, backups may travel through one or more firewalls. Through such a configuration, it is very difficult to get the firewall ports opened for the various necessary Bacula service ports. The better way to tackle this is through the use of ssh' port local and remote port forwarding capability. Port 22 is becomes the only necessary port to open on a firewall. The ssh-tunnel.sh script helps make this happen.
To build the server, when it came to package selection, I unselected all packages, and then chose just the database package which installed PostgreSQL.
After the basic server finished installing and rebooted, I manually installed the following packages:
apt-get install bacula-common-pgsql apt-get install bacula-client apt-get install bacula-director-common apt-get install bacula-director-pgsql apt-get install bacula-sd-pgsql apt-get install bacula-server
If starting with a new database, then dbconfig-common can be used. If migrating an older database, don't use dbconfig-common, and use the manual methods I'll describe further on. There is further documentation in /usr/share/doc/bacula-director-pgsql.
During installation of the bacula packages, a new user of 'bacula' is created, as well as a group called 'tape'. The 'bacula' user has a home directory of /var/lib/bacula.
Into that directory, create a .ssh directory for any authorized_keys and known_hosts required. I also created a keys subdirectory to hold the public/private keys for ssh'ing into other servers for processing backups. I called the two files 'bacula' and 'bacula.pub'. These will be referenced in my customized ssh-tunnel.sh script.
Run
dpkg-reconfigure exim4-config
to reconfigure the mail system to allow outbound mail delivery.
My backups go onto a remote file share. I created an entry in /etc/fstab along the lines of:
10.1.1.1:/bu /mnt/nas nfs rw,hard,intr,async,nodev,nosuid 0 0
Ensure that the NFS client is installed through:
apt-get install nfs-common
In /etc/postgresql/8.4/main/pg_hba.conf, I have lines along:
host bacula bacula 127.0.0.1/32 trust host bacula sysadmin 127.0.0.1/32 trust local bacula bacula trust local bacula sysadmin trust
As an aside, a useful command to find out database information is through the use of:
psql -l
When migrating the database to 8.4, there are modifcations to the pg_dump command required (which are required to prevent import errors along the lines of 'ERROR: invalid byte sequence for encoding "UTF8"', basically resolving the UTF-8 to SQL_ASCII issues in Bacula):
pg_dump -E SQL_ASCII -U bacula bacula > /var/lib/bacula/bacula.sql
On the new server, use the following to import the database:
dropdb bacula su - postgres psql create role bacula; create database bacula owner=bacula encoding='SQL_ASCII' template=template0; /q psql bacula </var/lib/bacula/bacula.sql
Basic instructions for updating the database from Bacula table version 10 to Bacula table version 11 is found in /usr/share/bacula-director/update_postgresql_tables:
BEGIN; ALTER TABLE file ALTER fileid TYPE bigint ; ALTER TABLE basefiles ALTER fileid TYPE bigint; ALTER TABLE job ADD COLUMN readbytes bigint default 0; ALTER TABLE media ADD COLUMN ActionOnPurge smallint default 0; ALTER TABLE pool ADD COLUMN ActionOnPurge smallint default 0; -- Create a table like Job for long term statistics CREATE TABLE JobHisto (LIKE Job); CREATE INDEX jobhisto_idx ON JobHisto ( starttime ); UPDATE Version SET VersionId=11; COMMIT;
Once the configuration files for the director, storage manager, and file manager are ready, bacula can be managed through 'bconsole'.
My modified /etc/bacula/scripts/ssh-tunnel.sh looks like:
#!/bin/sh # script for creating / stopping a ssh-tunnel to a backupclient # Stephan Holl sholl@gmx.net # Modified by Joshua Kugler joshua.kugler@uaf.edu # Modified by Ray Burkholder ray@oneunified.net # # # variables USER=bacula CLIENTADDR=$2 # CLIENTPORT is local end CLIENTPORT=$3 #LOCAL=your.backup.server.host.name # local is a local address and uses ssh's remote/local port forwarding LOCAL=127.0.0.1 SSH=/usr/bin/ssh SSHOPTIONS=-vfnCN2 LOG1=/var/lib/bacula/log1.log LOG2=/var/lib/bacula/log2.log #LOG1=/dev/null #LOG2=/dev/null # location of the public/private keys used with ssh to gain access to remote servers KEY=/etc/bacula/keys/bacula case "$1" in start) # create ssh-tunnel echo "Starting SSH-tunnel to $CLIENTADDR..." $SSH $SSHOPTIONS -o PreferredAuthentications=publickey -i $KEY -l $USER \ -R 9101:$LOCAL:9101 -R 9103:$LOCAL:9103 -L $CLIENTPORT:$LOCAL:9102 $CLIENTADDR \ >> $LOG1 2>> $LOG2 exit $? ;; stop) # remove tunnel echo "Stopping SSH-tunnel to $CLIENTADDR..." # find PID killem PID=`ps ax | grep "$SSH $SSHOPTIONS -o PreferredAuthentications=publickey -i $KEY" \ | grep "$CLIENTADDR" | awk '{ print $1 }'` kill $PID exit $? ;; *) # usage: echo " " echo " Start SSH-tunnel to client-host" echo " to bacula-director and storage-daemon" echo " " echo " USAGE:" echo " ssh-tunnel.sh {start|stop} client.fqdn" echo "" exit 1 ;; esac
The links I used for getting started with ssh-tunnels are found at:
- /usr/share/doc/bacula-common/examples/ssh-tunnel-README.txt.gz
- http://wiki.bacula.org/doku.php?id=sshtunnel
In /etc/hosts file, 127.0.0.1 should be the only line referring to the local server. The exteral port ip address should be commented out:
127.0.0.1 localhost bu.example.com bu #10.10.10.1 bu.example.com bu
In the bacula-dir.conf configuration file, a typical client configuration will look similar to:
Client { Name = mail-fd Address = 127.0.0.1 FDPort = 9130 # specific port for this client, allows multiple simultaneous backups Catalog = MyCatalog Password = "xxxxxx" # password for FileDaemon File Retention = 120 days Job Retention = 4 months AutoPrune = yes # Prune expired Jobs/Files }
The special characteristic of the above configuration is the use of a unique port number for FDPort. Each client in the bacula-dir.conf should have a unique port number. This allows bacula to tunnel via ssh to remote clients and redirect them to the storage manager on the local server.
The definition of the storage device in bacula-dir.conf will have Address=127.0.0.1 and SDPort=9103.
The job description for each client should have something similar to:
Job { Name = "mail-fd" Client = mail-fd JobDefs = "DefaultJob" FileSet = "FileSet_mail" Storage = storageSshClients Write Bootstrap = "/var/lib/bacula/mail.bsr" Priority = 12 Run Before Job = "/etc/bacula/scripts/ssh-tunnel.sh start mail.example.com 9130" Run After Job = "/etc/bacula/scripts/ssh-tunnel.sh stop mail.example.com 9130" }
When using Bacula in console mode, a useful command to find out the meaning of the backup status codes:
*sqlquery Entering SQL query mode. Terminate each query with a semicolon. Terminate query mode with a blank line. Enter SQL query: select * from status; +-----------+---------------------------------+ | jobstatus | jobstatuslong | +-----------+---------------------------------+ | C | Created, not yet running | | R | Running | | B | Blocked | | T | Completed successfully | | E | Terminated with errors | | e | Non-fatal error | | f | Fatal error | | D | Verify found differences | | A | Canceled by user | | F | Waiting for Client | | S | Waiting for Storage daemon | | m | Waiting for new media | | M | Waiting for media mount | | s | Waiting for storage resource | | j | Waiting for job resource | | c | Waiting for client resource | | d | Waiting on maximum jobs | | t | Waiting on start time | | p | Waiting on higher priority jobs | +-----------+---------------------------------+ Enter SQL query: End query mode.
For the bacula entry in /etc/passwd, change /bin/false to be /bin/sh.
For each server to which will be connected via ssh, within the context of the bacula user, use the following command to update ~/.ssh/known_hosts:
ssh -l bacula -i /etc/bacula/keys/bacula -v server.example.com