In this article, we will set up a Storage Gateway virtual machine on-premises that will cache and buffer backup data from Bacula.

The Storage Gateway virtual machine will then upload the data to the VTL service, having a cost-effective backup solution, enabling at the same time offsite storage of our data backups.

What is Bacula?

Bacula is an Open Source suite to manage your backups. It is very customizable and allows a very powerful storage configuration. You can store your backups in tapes (single drive or using a media changer) or to a local disk.

Bacula architecture is made of decoupled programs that makes easy to combine different storage systems. You can use multiple storage backends from the same Bacula director.

What is Virtual Tape Library?

The Virtual Tape Library (or VTL) is a tape-based storage solution of Amazon Web Services Storage Gateway. You can use the VTL service to archive backup data in Amazon Glacier, providing seamless integration between on-premises environment and the AWS cloud.

The architecture will look like this, except that we will use a single Data Center (source):

By deploying the Storage Gateway virtual machine on-premises, we will benefit of having a local cache of our data, making possible both quick access to the most recent data backed up and a buffer between our data center and the S3 storage where the virtual tapes live.

It is highly recommended that you use individual disk devices to be used as the upload buffer and cache storage in your Storage gateway. Keep in mind that both buffer sizes will have direct impact in performance because their main use is to avoid network latencies between your Storage Gateway and AWS. You can use RAW iSCSI devices if you happen to have some kind of SAN.

Connections from Bacula to VTL

You will need to install iSCSI Initiator Tools for your Linux distribution (open-iscsi in Debian and derivatives, iscsi-initiator-utils if you are using a Red Hat flavor). In our particular case, we are going to use Debian.

To discover the targets that the VTL appliance exposes, you can follow the instructions from the VTL documentation, in a nutshell:

[code language=”shell”]
# This will output your gateway’s IQN that you will use in the next command
sudo /sbin/iscsiadm –mode discovery –type sendtargets –portal GATEWAY_IP:3260

# This will connect the target (devices exposed by the VTL virtual machine)
sudo /sbin/iscsiadm –mode node –targetname IQN_YOU_GOT_IN_THE_PREVIOUS_COMMAND
–portal GATEWAY_IP:3260,1 –login
[/code]

Note that by default your system can detect the devices as generic SCSI devices, but for Bacula to operate correctly, we need to load the ‘st‘ driver. You can force that by adding the module name to /etc/modules.

You can see the newly added devices using lsscsi -g:

[code language=”shell”]
# lsscsi -g
[3:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st3 /dev/sg5
[4:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st2 /dev/sg3
[5:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st0 /dev/sg2
[6:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st1 /dev/sg4
[7:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st9 /dev/sg12
[8:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st4 /dev/sg7
[9:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st5 /dev/sg6
[10:0:0:0] mediumx AWS Gateway-VTL 0100 – /dev/sg8
[11:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st8 /dev/sg9
[12:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st7 /dev/sg10
[13:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st6 /dev/sg11
[/code]

As you can see, we have 10 drives and a media changer, and we can use any combination of the drives if we have the storage and bandwidth capacity. In this post, we will only be using one of them, but keep in mind that you can use them all if you want to.

As Linux nowadays uses udev to autodiscover devices, it can happen that the device names are not preserved between reboots. Since the Bacula configuration needs a static device name, we will write a udev rule to match the media changer and give it a predefined name. Add the following lines to /etc/udev/rules.d/80-drivers.rules

[code language=”shell”]
# type 8 devices are “Medium Changers”
SUBSYSTEM==”scsi_generic”, SUBSYSTEMS==”scsi”, ATTRS{type}==”8″,
IMPORT{program}=”scsi_id –sg-version=3 –export –whitelisted -d $devnode”,
SYMLINK+=”tape/by-id/scsi-$env{ID_SERIAL}-changer”
[/code]

This will create a link under /dev/tape/by-id/ pointing to the detected changer device. For example:
[code language=”shell”]
# ll /dev/tape/by-id/
total 0
lrwxrwxrwx 1 root root 9 Mar 11 14:41 scsi-2414d5a4e5f5347572d454632-changer -> ../../sg8
[/code]

MTX: the changer helper

Bacula is agnostic about its storage backends. It does not know how to manage a media changer or write to a tape. It uses the underlying UNIX phylosophy of “everything is a file” and it uses helper tools to manage tapes and drives.

One of the backend tools is mtx, a program used to control media changers. You can use mtx manually to operate with the changer, but if you make any permament change (e.g. load/unload tapes) remember always to execute the update slots command in the bacula console.

Status

List Data Transfer Elements (drives) as well as Storage Elements (slots) with their contents. Example:
[code language=”shell”]
# mtx -f /dev/tape/by-id/scsi-2414d5a4e5f5347572d454632-changer status | head -n 50
Storage Changer /dev/tape/by-id/scsi-2414d5a4e5f5347572d454632-changer:10 Drives, 3200 Slots ( 1600 Import/Export )
Data Transfer Element 0:Full (Storage Element 1 Loaded):VolumeTag = TAP964B33
Data Transfer Element 1:Empty
Data Transfer Element 2:Empty
Data Transfer Element 3:Empty
Data Transfer Element 4:Empty
Data Transfer Element 5:Empty
Data Transfer Element 6:Empty
Data Transfer Element 7:Empty
Data Transfer Element 8:Empty
Data Transfer Element 9:Empty
Storage Element 1:Empty:VolumeTag=
Storage Element 2:Full :VolumeTag=TAP944B31
Storage Element 3:Full :VolumeTag=TAPA44B01
Storage Element 4:Full :VolumeTag=TAPA54B00
Storage Element 5:Full :VolumeTag=TAP5B4AFE
[…]
[/code]

Note that there are 3200 slots, 1600 regular slots and 1600 Import/Export slots, used when archiving and retreiving tapes to/from the Virtual Tape Shelf.

Load/Unload

You can manually load or unload tapes from the slots (Storage Element) to the drives (Data Transfer Element).
Be careful, although, if Bacula is operating with the drives not to interfere with its normal operation.

For example, to load the tape located at slot 3 to the drive number 9:
[code language=”shell”]
# mtx -f /dev/tape/by-id/scsi-2414d5a4e5f5347572d454632-changer load 3 9
Loading media from Storage Element 3 into drive 9…done
# mtx -f /dev/tape/by-id/scsi-2414d5a4e5f5347572d454632-changer status
Storage Changer /dev/tape/by-id/scsi-2414d5a4e5f5347572d454632-changer:10 Drives, 3200 Slots ( 1600 Import/Export )
Data Transfer Element 0:Full (Storage Element 1 Loaded):VolumeTag = TAP964B33
Data Transfer Element 1:Empty
[…]
Data Transfer Element 8:Empty
Data Transfer Element 9:Full (Storage Element 3 Loaded):VolumeTag = TAPA44B01
Storage Element 1:Empty:VolumeTag=
Storage Element 2:Full :VolumeTag=TAP944B31
Storage Element 3:Empty:VolumeTag=
[/code]

The same for unloading (the argument order is the same):
[code language=”shell”]
# mtx -f /dev/tape/by-id/scsi-2414d5a4e5f5347572d454632-changer unload 3 9
Unloading drive 9 into Storage Element 3…done
# mtx -f /dev/tape/by-id/scsi-2414d5a4e5f5347572d454632-changer status | head -n 30
Storage Changer /dev/tape/by-id/scsi-2414d5a4e5f5347572d454632-changer:10 Drives, 3200 Slots ( 1600 Import/Export )
Data Transfer Element 0:Full (Storage Element 1 Loaded):VolumeTag = TAP964B33
Data Transfer Element 1:Empty
[…]
Data Transfer Element 8:Empty
Data Transfer Element 9:Empty
Storage Element 1:Empty:VolumeTag=
Storage Element 2:Full :VolumeTag=TAP944B31
Storage Element 3:Full :VolumeTag=TAPA44B01
[/code]

The mtx-changer wrapper

The mtx tool is handy to perform manual management of the devices, but bacula comes with a nice wrapper called mtx-changer (in Debian systems, you can find int in /etc/bacula/scripts/mtx-changer), the one that Bacula will actually use.

The wrapper is indeed intended to be modified to suit your particular needs, although it should work out of the box for most changers. In the particular case of the VTL, it will filter out the 1600 Import/Export slots, so if you want to use them directly form the bacula console (for example, when you create a new tape or retrieve an archived tape from the Virtual Tape Shelf, the VTL will insert it in one of the available I/E slots), you will have to tweak it a bit. In our case, we are happy to interact with the Import/Export slots using mtx interface in a manual fashion.

Bacula configuration

Next, we need to modify the Bacula configuration to use the new devices. We will change two files:

Storage Daemon

To configure the Storage Daemon, you just have to create the resources in your Storage Daemon configuration (bacula-sd.conf file). Let’s see the resource definition and then we will break down the meaning of the directives:
[code language=”shell”]
Autochanger {
Name = “VTL Autochanger”
Device = Drive-1
Changer Command = “/etc/bacula/scripts/mtx-changer %c %o %S %a %d”

Changer Device = /dev/tape/by-id/scsi-2414d5a4e5f5347572d454632-changer
}

Device {
Name = Drive-1
Drive Index = 0
Media Type = ULT3580-TD5
Archive Device = “/dev/tape/by-path/ip-GATEWAY-IP:3260-iscsi-YOUR-IQN-tapedrive-01-lun-0-nst”
RemovableMedia = yes;
RandomAccess = no;
AutoChanger = yes
}
[/code]

Autochanger: The definition of the properties of the medium changer:

Device: The definition each of the tape drives:

Director

We will add a new Storage and Pool resources to the bacula-dir.conf configuration file:
[code language=”shell”]
Storage {
Name = VTL
Address = storage-daemon.fqdn # N.B. Use a fully qualified name here
SDPort = 9103
Password = “password”
Device = “VTL Autochanger”
Media Type = ULT3580-TD5
Autochanger = yes
}

Pool {
Name = Backup
Pool Type = Backup
Storage = VTL
[…]
}
[/code]
The 2 important directives are:

1. Storage: The definition of the storage for our jobs.

2. Pool: Definition of a pool of tapes that Bacula can use to make backups.

Then, we are all set. Now we just need to use the Pool resource that uses the VTL storage backend in our backup jobs.

Wrapping up

As we have seen, Bacula configuration possibilities are very powerful, but it can be quite complex to set up (What? do I need to add 4 resources in two different files just to use a tape drive?), but there is where its flexibility lies.

AWS Virtual Tape Library is a perfect match for Bacula. We have local easy-to-access data (resting in the Cache Buffer in the VTL appliance on-premises) as well as safe, cloud-based remote offsite backups. We can leverage Glacier to reduce costs impact as well.

TAGS: amazon web services, aws, backups, bacula, herramientas, how-to, howto, procedimientos, recipes, storage

speech-bubble-13-icon Created with Sketch.
Comments
Jhonatan | November 4, 2016 12:58 am

Hace poco me di con la sorpresa que S3 solo soporta hasta 4GB de tamaño de archivo en la subida. Como maneja el gateway vtl si los volumenes de bácula que intento respaldar en la nube tiene como tamaño límite 100GB cada uno?

Reply
L. Alberto Giménez | November 4, 2016 8:46 am

Hola Jhonatan, muchas gracias por tu comentario!

Los límites que comentas no son correctos. Utilizando la API de S3 se pueden hacer subidas de objetos de hasta 5GB en una única operación, pero este límite aumenta hasta los 5 TB si se hacen uploads multipart, por lo que no hay ningún problema en mantener cintas virtuales de 100GB con VTL.

Espero haberte ayudado, un saludo!

L. Alberto

Reply
Drew | October 28, 2016 5:12 pm

Your UDEV rules don’t work in CentOS 7:

# type 8 devices are “Medium Changers”
SUBSYSTEM==”scsi_generic”, SUBSYSTEMS==”scsi”, ATTRS{type}==”8″,
IMPORT{program}=”scsi_id –sg-version=3 –export –whitelisted -d $devnode”,
SYMLINK+=”tape/by-id/scsi-$env{ID_SERIAL}-changer”

It breaks LVM partitions, not all will mount and the system is thrown into a maintenance mode at bootup.

Reply
Drew | October 28, 2016 5:27 pm

Sorry, meant to add:

Adding the full path to scsi_id which is /usr/lib/udev/scsi_id still results in failure, it kicks off a LVM2 PV scan and fails.. into maintenance mode.

Reply
L. Alberto Giménez | October 31, 2016 2:10 pm

Hi Drew! Thanks for commenting.

It is quite strange what is happening in your case. The udev rule just creates a symlink, and only for the changer device (TYPE=8). It should not tamper with your other devices.

Anyway, I had not the chance to check the procedure described in the article in a CentOS 7 system, so I’m afraid I can’t help here.

Have you checked the official AWS documentation to see if there are any particular steps to keep in mind when using CentOS 7?

Best regards,
L. Alberto

Reply
Jhonatan | September 7, 2016 2:06 am

Hola Alberto, desde mi consola AWS veo que me da la opción de descargar la VM gateway pero me limita a descargarlo para VMware?, no existe forma de usarlo en KVM?, entiendo que este gateway está entre Bácula y AWS

Reply
L. Alberto Giménez | September 8, 2016 2:39 pm

Hola Jhonatan! AWS no soporta hipervisores KVM, únicamente puedes desplegar su appliance en VMWare ESXi, Microsoft Hyper-V y una instancia EC2. Por favor, visita http://docs.aws.amazon.com/storagegateway/latest/userguide/Requirements.html#requirements-host para más detalles.

Espero haberte ayudado.
Saludos,
L. Alberto Giménez

Reply
Fabrice Bacchella | September 6, 2016 10:20 am

I’m not sure about your udev command. On a RHEL7, it returns:

# /usr/lib/udev/scsi_id –sg-version=4 –export –whitelisted -d /dev/sg5

ID_TYPE=generic

Reply
Fabrice Bacchella | September 6, 2016 10:39 am

Never mind, I was dump.

Reply
L. Alberto Giménez | September 8, 2016 2:45 pm

Thanks for commenting, don’t hesitate to ask anything that isn’t clear!

Regards,
L. Alberto Giménez

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*
*