1. Context 6.3 6.4 6.5 1.111.5 2 5 6 6 6 6 7 7 7 8 9 10 Backup to tape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Backup to standby disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . Backup to CDROM/DVD . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.111.5 Maintain an effective data backup strategy Weight 3 Linux Professional Institute Certification — 102 7 8 9 Raid Systems Raid tape array Backup Server 10 Broken Mirror 11 Command line tools 12 Backup Applications 13 Rotation strategies 14 License of this document Grant Parnell Geoffrey Robertson ge@ffrey.com Nick Urbanik nicku@nicku.org This document Licensed under GPL—see section 14 2005 July Outline 1 Context Topic 111 Administrative Tasks [21] Contents 1 2 3 4 Context Objectives Deciding what to back up Types of data 4.1 Databases . . . . . . . . . 4.2 Log files . . . . . . . . . . 4.3 Home directories . . . . . 4.4 Code . . . . . . . . . . . . 4.5 Web Sites . . . . . . . . . 4.6 Web sites with volatile data Important Linux Directories Methods of Backup and Restore 6.1 Another directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Standby partition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 3 3 3 4 4 4 4 4 5 5 5 5 1.111.1 Manage users and group accounts and related system files [4] 1.111.2 Tune the user environment and system environment variables [3] 1.111.3 Configure and use system log files to meet administrative and security needs [3] 1.111.4 Automate system administration tasks by scheduling jobs to run in the future [4] 1.111.5 Maintain an effective data backup strategy [3] 1.111.6 Maintain system time [4] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Objectives 5 6 Description of Objective Candidate should be able to plan a backup strategy and backup filesystems automatically to various media. Tasks include dumping a raw device to a file or vice versa, performing partial and manual backups, verifying the integrity of backup files and partially or fully restoring backups. 3. Deciding what to back up Key files, terms, and utilities include: cpio — creates an archive of data dd — can copy raw filesystems dump — backs up ext2, ext3 systems 1.111.5 3 4.2 Log files 1.111.5 4 4.2 Log files Logs People don’t tend to read them unless something goes wrong in which case they’re valuable. These need to be kept but don’t need to be restored in a hurry. 4.3 Home directories Home directories This is a mixed bag of everything but some policies could be instated to make the admin’s life easier. E.G., Making specific sub-directories for things and assigning them different backup/restore priorities. Often the existence of a home directory is more important than the rest of the contents as it may make a user unable to login without it. restore — restores files from dump backups tar — creates “tarballs” and creates archives on tape 3 Deciding what to back up Backups Decide what data is important and how long you can do without it. • Is this used 24×7 or just business hours? • During business hours how long can you do without it? 4 hours, 30 minutes, 5 minutes? • How up-to-date is it required to get you running in an emergency? • Are you backing up for archival or high availability or espionage? 4.4 Code Code repositories Programmers should be accustomed to doing regular backups anyway, they often need to revert to an old version to figure out what they broke. Any tools used such as CVS that have a central repository should be backed up almost as often as programmers commit code, at least once a day but they could probably cope with it being missing for half a day. 4 Types of data 4.5 Web Sites High availability — read only Websites frequently used by your clients. They can contain dynamic data but customers don’t update it. This sort of scenario lends itself to frequent replication to a backup server. Examples of Data Configurations of running servers. You need these 24×7 but they don’t change much. 4.1 Databases Databases / Transactions — financial & otherwise These are updated frequently and need to balance. Associated with these are logs & duplication & other means of rollback & integrity checking. With databases it’s often a good idea to dump them in a good portable format, especially if the inbuilt format is not cross platform or cross version compatible. Example: $ mysqldump mydata >mydata.dump ← This will give you a text file which can be used on most mysql versions and possibly adapted to other database packages. 4.6 Web sites with volatile data High availability — interactive Taking a website again, this one might allow the customer to do such things as place orders. The website maintains some state information to allow building of an order. This is the most difficult, the state information can be stored in a replicated database. In the event of web server failure the other one comes into play and the customer may have to login again but the information is kept. (Otherwise complex designs and expensive hardware can be used to seamlessly migrate the state to the other webserver). 5. Important Linux Directories 1.111.5 5 6.4 Backup to standby disk 1.111.5 6 5 Important Linux Directories /var/spool/mail /var/lib/mysql /var/log /etc /home /home//mail /home//.ssh /usr/local daily backup databases — backup the dumps, and possibly the binary. from “don’t care” to “backup daily” backup config changes be selective, but if you can’t, backup daily. contains the user’s mail folders (may also be Mail or Maildir) If you login using ssh keys only, this is a must have. locally installed apps & data Application specifics simple GUI. The reason for using the basic systems is you can restore from them if you have to. Important Linux directories 6.4 Backup to standby disk Backup to standby disk This can offer peace of mind and a fairly cheap backup for people that don’t require 24×7 service. Basically a removable drive bay houses another hard disk of similar capacity and the entire system is backed up. This can be done partition by partition or file by file using dd, cpio or rsync. Additional steps can be taken to ensure that the backup is also bootable. The backup drive should be removed once done and treated like a tape. The disadvantage here is that you most likely will need to power down the system twice for one backup. Alternately, if you have an external USB or fire-wire storage medium it becomes possible to do this without downtime. 6 Methods of Backup and Restore 6.1 Another directory Backup & Restore methods This is the poor mans backup and does not offer much peace of mind. It does protect against accidental deletion & corruption by users. One advantage is that it can be very quick for things such as log files. You can also keep multiple copies, one for every day of the week for example. See /etc/logrotate.conf. 6.5 Backup to CDROM/DVD Backup to CDROM/DVD Under Linux (as far as I know) there’s no software to directly write data without creating an image first. This means there must be sufficient space available. It would be possible to create a bootable CD with restore software and a compressed filesystem but I haven’t seen this. It may be OK if you don’t have a large filesystem or you have a DVD writer or you’re not backing up everything. 7 6.2 Standby partition Backup to a standby partition This has about the same level of peace of mind as the above. The backup partition can be left un-mounted after the backup. The backup is slower than the above but the restore operation can be quick. See also “Broken Mirror” method below. Raid Systems 6.3 Backup to tape Backup to tape This is probably the most common backup used in the commercial world. It’s easy to backup the lot every day provided you have the tape capacity. If you don’t, you become more selective as to what to backup. There’s a variety of software to do this but there are three main basic systems: tar, cpio and dump. Often commercial software uses these basic systems and provide for labelling & indexing as well as multi-server capability from a RAID System Not strictly a backup but a RAID system can protect against hard drive failure by providing redundancy. Data is written simultaneously to 2 or more hard drives and can include parity information. It does not protect against corrupt databases and people removing files. It will corrupt & remove files equally well on all disks. Linux can do RAID in software very well but the ideal is a hardware solution involving hot swapable disks so they can be replaced while the system is fully running. A RAID system can mean the difference between going on-site at 3am and saying “Oh dear, we’ll replace that first thing in the morning”. Just ensure that you do have a replacement readily available and do not have to wait a week. 8 Raid tape array RAID Tape array 9. Backup Server 1.111.5 7 12. Backup Applications tar 1.111.5 8 In a similar manner to RAID 5 disks, data is written in parallel to 5 tape drives which increases throughput and data integrity. 9 Backup Server Tape ARchive — you all know how to unpack tgz files, and maybe even create them. Just remove the ’f’ option. It also can be an advantage not to use compression as some drives have this built in. Also, a portion of the tape being corrupt can ruin the rest of the data, whereas you can skip corrupt bits and pickup the next file if not compressed. Example: tar -c /home cd /tmp; tar -x Backup Server All of the methods discussed so far involve direct transfer from server to backup medium. If you have a number of servers it may not be practical to install backup devices on each. Another way is to remotely access the required medium directly (/dev/rmt0) but arbitration of access can be an issue. An increasingly popular way is to provide a super-server with a huge amount of disk space capable of holding everything required by the other servers. Transferring the data can happen at any time in either a batch or continuous process. A batch would be say backup a whole directory at once whereas a continuous operation might be transmitting log information or database updates. The backup server itself may then employ any one or more methods to perform backups of itself, possibly based on some statistical analysis. An example of this is a system called ADSM which employs RAID arrays, multiple tape drives, a tape robot with barcode reader and intelligent software that tells the operators which tapes are to go off-site and which ones it wants back. It essentially is a huge cache that stores frequently changing data locally and stores old data off-site. Backup Software cpio CP I/O — Similar capabilities of tar but different methodology. Example: $ find /home | cpio -oB >/dev/tape $ cd /tmp; cpio -idB and Nick Urbanik . Permission is granted to make and distribute verbatim copies or modified versions of this document provided that this copyright notice and this permission notice are preserved on all copies under the terms of the GNU General Public License as published by the Free Software Foundation—either version 2 of the License or (at your option) any later version.