I’ve run my share of Hosting operations as a Sr. Admin before and one thing that always *killed* me… Server failures and restorations.
Every shop I’ve been in has primarily been a cPanel shop. It’s what the market demanded, so it’s what we sold. Not only was it simply “what the masses wanted”, it offered an entire suite of scripts, apps and A+ quality support and upgrades that almost brought a tear to my eye. 🙂
However, the one thing it did not provide, was a quick n’ dirty backup and restore system, let me explain.
cPanels backup system is based around the single user ideology. When you backup a server, it does this on a per cpuser account basis. Now this has it’s pros and it’s cons. It’s nice in the fact that restoring/migrating/tinkering with a single account is easy and *very* fast when you need to do it. The downfall is that when you have to restore 1500+ accounts all at one shot, it’s *slow*. For every single account, it copies over the user data, rebuilds certain configs, clears out the package, brings the domain up and then moves onto the next account.
Now, from an customer support standpoint, this is somewhat nice when restoring a server as you can say “we’re restoring accounts that begin with A now, your account will go live as soon as it’s restored.” The bad side of this is that as soon as those domains start to go live, your server starts to go under load. As you restore more accounts, your configuration files get longer (taking longer to modify/update), Apache starts serving requests, mail is being processed, additional load is placed on the server.
This, gets very slow, very quickly.
Now back in the day, we had a set of servers that housed upwards of 8000+ accounts per machine. They were super lighweight, averaging 25-50M of data and *no* webtraffic, however with a standard pkgacct/restorepkg procedure, at 30-45 seconds per account, we were looking at somewhere in the neighborhood of 2-3 days to fully restore a server. From a customer support standpoint, this *SUCKS* You’ve got thousands of users down for multiple days while pkgacct sits there and does it’s thing… Not pretty, in fact, just plain unacceptable.
Necessity being the mother of invention, myself and another admin set out to find the “better way” of doing this. These accounts were only using around 10 or so G of disk space in total on the machine, it was fast hardware, there *HAD* to be a better way to do this and keep everything in cPanel happy. One highly tuned NFS mount later (jumbo frames on a private network are your friend here, trust me. :P), and we had our methods in place.
If you open up /scripts/cpbackup on any cPanel machine (checked on 11.25 for this instance) and go to line 240, you’ll see two arrays. @FILES and @DIRS. These are the core files that cPanel needs to do it’s thing.
my @FILES = qw(
my @DIRS = qw(
You’ll also need to get /home/ (or wherever your user accounts are stored) and your MySQL DBs as well.
So, rsync these files and directories to your backup server, do a ‘mysqldump -A’ for the global databases (careful with remove mysql servers, this can get ugly) and rsync on over /home/ to your NFS mount and voila! You’ve got EVERYTHING necessary to do a super fast backup (line speed and disk speed being your limitations here) of your cPanel server. Now subsequent backups using rsync will only update files that change, not the entire home directory, so they should be fairly lightweight and fast as well. You can also get jiggy with –link-dest options in Rsync to create multiple nightly snapshots in a differential format as well.
Now, if the proverbial poo hits the cooling device, you perform a fresh install of your OS of choice, grab your cPanel installer and run it, and then just reverse rsync from your backup mount (make sure to use –delete!). After that, one /scripts/upcp –force and your server is fully live!
In the case I mentioned above, this cut a server restoration down from days, to about an hour. It allowed us to make 150+ cPanel servers perform nightly backups (a huge selling point to this day) and all with the tools built into a standard CentOS + cPanel setup.
If there’s any desire for a script to handle this, just go ahead and post in the comments below and I’ll whip something up nice and fast for you. It’s fairly trivial and you can have this rolling in your operation in no time at all.
One item of note, you lose the ability to do single account restores from backup with this method. It *is* an all or nothing, disaster recovery shot. You can always copy over individual files and directories, but you lose the elegance of the cPanel native granular backups.