ServerAdmins.NET

This took me awhile to figure out today, so I figure it’s worth a post (and we all know I’m WAAAY behind on those!).

The problem I ran into was with an OpenStack (diablo build) instance that we use internally at my work. To make a long story short, I chose to go around the nova-network method of routing and use my own hardware gateway as opposed to using the software appliance provided by the OpenStack setup.

Now, a brief bit of Background here.

1. Openstack is a “Virtualization Fabric”, i.e., a piece of software that ties together hypervisors with a nice API. It handles the networking, connection to the hypervisors, sercurity and let’s you access everything with an API. This is the same software that the Rackspace Cloud is setup and comparable to what VPS.Net uses (onapp.com).

2. Openstack is comprised of components, nova-network handles the networking, routing and firewalling. Nova-api handles the API calls. Nova-scheduler handles all the scheduling. Nova-compute actually interfaces with the hypervisors and creates/manages the VMs with nova-volume providing an EBS style of volume provisioning.

3. When you provision a VM, it automatically routes through the machine you’ve designated to run nova-network, I didn’t like this.

So, I decided to change how the networking on the VMs routes to the public internet. There’s a service called ‘dnsmasq’ that runs on the nova-network machine which hands out IP and basic configuration information to your VMs when they come up. It gives your VMs their IP, Gateway, subnet mask, resolvers and a few other items as well. So here’s, where things needed to change.

Step 1. Know your real network gateway!

Traditionally in an openstack setup, your VMs will route like this…

VM->Nova-network appliance->network gateway->Internet.

I was modifying it to route as follows…

VM->Network Gateway->Internet.

This is handled by adding one line to your nova.conf file, and creating a super basic configuation file to go along with it. By adding ‘–dnsmasq_config_file=/etc/nova/dnsmasq.conf’ to your nova.conf file, and ensuring that /etc/nova/dnsmasq.conf looks like the following snippet, you’ll 1. route through your own gateway, and 2. use your own resolvers.

—-
dhcp-option=6,10.1.100.254
dhcp-option=3,10.1.100.254
—-

dhcp-option 6 should be set to the resolver you want to use, and dhcp-option 3 should be set to the gateway you want to use.

After that, do a ‘killall dnsmasq; /etc/init.d/nova-network restart’, and then restart your instances and you should be good to go!

Guys and Gals,

Awhile back, I had a list of URLs that were handy to me, as an admin. Mailchecker sites that’d hit up SPF/MX/RBL/header checks, DNS verification sites and gimmicky little things like “what is my ip” that’d get used fairly frequently. The sad thing is that the vast majority of my old go-to sites are now gimped in favor of “please pay us”, inundated with ads, or simply gone to hell in a handbasket. What’s an admin to do?

Code. That’s what an admin is to do. So I present to you, the very first alpha release of MXReport.com, my site that’ll hopefully contain all of those gimpy little scripts that have been so handy in the past.

The goal of MXReport.com are as follows.

Fast. It has to be fast.
Curl/wget/LWPable (or your web autobot dejour).
Efficient. No bumbling through 234234 menus to find the one little script you want.
Fast.
Free (no ads either).
Open. (github links will be dropped once I get things a bit more under control)
oh yeah and did I mention, Fast?

With that in mind, please feel free to kick around http://mxreport.com. Is it proper and fully ready? Nah. Does it do all the fancy stuff? Nope. Does it do RBL Checks quickly? Yep! Sure, there’s bugs, which are fairly easy to trip over once you start playing with input, but those should be resolved fairly soon. In the meantime, kick it around and let me know what you think. If you’ve got ideas, please feel free to send them on over.

This isn’t something for me. This is something for Us.

Hey guys, just so you know, I’m in the process of rebuilding and restoring serveradmins.NET. I hit an odd bug on the FreeBSD 7.4->8.2 upgrade which resulted in the default FreeBSD kernel simply not booting due to ACPI issues on this hardware.

As this machine is colo’d far away, and important to have live, I opted for the reinstall and rebuild quickly rather than running down the bug.

Please tolerate the breakage for a little bit, I’ll get you all sorted out quickly.

Edit: Done. Things should be good again now, sorry for the url breakage.

Hey there!

I figure it’s been a little bit, so I’ve gone ahead and decided to update the XCache installer for version 1.3.0, and my favorite control panel, cPanel.

This is incredibly easy to do and should get a basic/barebones XCache installation up and going fairly quickly on your CentOS + cPanel machine.

First, let’s grab our XCache sources…

cd /usr/src/
wget http://xcache.lighttpd.net/pub/Releases/1.3.0/xcache-1.3.0.tar.gz


Now, go ahead and unarchive the source, and change into the XCache build dir…
tar -xzvf xcache-1.3.0.tar.gz
cd xcache-1.3.0


Let’s prep our sources for the current PHP ecosystem…

root@SERVER [/usr/src/xcache-1.3.0]# phpize
Configuring for:
PHP Api Version: 20041225
Zend Module Api No: 20060613
Zend Extension Api No: 220060519


Now that should have gotten everything sorted with our sources so the XCache install is ready for our PHP version and environment, Let’s go ahead and kick off the build! Keep in mind, this build should be fairly fast and lightweight.


./configure && make && make install

The above line basically says “Run configure, if that is successful w/o errors, run a make, and if that finishes w/o error, run the install”. Once this is complete, you should see a line that says something like the following…

Installing shared extensions: /usr/local/lib/php/extensions/no-debug-non-zts-20060613/

This is going to be where our xcache.so file is located, however when cPanel builds your php.ini file, it’s going to have that set already. So getting the base module loaded is as simple as running the following command.

echo "extension=xcache.so" >> /usr/local/lib/php.ini

After that, you should be able to do a quick check of the PHP CLI and verify that it loaded properly with the ‘php -v’ command…

root@SERVER [/usr/src/xcache-1.3.0]# php -v
PHP 5.2.13 (cli) (built: Jun 16 2010 09:27:33)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies
with XCache v1.3.0, Copyright (c) 2005-2009, by mOo


And there you go, you can safely restart Apache now and you should be serving cached content. Please keep in mind, you’ll need to configure your XCache setup to ensure everything is working properly, all we’ve done here is install the base module.

I’d suggest taking a look at our other article regarding the tuning of XCache for a bit more information on this.

Quick one here, mostly for my own notes than anything else.

An easy way to see the existing runlevel is with the ‘who -r’ command.


root@server:~ # who -r
run-level 3 May 22 03:54 last=S


Here we show the last boot and current runlevel (3). In order to modify this, you can open up /etc/inittab and look for the following line.


id:3:initdefault:


This shows that runlevel 3 is the default one kicked off by init on a bootup.

For what it’s worth, this refers to the directories in /etc/rc.d/rcX.d/, where X is #. The number correlates to the runlevel of your machine, and all of the scripts stored in that directory are executed when that runlevel is enabled.

TMYK!

Recently I’ve caught myself writing more code on my Macbook and for the first time in… YEARS, not needing an internet connection to actually be productive. To be quite honest, it’s a bit weird.

So, the first thing I wanted was the ability to run/exec CGI via a webserver. At first, I was thinking I’d go about a source build of Apache, and then I remembered OSX comes with one built in. Huzzah!

So, first let’s enable the builtin webserver.

System Preferences -> Sharing, and then click on “Web Sharing”. Go ahead and close out the window now, you’ve just fired off Apache. (hooray!). You should now be able to open your browser of choice and go to http://127.0.0.1/~

Now, the document root is going to be off of your user account in /sites/. My user is chrismm on my machine, so /Users/chrismm/Sites/ is my public HTML. Any web content you want to view with the above URL needs to go off of this directory. Go ahead and add the obligatory "Helloworld.html", I'll wait.

Now that that's done and functioning, I needed to get some hawt hawt Perl/CGI action going on. This, was incredibly easy.

OSX's main httpd.conf does a per-user http include in /private/etc/apache2/users/.conf, so once again for me, it was /private/etc/apache2/users/chrismm.conf. This file contains just a super basic Directory configuration for apache to parse and we only need to make one quick change here…

First and foremost, we need to be the root user, so go ahead and open up Terminal (Finder->Applications->Utilities -> Terminal), and type ‘sudo su -’ . Here, you’ll be prompted to enter your password.

Keep in mind as soon as you do this, you are the root user on your machine, ***BE VERY CAREFUL AS IF YOU ARE NOT YOU CAN END UP WITH A WORTHLESS WORKSTATION ***.

Now, go ahead and open up the include for your user, at /private/etc/apache2/users/.conf, and make the following changes.

Original:

maynard:beta root# cat /private/etc/apache2/users/chrismm.conf

Options Indexes MultiViews
AllowOverride None
Order allow,deny
Allow from all




maynard:beta root# cat /private/etc/apache2/users/chrismm.conf

Options Indexes MultiViews ExecCGI
AllowOverride None
Order allow,deny
Allow from all



See that? All I added was “ExecCGI” at the end of that Options statement. We’re halfway done now!

One more change…

Go ahead and open up ‘/private/etc/apache2/original/httpd.conf’ in your text editor of choice, and search for ” #AddHandler cgi-script .cgi”

Go ahead and remove the ‘#’ mark, save and exit.

Now, let’s restart the webserver…


killall -HUP httpd


Done!

And now, you can use your CGI and be happy.

A little post here, more for reference than anything… Here’s a list of the common ports used by cPanel for your firewall building pleasure…

TCP
21 FTP
22 SSH
25 SMTP
26 SMTP (Alternative port, configurable via WHM)
80 HTTP
110 POP3
143 IMAP
443 HTTPS
465 SMTPS
993 IMAP4
995 POP3
2082 cPanel (NON Encrypted)
2083 cPanel (HTTPS)
2086 WHM (NON Encrypted)
2087 WHM (Encrypted)
2095 Webmail (NON Encrypted)
2096 Webmail (Encrypted)
3306 MySQL (Only if allowing Remote Connections)

–UDP
53 DNS

It should be noted that if you’re firewalling off your server, you should open up a portrange for the FTP PassivePort Rolloff. Typically these are higher up ports and a wide range, such as 30000-50000. You’ll also need to modify your FTPd configuration to use this portrange as well. If you don’t, you’ll see issues with FTP connections dropping, successful logins, but a hang, pause or delay when dong an ls or beginning a transfer that results in a dropped connection. Typically this will affect ftp clients that are behind a firewall/router doing NAT translations.

Anyway, that’s it, quick and simple. Enjoy

Of all the different OS’s and the multitude of firewall implementations for each, my favorite has to be FreeBSD’s IPFW ruleset. It’s powerful, super efficient, and most of all, *EASY* to read!

So I’m just going to paste a sample firewall below, notate it heavily for you, and show you how to get it enabled.

For newer FreeBSD builds, it’s enough to add firewall_enable="YES" to your /etc/rc.conf file and reboot. This will load all the necessary kernel modules, and get you in shape. You should know that by default FreeBSD defaults to a ‘default deny’ policy. This is *VERY* important to know. What it means is that by default, your server isn’t going to open up any ports. You *WILL* be locked out if you do not have a firewall in place. Don’t say I didn’t warn you!

So, with that being said, let’s get our firewall in built and in place, and then we’ll go about adding in the necessary options to ensure it comes up properly on boot. It should noted here that you can build in a “Default to accept” option into the kernel or “firewall_type=”open” to the /etc/rc.conf to start up wide open. This is the suggested practice as the chances are, you’re not close enough to your server to console it.

Let’s get our basic firewall going. We’re going to open up 22 for SSH, 80 for web traffic, 25 for SMTP and 110/143 for POP3/IMAP mail. The syntax is simple enough you should be able to customize it for your own needs, but always make sure your SSH port is accessible. Also if you run SSH on an alternative port, you should modify that here. You don’t want to lock yourself out now, do you?

So go ahead, login as root to your server, open up your text editor of choice. For the purpose of this post, I’m going to assume you’re creating the file /etc/firewall.sh. If not, modify where appropriate. You should also make sure this file is executable by root, otherwise it won’t fire off properly on reboot.


#!/bin/sh

Done!

Save your /etc/firewall.sh file, then make sure permissions are right on it…


chmod 755 /etc/firewall.sh && chown root:wheel /etc/firewall.sh


Open up your handy text editor again, and add the following lines to /etc/rc.conf at the bottom.


firewall_enable="YES"
firewall_script="/etc/firewall.sh"


Done!

Please keep in mind that if you set “firewall_type=open” above, it may overrule this firewall_script variable, so it should be removed.

Congrats, reboot and enjoy your new firewall.

If this is something you’re interested in, leave a comment and let me know how and I can write a few more articles on more complex configurations quite easily.

I’ve run my share of Hosting operations as a Sr. Admin before and one thing that always *killed* me… Server failures and restorations.

Every shop I’ve been in has primarily been a cPanel shop. It’s what the market demanded, so it’s what we sold. Not only was it simply “what the masses wanted”, it offered an entire suite of scripts, apps and A+ quality support and upgrades that almost brought a tear to my eye.

However, the one thing it did not provide, was a quick n’ dirty backup and restore system, let me explain.

cPanels backup system is based around the single user ideology. When you backup a server, it does this on a per cpuser account basis. Now this has it’s pros and it’s cons. It’s nice in the fact that restoring/migrating/tinkering with a single account is easy and *very* fast when you need to do it. The downfall is that when you have to restore 1500+ accounts all at one shot, it’s *slow*. For every single account, it copies over the user data, rebuilds certain configs, clears out the package, brings the domain up and then moves onto the next account.

Now, from an customer support standpoint, this is somewhat nice when restoring a server as you can say “we’re restoring accounts that begin with A now, your account will go live as soon as it’s restored.” The bad side of this is that as soon as those domains start to go live, your server starts to go under load. As you restore more accounts, your configuration files get longer (taking longer to modify/update), Apache starts serving requests, mail is being processed, additional load is placed on the server.

This, gets very slow, very quickly.

Now back in the day, we had a set of servers that housed upwards of 8000+ accounts per machine. They were super lighweight, averaging 25-50M of data and *no* webtraffic, however with a standard pkgacct/restorepkg procedure, at 30-45 seconds per account, we were looking at somewhere in the neighborhood of 2-3 days to fully restore a server. From a customer support standpoint, this *SUCKS* You’ve got thousands of users down for multiple days while pkgacct sits there and does it’s thing… Not pretty, in fact, just plain unacceptable.

Necessity being the mother of invention, myself and another admin set out to find the “better way” of doing this. These accounts were only using around 10 or so G of disk space in total on the machine, it was fast hardware, there *HAD* to be a better way to do this and keep everything in cPanel happy. One highly tuned NFS mount later (jumbo frames on a private network are your friend here, trust me. ), and we had our methods in place.

If you open up /scripts/cpbackup on any cPanel machine (checked on 11.25 for this instance) and go to line 240, you’ll see two arrays. @FILES and @DIRS. These are the core files that cPanel needs to do it’s thing.


my @FILES = qw(
/etc/exim.conf
/etc/exim.conf.local
/etc/exim.conf.localopts
/etc/namedb/named.conf
/etc/rc.conf
/etc/named.conf
/etc/proftpd.conf
/etc/localdomains
/etc/httpd/conf/httpd.conf
/usr/local/apache/conf/httpd.conf
/etc/group
/etc/shadow
/etc/master.passwd
/etc/passwd
/etc/fstab
/root/.my.cnf
/etc/ips
/etc/ips.remotemail
/etc/ips.remotedns
/etc/reservedips
/etc/reservedipreasons
/etc/quota.conf
/etc/wwwacct.conf
/etc/remotedomains
/etc/rndc.conf
/etc/secondarymx
/etc/my.cnf
/usr/local/cpanel/3rdparty/interchange/interchange.cfg
);

my @DIRS = qw(
/etc/namedb
/var/lib/rpm
/var/lib/named/chroot/var/named/master
/etc/valiases
/etc/proftpd
/var/named
/etc/vfilters
/var/cpanel
/var/spool/cron
/var/cron/tabs
/var/spool/fcron
/usr/local/frontpage
/var/log/bandwidth
/etc/vdomainaliases
/usr/share/ssl
/etc/ssl
/var/ssl
/usr/local/cpanel/3rdparty/mailman
);


You’ll also need to get /home/ (or wherever your user accounts are stored) and your MySQL DBs as well.

So, rsync these files and directories to your backup server, do a ‘mysqldump -A’ for the global databases (careful with remove mysql servers, this can get ugly) and rsync on over /home/ to your NFS mount and voila! You’ve got EVERYTHING necessary to do a super fast backup (line speed and disk speed being your limitations here) of your cPanel server. Now subsequent backups using rsync will only update files that change, not the entire home directory, so they should be fairly lightweight and fast as well. You can also get jiggy with –link-dest options in Rsync to create multiple nightly snapshots in a differential format as well.

Now, if the proverbial poo hits the cooling device, you perform a fresh install of your OS of choice, grab your cPanel installer and run it, and then just reverse rsync from your backup mount (make sure to use –delete!). After that, one /scripts/upcp –force and your server is fully live!

In the case I mentioned above, this cut a server restoration down from days, to about an hour. It allowed us to make 150+ cPanel servers perform nightly backups (a huge selling point to this day) and all with the tools built into a standard CentOS + cPanel setup.

If there’s any desire for a script to handle this, just go ahead and post in the comments below and I’ll whip something up nice and fast for you. It’s fairly trivial and you can have this rolling in your operation in no time at all.

One item of note, you lose the ability to do single account restores from backup with this method. It *is* an all or nothing, disaster recovery shot. You can always copy over individual files and directories, but you lose the elegance of the cPanel native granular backups.

Okay, I’m lazy. I fully admit it. Want proof? Instead of writing up a huge long post articulating something with awesome analogies, I’m only going to talk about one command today.

Fuser.

Why?

fuser is awesome. Not awesome in a “run it and it fixes everything” way, but awesome in a “What in the hell is binding to this port??” kind of way. Two classic scenarios where this is handy…

1. Apache won’t start, “Can’t bind to port ::80″ or “Can’t bind to port ::443″, etc. This typically means something else is already tied to that port, and won’t relinquish it…

2. A security scan of your machine shows something funny running on port 6667… You didn’t start this or know what it is.

What to do now? Well you can sift through netstat output, but that’s, well, boring and slightly annoying.

netstat output

[root@vps ~]# netstat -anp |grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3266/httpd
tcp 0 0 127.0.0.1:58725 127.0.0.1:80 TIME_WAIT -
tcp 0 0 10.10.10.10:2078 192.168.1.23:63024 ESTABLISHED 18088/cpdavd - acce
unix 3 [ ] STREAM CONNECTED 49222880 11574/dovecot-auth /var/run/dovecot/login/default
unix 2 [ ] DGRAM 6804658 14078/named


Okay now we see that 3266/httpd is running on 80. Then we do this to find the process..


[root@vps ~]# ps auxwww |grep 3266
nobody 3266 0.0 0.3 65704 3516 ? S 17:05 0:00 /usr/local/apache/bin/httpd -k start -DSSL
root 21665 0.0 0.0 6024 640 pts/2 S+ 18:51 0:00 grep 3266
[root@vps ~]#


Okay there we go…

Now for hacked systems, this could (and probably) is fully forged for a lot of remote shells. Going back to my previous post at http://serveradmins.net/ssh-on-nonstandard-ports-how-to-not-do-it/ which talks about priveleged ports, you could in theory have trojaned ‘ps’, top, etc masking that real process. It may *look* like httpd, but bound to a port like 23425… So dont’t trust that too much, but a bit on that in a second.

The fuser approach…


[root@vps ~]# fuser -n tcp 80
80/tcp: 3266 3267 3268 3269 3271 16078 18274
[root@vps ~]#


Oh look at that a list of all pids bound to that port. Nice, clean, to the point and easily parsable. fuser rocks.

Now a bit more about the masked processes… To run those down, here’s a quick tip. Forget ps/top and your other normal utilities, /proc/ is your friend here…

Proc looks like this on a linux box…

[root@vps ~]# cd /proc/
[root@vps proc]# ls -al
total 1
dr-xr-xr-x 78 root root 0 Jan 26 09:58 .
drwxr-xr-x 24 chrismm chrismm 1024 Feb 4 22:06 ..
dr-xr-xr-x 4 root root 0 Feb 5 05:05 1
dr-xr-xr-x 4 root root 0 Feb 5 05:05 11573
dr-xr-xr-x 4 root root 0 Feb 5 05:05 11574
dr-xr-xr-x 4 dovecot dovecot 0 Feb 5 05:05 11575
dr-xr-xr-x 4 dovecot dovecot 0 Feb 5 05:05 11576
...
...


These directories match the pids of the running process… So if you have something advertising itself as ‘httpd’ on port 234234 and you know it’s pid 3266, you’d just do the following…


[root@vps proc]# cd /proc/3266
[root@vps 3266]# ls -al
total 0
dr-xr-xr-x 4 nobody nobody 0 Feb 5 17:08 .
dr-xr-xr-x 78 root root 0 Jan 26 09:58 ..
-r-------- 1 root root 0 Feb 5 18:56 auxv
-r--r--r-- 1 root root 0 Feb 5 17:08 cmdline
-rw-r--r-- 1 root root 0 Feb 5 18:56 coredump_filter
-r--r--r-- 1 root root 0 Feb 5 18:56 cpuset
lrwxrwxrwx 1 root root 0 Feb 5 18:54 cwd -> /
-r-------- 1 root root 0 Feb 5 18:56 environ
lrwxrwxrwx 1 root root 0 Feb 5 17:10 exe -> /usr/local/apache/bin/httpd
dr-x------ 2 root root 0 Feb 5 18:49 fd
?r--r--r-- 1 root root 0 Feb 5 18:56 io
-r-------- 1 root root 0 Feb 5 18:56 limits
-rw-r--r-- 1 root root 0 Feb 5 18:56 loginuid
-r--r--r-- 1 root root 0 Feb 5 18:54 maps
-rw------- 1 root root 0 Feb 5 18:56 mem
-r--r--r-- 1 root root 0 Feb 5 18:56 mounts
-r-------- 1 root root 0 Feb 5 18:56 mountstats
-r--r--r-- 1 root root 0 Feb 5 18:56 numa_maps
-rw-r--r-- 1 root root 0 Feb 5 18:56 oom_adj
-r--r--r-- 1 root root 0 Feb 5 18:56 oom_score
lrwxrwxrwx 1 root root 0 Feb 5 18:54 root -> /
-r--r--r-- 1 root root 0 Feb 5 18:56 schedstat
-r-------- 1 root root 0 Feb 5 18:56 smaps
-r--r--r-- 1 root root 0 Feb 5 17:08 stat
-r--r--r-- 1 root root 0 Feb 5 17:10 statm
-r--r--r-- 1 root root 0 Feb 5 17:08 status
dr-xr-xr-x 3 nobody nobody 0 Feb 5 18:56 task
-r--r--r-- 1 root root 0 Feb 5 18:56 wchan
[root@vps 3266]#


Bam, there you go. cwd and exe are the things you’re looking for It shows you the dir it was spawned from (typically a users homedirectory on a shared hosting machine) and the full path/binary actually being executed (usually lame perl/php listeners)… Also the ./fd/ folder is kind of neat as it shows you all the open file handles tied up by that pid as well.

Anyway, /proc/ examination too, is for another day, I just wanted to ramble on about one of my favorite, neat little single use utilities that no one else seems to know about. fuser. Enjoy. =)