Skip to content

ServerAdmins.NET

Stuff for Server Admins…

A little post here, more for reference than anything… Here’s a list of the common ports used by cPanel for your firewall building pleasure…

TCP
21 FTP
22 SSH
25 SMTP
26 SMTP (Alternative port, configurable via WHM)
80 HTTP
110 POP3
143 IMAP
443 HTTPS
465 SMTPS
993 IMAP4
995 POP3
2082 cPanel (NON Encrypted)
2083 cPanel (HTTPS)
2086 WHM (NON Encrypted)
2087 WHM (Encrypted)
2095 Webmail (NON Encrypted)
2096 Webmail (Encrypted)
3306 MySQL (Only if allowing Remote Connections)

–UDP
53 DNS

It should be noted that if you’re firewalling off your server, you should open up a portrange for the FTP PassivePort Rolloff. Typically these are higher up ports and a wide range, such as 30000-50000. You’ll also need to modify your FTPd configuration to use this portrange as well. If you don’t, you’ll see issues with FTP connections dropping, successful logins, but a hang, pause or delay when dong an ls or beginning a transfer that results in a dropped connection. Typically this will affect ftp clients that are behind a firewall/router doing NAT translations.

Anyway, that’s it, quick and simple. :) Enjoy

Of all the different OS’s and the multitude of firewall implementations for each, my favorite has to be FreeBSD’s IPFW ruleset. It’s powerful, super efficient, and most of all, *EASY* to read!

So I’m just going to paste a sample firewall below, notate it heavily for you, and show you how to get it enabled.

For newer FreeBSD builds, it’s enough to add firewall_enable="YES" to your /etc/rc.conf file and reboot. This will load all the necessary kernel modules, and get you in shape. You should know that by default FreeBSD defaults to a ‘default deny’ policy. This is *VERY* important to know. What it means is that by default, your server isn’t going to open up any ports. You *WILL* be locked out if you do not have a firewall in place. Don’t say I didn’t warn you!

So, with that being said, let’s get our firewall in built and in place, and then we’ll go about adding in the necessary options to ensure it comes up properly on boot. It should noted here that you can build in a “Default to accept” option into the kernel or “firewall_type=”open” to the /etc/rc.conf to start up wide open. This is the suggested practice as the chances are, you’re not close enough to your server to console it. :)

Let’s get our basic firewall going. We’re going to open up 22 for SSH, 80 for web traffic, 25 for SMTP and 110/143 for POP3/IMAP mail. The syntax is simple enough you should be able to customize it for your own needs, but always make sure your SSH port is accessible. Also if you run SSH on an alternative port, you should modify that here. You don’t want to lock yourself out now, do you?

So go ahead, login as root to your server, open up your text editor of choice. For the purpose of this post, I’m going to assume you’re creating the file /etc/firewall.sh. If not, modify where appropriate. :) You should also make sure this file is executable by root, otherwise it won’t fire off properly on reboot.


#!/bin/sh

set -e

#First, let's clear out any chance of conflicting with other FreeBSD firewall configurations
# and make sure we're starting from a fresh slate.

/sbin/sysctl net.inet.ip.forwarding=0 >/dev/null
/sbin/ipfw -q list >/dev/null 2>&1 || /sbin/kldload ipfw
(/sbin/ipf -D) >/dev/null 2>&1 || true
(/sbin/kldunload ipl) >/dev/null 2>&1 || true
(/sbin/pfctl -d) >/dev/null 2>&1 || true
(/sbin/kldunload pf) >/dev/null 2>&1 || true
/sbin/ipfw -q /dev/stdin < < EOF
flush
delete set 31

#Open up our Loopback device. There's almost never any reason to filter this.
add allow ip from any to any via lo0

#Allow checking/maintenance of stateful rulesets
add check-state

#Kill off any active/open sessions, pre-fw init. We do this
#to ensure that any connection to unauthorized ports is dealt with
#and that all connections adhere to the policy...

add reset tcp from any to any established

##########################
## Add Inbound Service Allowances ##
##########################
#Port 80: www
add allow tcp from any to me 80 setup in
#Port 22: SSH - Stateful connection (it's going to maintain a connection, not come and go)
add allow tcp from any to me 22 setup in keep-state

#Port 21, and 30000-50000, FTP and Passive port rolloff.
add allow tcp from any to me 21 setup in
add allow tcp from any to me 30000-50000 setup in keep-state

#Port 53: DNS
add allow udp from any to me 53 in

#Port 25, 110, 143. SMTP, POP3 and IMAP
add allow tcp from any to me 25 in
add allow tcp from any to me 110 in
add allow tcp from any to me 143 in

#ICMP/ping requests should be allowed through, fun stuff happens
#if you don't allow this. (see ptmu)
add icmp from any to me icmptypes 8 in

#################################
## Lets Deny some Packets. WOOOT ##
#################################

#These rules denies everything else, not explicitly listed above
add deny tcp from any to me setup in
add deny ip from any to me in

##################
## Outbound Ahoy! ##
##################

#Wide open outbound filtering. You could modify this section to default to deny
#And then allow specific ports out. I'm not going to do that here, but you should
#be able to figure it out. :)

add allow tcp from me to any setup out keep-state
add allow ip from me to any out keep-state

###########################
## Last chain to make sure ##
###########################

#One more time, just to be positive...
add deny tcp from any to any setup
add deny ip from any to any
enable firewall
EOF

#I set this because the default FreeBSD behavior is to keep a table open
#for a session for 1 hour. That's a LONG time on a production server. We
#Go for 10 mins here, but could be lowered drastically.

#Set TTL on Dynamic Rules to 10 Mins. Formerly 1 hour.
/sbin/sysctl net.inet.ip.fw.dyn_ack_lifetime=600 >/dev/null

Done!

Save your /etc/firewall.sh file, then make sure permissions are right on it…


chmod 755 /etc/firewall.sh && chown root:wheel /etc/firewall.sh

Open up your handy text editor again, and add the following lines to /etc/rc.conf at the bottom.


firewall_enable="YES"
firewal_script="/etc/firewall.sh"

Done!

Please keep in mind that if you set “firewall_type=open” above, it may overrule this firewall_script variable, so it should be removed.

Congrats, reboot and enjoy your new firewall. :)

If this is something you’re interested in, leave a comment and let me know how and I can write a few more articles on more complex configurations quite easily. :)

I’ve run my share of Hosting operations as a Sr. Admin before and one thing that always *killed* me… Server failures and restorations.

Every shop I’ve been in has primarily been a cPanel shop. It’s what the market demanded, so it’s what we sold. Not only was it simply “what the masses wanted”, it offered an entire suite of scripts, apps and A+ quality support and upgrades that almost brought a tear to my eye. :)

However, the one thing it did not provide, was a quick n’ dirty backup and restore system, let me explain.

cPanels backup system is based around the single user ideology. When you backup a server, it does this on a per cpuser account basis. Now this has it’s pros and it’s cons. It’s nice in the fact that restoring/migrating/tinkering with a single account is easy and *very* fast when you need to do it. The downfall is that when you have to restore 1500+ accounts all at one shot, it’s *slow*. For every single account, it copies over the user data, rebuilds certain configs, clears out the package, brings the domain up and then moves onto the next account.

Now, from an customer support standpoint, this is somewhat nice when restoring a server as you can say “we’re restoring accounts that begin with A now, your account will go live as soon as it’s restored.” The bad side of this is that as soon as those domains start to go live, your server starts to go under load. As you restore more accounts, your configuration files get longer (taking longer to modify/update), Apache starts serving requests, mail is being processed, additional load is placed on the server.

This, gets very slow, very quickly.

Now back in the day, we had a set of servers that housed upwards of 8000+ accounts per machine. They were super lighweight, averaging 25-50M of data and *no* webtraffic, however with a standard pkgacct/restorepkg procedure, at 30-45 seconds per account, we were looking at somewhere in the neighborhood of 2-3 days to fully restore a server. From a customer support standpoint, this *SUCKS* You’ve got thousands of users down for multiple days while pkgacct sits there and does it’s thing… Not pretty, in fact, just plain unacceptable.

Necessity being the mother of invention, myself and another admin set out to find the “better way” of doing this. These accounts were only using around 10 or so G of disk space in total on the machine, it was fast hardware, there *HAD* to be a better way to do this and keep everything in cPanel happy. One highly tuned NFS mount later (jumbo frames on a private network are your friend here, trust me. :P ), and we had our methods in place.

If you open up /scripts/cpbackup on any cPanel machine (checked on 11.25 for this instance) and go to line 240, you’ll see two arrays. @FILES and @DIRS. These are the core files that cPanel needs to do it’s thing.


my @FILES = qw(
/etc/exim.conf
/etc/exim.conf.local
/etc/exim.conf.localopts
/etc/namedb/named.conf
/etc/rc.conf
/etc/named.conf
/etc/proftpd.conf
/etc/localdomains
/etc/httpd/conf/httpd.conf
/usr/local/apache/conf/httpd.conf
/etc/group
/etc/shadow
/etc/master.passwd
/etc/passwd
/etc/fstab
/root/.my.cnf
/etc/ips
/etc/ips.remotemail
/etc/ips.remotedns
/etc/reservedips
/etc/reservedipreasons
/etc/quota.conf
/etc/wwwacct.conf
/etc/remotedomains
/etc/rndc.conf
/etc/secondarymx
/etc/my.cnf
/usr/local/cpanel/3rdparty/interchange/interchange.cfg
);

my @DIRS = qw(
/etc/namedb
/var/lib/rpm
/var/lib/named/chroot/var/named/master
/etc/valiases
/etc/proftpd
/var/named
/etc/vfilters
/var/cpanel
/var/spool/cron
/var/cron/tabs
/var/spool/fcron
/usr/local/frontpage
/var/log/bandwidth
/etc/vdomainaliases
/usr/share/ssl
/etc/ssl
/var/ssl
/usr/local/cpanel/3rdparty/mailman
);

You’ll also need to get /home/ (or wherever your user accounts are stored) and your MySQL DBs as well.

So, rsync these files and directories to your backup server, do a ‘mysqldump -A’ for the global databases (careful with remove mysql servers, this can get ugly) and rsync on over /home/ to your NFS mount and voila! You’ve got EVERYTHING necessary to do a super fast backup (line speed and disk speed being your limitations here) of your cPanel server. Now subsequent backups using rsync will only update files that change, not the entire home directory, so they should be fairly lightweight and fast as well. You can also get jiggy with –link-dest options in Rsync to create multiple nightly snapshots in a differential format as well.

Now, if the proverbial poo hits the cooling device, you perform a fresh install of your OS of choice, grab your cPanel installer and run it, and then just reverse rsync from your backup mount (make sure to use –delete!). After that, one /scripts/upcp –force and your server is fully live!

In the case I mentioned above, this cut a server restoration down from days, to about an hour. It allowed us to make 150+ cPanel servers perform nightly backups (a huge selling point to this day) and all with the tools built into a standard CentOS + cPanel setup.

If there’s any desire for a script to handle this, just go ahead and post in the comments below and I’ll whip something up nice and fast for you. It’s fairly trivial and you can have this rolling in your operation in no time at all.

One item of note, you lose the ability to do single account restores from backup with this method. It *is* an all or nothing, disaster recovery shot. You can always copy over individual files and directories, but you lose the elegance of the cPanel native granular backups.

Okay, I’m lazy. I fully admit it. Want proof? Instead of writing up a huge long post articulating something with awesome analogies, I’m only going to talk about one command today.

Fuser.

Why?

fuser is awesome. Not awesome in a “run it and it fixes everything” way, but awesome in a “What in the hell is binding to this port??” kind of way. Two classic scenarios where this is handy…

1. Apache won’t start, “Can’t bind to port ::80″ or “Can’t bind to port ::443″, etc. This typically means something else is already tied to that port, and won’t relinquish it…

2. A security scan of your machine shows something funny running on port 6667… You didn’t start this or know what it is.

What to do now? Well you can sift through netstat output, but that’s, well, boring and slightly annoying.

netstat output

[root@vps ~]# netstat -anp |grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3266/httpd
tcp 0 0 127.0.0.1:58725 127.0.0.1:80 TIME_WAIT -
tcp 0 0 10.10.10.10:2078 192.168.1.23:63024 ESTABLISHED 18088/cpdavd - acce
unix 3 [ ] STREAM CONNECTED 49222880 11574/dovecot-auth /var/run/dovecot/login/default
unix 2 [ ] DGRAM 6804658 14078/named

Okay now we see that 3266/httpd is running on 80. Then we do this to find the process..


[root@vps ~]# ps auxwww |grep 3266
nobody 3266 0.0 0.3 65704 3516 ? S 17:05 0:00 /usr/local/apache/bin/httpd -k start -DSSL
root 21665 0.0 0.0 6024 640 pts/2 S+ 18:51 0:00 grep 3266
[root@vps ~]#

Okay there we go…

Now for hacked systems, this could (and probably) is fully forged for a lot of remote shells. Going back to my previous post at http://serveradmins.net/ssh-on-nonstandard-ports-how-to-not-do-it/ which talks about priveleged ports, you could in theory have trojaned ‘ps’, top, etc masking that real process. It may *look* like httpd, but bound to a port like 23425… So dont’t trust that too much, but a bit on that in a second. :)

The fuser approach…


[root@vps ~]# fuser -n tcp 80
80/tcp: 3266 3267 3268 3269 3271 16078 18274
[root@vps ~]#

Oh look at that a list of all pids bound to that port. Nice, clean, to the point and easily parsable. fuser rocks. :)

Now a bit more about the masked processes… To run those down, here’s a quick tip. Forget ps/top and your other normal utilities, /proc/ is your friend here…

Proc looks like this on a linux box…

[root@vps ~]# cd /proc/
[root@vps proc]# ls -al
total 1
dr-xr-xr-x 78 root root 0 Jan 26 09:58 .
drwxr-xr-x 24 chrismm chrismm 1024 Feb 4 22:06 ..
dr-xr-xr-x 4 root root 0 Feb 5 05:05 1
dr-xr-xr-x 4 root root 0 Feb 5 05:05 11573
dr-xr-xr-x 4 root root 0 Feb 5 05:05 11574
dr-xr-xr-x 4 dovecot dovecot 0 Feb 5 05:05 11575
dr-xr-xr-x 4 dovecot dovecot 0 Feb 5 05:05 11576
...
...

These directories match the pids of the running process… So if you have something advertising itself as ‘httpd’ on port 234234 and you know it’s pid 3266, you’d just do the following…


[root@vps proc]# cd /proc/3266
[root@vps 3266]# ls -al
total 0
dr-xr-xr-x 4 nobody nobody 0 Feb 5 17:08 .
dr-xr-xr-x 78 root root 0 Jan 26 09:58 ..
-r-------- 1 root root 0 Feb 5 18:56 auxv
-r--r--r-- 1 root root 0 Feb 5 17:08 cmdline
-rw-r--r-- 1 root root 0 Feb 5 18:56 coredump_filter
-r--r--r-- 1 root root 0 Feb 5 18:56 cpuset
lrwxrwxrwx 1 root root 0 Feb 5 18:54 cwd -> /
-r-------- 1 root root 0 Feb 5 18:56 environ
lrwxrwxrwx 1 root root 0 Feb 5 17:10 exe -> /usr/local/apache/bin/httpd
dr-x------ 2 root root 0 Feb 5 18:49 fd
?r--r--r-- 1 root root 0 Feb 5 18:56 io
-r-------- 1 root root 0 Feb 5 18:56 limits
-rw-r--r-- 1 root root 0 Feb 5 18:56 loginuid
-r--r--r-- 1 root root 0 Feb 5 18:54 maps
-rw------- 1 root root 0 Feb 5 18:56 mem
-r--r--r-- 1 root root 0 Feb 5 18:56 mounts
-r-------- 1 root root 0 Feb 5 18:56 mountstats
-r--r--r-- 1 root root 0 Feb 5 18:56 numa_maps
-rw-r--r-- 1 root root 0 Feb 5 18:56 oom_adj
-r--r--r-- 1 root root 0 Feb 5 18:56 oom_score
lrwxrwxrwx 1 root root 0 Feb 5 18:54 root -> /
-r--r--r-- 1 root root 0 Feb 5 18:56 schedstat
-r-------- 1 root root 0 Feb 5 18:56 smaps
-r--r--r-- 1 root root 0 Feb 5 17:08 stat
-r--r--r-- 1 root root 0 Feb 5 17:10 statm
-r--r--r-- 1 root root 0 Feb 5 17:08 status
dr-xr-xr-x 3 nobody nobody 0 Feb 5 18:56 task
-r--r--r-- 1 root root 0 Feb 5 18:56 wchan
[root@vps 3266]#

Bam, there you go. cwd and exe are the things you’re looking for It shows you the dir it was spawned from (typically a users homedirectory on a shared hosting machine) and the full path/binary actually being executed (usually lame perl/php listeners)… Also the ./fd/ folder is kind of neat as it shows you all the open file handles tied up by that pid as well.

Anyway, /proc/ examination too, is for another day, I just wanted to ramble on about one of my favorite, neat little single use utilities that no one else seems to know about. fuser. Enjoy. =)

I’m feeling a bit lazy tonight, and wanted to get an update here, so for a bit I’ll show you a handy little tool to update your ports tree on FreeBSD. After that, I’ll show you the ugly, old method.

Quick and easy…

Newer versions of FreeBSD come equipped with the ‘portsnap’ utility. This, makes it *VERY* simple to update your ports tree.

For your first run, do this…

portsnap fetch && portsnap extract

This is going to grab a snapshot of the current ports tree, and simply extract it over your new tree, replacing *everything* as it goes. You should only run the ‘extract’ command the first time you run portsnap.

After that, you’ll want to run the following for any further updates…


portsnap fetch && portsnap update

Not only is this much quicker, it doesn’t overwrite everything. :)

If you want to use this in a cron’d task, you should use the ‘portsnap cron 1′ command. It should be noted the number appended to the end of this is the number in seconds that portsnap will randomize the start of the app from. For example, if you say ‘cron 2000′, portsnap will kick off *sometime* in the next 2000 seconds. The reasoning for this is for larger serverfarms. If you’re running that in cron on all of them and give portsnap a large window, it will keep them all from starting at the same time, loading the BSD servers and abusing your bandwidth. I used 1 in the command above as I wasn’t really looking to use that. :) Keep in mind this will only fetch the updates, you still need to update the tree afterwords…

A cron entry for this would look something like the following…


0 3 * * * root /usr/sbin/portsnap cron && /usr/sbin/portsnap update

So for normal, day to day operation once you’ve initialized your ports tree the following is what you’ll want to use and update.


/usr/sbin/portsnap fetch && /usr/sbin/portsnap update

Now, if you don’t have portsnap, you should use the following method to update your ports tree. We’re going to go oldschool with cvsup here.

First of all, let’s find our fastest cvsup mirror…


[root@R34 ~]# cd /usr/ports/sysutils/fastest_cvsup/
[root@R34 /usr/ports/sysutils/fastest_cvsup]# make && make install

This is going to install the ‘fastest_cvsup’ port… Afterwords, for the US locale, you can run the following to find your fastest cvsup mirror…


[root@R34 /usr/ports/sysutils/fastest_cvsup]# fastest_cvsup -c us
>> Querying servers in countries: us
--> Connecting to cvsup.us.freebsd.org [72.233.193.64]...
- server replied: ! Access limit exceeded; try again later
- time taken: 69.51 ms
--> Connecting to cvsup2.us.freebsd.org [130.94.149.166]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 27.19 ms
--> Connecting to cvsup3.us.freebsd.org [128.31.0.28]...
- server replied: ! Access denied
- time taken: 31.65 ms
--> Connecting to cvsup4.us.freebsd.org [149.20.64.73]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 55.77 ms
--> Connecting to cvsup5.us.freebsd.org [208.83.20.166]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 36.99 ms
--> Connecting to cvsup6.us.freebsd.org [64.202.113.190]...
* error: connect: Invalid argument
--> Connecting to cvsup7.us.freebsd.org [64.215.216.140]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 26.64 ms
--> Connecting to cvsup8.us.freebsd.org [216.165.129.134]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 6.23 ms
--> Connecting to cvsup9.us.freebsd.org [128.205.32.21]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 26.28 ms
--> Connecting to cvsup10.us.freebsd.org [69.147.83.48]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 54.01 ms
--> Connecting to cvsup11.us.freebsd.org [63.87.62.77]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 35.11 ms
--> Connecting to cvsup12.us.freebsd.org [128.205.32.24]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 26.86 ms
--> Connecting to cvsup13.us.freebsd.org [128.205.32.24]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 26.54 ms
--> Connecting to cvsup14.us.freebsd.org [216.87.78.137]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 34.63 ms
--> Connecting to cvsup15.us.freebsd.org [35.9.37.225]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 23.49 ms
--> Connecting to cvsup16.us.freebsd.org [128.143.108.35]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 23.47 ms
--> Connecting to cvsup17.us.freebsd.org [65.212.71.21]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 35.93 ms
--> Connecting to cvsup18.us.freebsd.org [128.205.32.84]...
- server replied: OK 17 0 SNAP_16_1h CVSup server ready
- time taken: 3026.06 ms

>> Speed Daemons:
- 1st: cvsup8.us.freebsd.org 6.23 ms
- 2st: cvsup16.us.freebsd.org 23.47 ms
- 3st: cvsup15.us.freebsd.org 23.49 ms
[root@R34 /usr/ports/sysutils/fastest_cvsup]#

Cvsup8 it is!

So now, let’s get our ports-supfile in place…


cp /usr/share/examples/cvsup/ports-supfile /root/

Now edit /root/ports-supfile and look for the following line…

*default host=CHANGE_THIS.FreeBSD.org

And modify it to read…
*default host=csup8.freebsd.org

Now, run the following to get *everything* up to date…


cvsup -g -L 2 /root/ports-supfile

Voila, you have an updated ports tree. :)

Hey there!

Today I wanted to focus on something that’s helping me do my job in a more efficient fashion. At a former workplace, I was responsible for ~200 high capacity webhosting machines, and a host of supporting machines. Back then, I was a huge fan of a management system mostly comprised of SSH Keys and a ton of bash scripts. It worked, quite well for the time, but if I could do it again, I’d go with a slightly more refined approach, which is what we’ll discuss today.

So, let’s get started. The first thing you’ll need is a working perl installation, a few devel libs and a handful of perl modules.

yum install gmp-devel
perl -MCPAN -e 'install Crypt::DH , Math::GMP, Net::SSH::Perl'

This is going to install the GMP math development libraries necessary for Math::GMP to compile. Math::GMP and Crypt::DH are prereqs for Net::SSH::Perl.

So once this is done, we can proceed. :)


#!/usr/local/bin/perl -w

use strict;
use warnings;
require Net::SSH::Perl;

#declare our login vars...

my $user = "root";
my $password = "SEKUREPASSWORD";
my $server = "localhost";

#Setup our SSH Connection...
my $ssh = Net::SSH::Perl->new($server,port=>22,use_pty=>1);

#Initiate out conneciton to the server...
$ssh->login($user, $password);

# Declare our variable for the request...
my $uptime;

# Run our SSH Command and retrieve the output...
($uptime) = $ssh->cmd("/usr/bin/uptime");

print "\n$uptime\n";

exit 0;

That’s a very basic/barebones SSH Connection script… If you have any questions or problems, please don’t hesitate to post in the comments. :)

Next up, we’ll go over a more complex variant of this script using subroutines and a few other nifty tricks. :)

Hey there!

I’m going to show you a few different ways to install Perl modules in a quick and easy way.

First up, the one liner. :)

perl -MCPAN -e 'install HTML::Template'
CPAN: CPAN::SQLite loaded ok (v0.199)
CPAN: LWP::UserAgent loaded ok (v5.834)
CPAN: Time::HiRes loaded ok (v1.9719)
Fetching with LWP:

http://www.stathy.com/CPAN/authors/01mailrc.txt.gz

CPAN: YAML loaded ok (v0.71)
Fetching with LWP:

http://www.stathy.com/CPAN/modules/02packages.details.txt.gz

Fetching with LWP:

http://www.stathy.com/CPAN/modules/03modlist.data.gz

Database was generated on Thu, 21 Jan 2010 20:40:30 GMT
Updating database file ...

Gathering information from index files ...
Obtaining current state of database ...
Populating database tables ...
.... snipped for brevity....
Running make install
Prepending /home/.cpan/build/HTML-Template-2.9-bALXdn/blib/arch /home/.cpan/build/HTML-Template-2.9-bALXdn/blib/lib to PERL5LIB for 'install'
Installing /usr/local/lib/perl5/site_perl/5.8.8/HTML/Template.pm
Appending installation info to /usr/local/lib/perl5/5.8.8/x86_64-linux/perllocal.pod
SAMTREGAR/HTML-Template-2.9.tar.gz
/usr/bin/make install UNINST=1 OTHERLDFLAGS=-L/usr/lib64 LDFLAGS=-L/usr/lib64 EXTRALIBDIR=/usr/lib64 -- OK
[root@vps ~]#

And there you go, quick and easy.

Now a lot of Perl modules are going to require other modules to be built, in which case, you’ll see something like this…

Writing Makefile for Net::SSH::Perl
---- Unsatisfied dependencies detected during ----
---- TURNSTEP/Net-SSH-Perl-1.34.tar.gz ----
Crypt::DSA [requires]
Convert::PEM [requires]
Crypt::RSA [requires]
Math::Pari [requires]
Crypt::IDEA [requires]
Digest::BubbleBabble [requires]
Crypt::DH [requires]
Math::GMP [requires]
Shall I follow them and prepend them to the queue
of modules we are processing right now? [yes]

Just go ahead and answer “yes” here, and let it continue… cpan *should* be smart enough to grab all of the required sources and build what you need, but sometimes, not so much. :) Perl modules are basically subroutines packaged into nice containers, ready to use. Now some of these require specific programs, libraries or even other perl modules to do what they do best. When a module has a large chain of dependencies and one of those fails, it can bring the whole show to a screeching halt.

For an example here, I’ll use Net::SSH::Perl, which happens to be what I use for a host of different things.

If you use the above listed one-liner to install it, i.e.,

perl -MCPAN -e 'install HTML::Template'

You’re going to end up seeing this…


Files=12, Tests=106, 1 wallclock secs ( 0.07 usr 0.04 sys + 0.40 cusr 0.14 csys = 0.65 CPU)
Result: PASS
TURNSTEP/Net-SSH-Perl-1.34.tar.gz
Tests succeeded but 2 dependencies missing (Crypt::DH,Math::GMP)
TURNSTEP/Net-SSH-Perl-1.34.tar.gz
[dependencies] -- NA
Running make install
make test had returned bad status, won't install without force

So, we have a dependency of Net::SSH::Perl that simply isn’t present. So let’s go ahead and get it installed…

On a RH Based system (CentOS/Trustix/RedHat Enterprise Linux), you can do the following…


yum install gmp-devel

On a Debian based distribution (Debian/Ubuntu, etc)

apt-get install libgmp-ocaml

On FreeBSD, I prefer prots builds personally, so let’s do the following…

cd /usr/ports/math/libgmp4
make && make install

So, now that you’ve got that taken care of, let’s proceed. :)


perl -MCPAN -e 'install Net::SSH::Perl'
...
...
Tests succeeded but one dependency not OK (Crypt::DH)
TURNSTEP/Net-SSH-Perl-1.34.tar.gz
[dependencies] -- NA
Running make install
make test had returned bad status, won't install without force

So, we need to build Crypt::DH… Apparently dependency handling isn’t too bright in this case. :) I’ll save you the trouble of the blow-by-blow here. We need to install Crypt::DH which depends on Math::BigInt::GMP. So, use your handy oneline skills, and get Math::BigInt::GMP installed, then do the same for Crypt::DH. You should now have a working Net::SSH::Perl installation. :)

You can specify multiple packages in the following way…


perl -MCPAN -e 'install Net::SSH, Term::ReadLine'

The other option, should cpan fail, is to just grab the module package yourself, which is typically a .tar.gz file, and perform the following.


wget http://search.cpan.org/CPAN/authors/id/T/TU/TURNSTEP/Net-SSH-Perl-1.34.tar.gz
tar -xzvf Net-SSH-Perl-1.34.tar.gz
cd ./Net-SSH-Perl-1.34
perl Makefile.PL
make && make install

That’s more or less what cpan is doing, except it will try to sort out requirements and dependencies for you (when it can).

So I hope you’ve learned a bit of something about getting Perl modules installed and running. If you have any questions, feel free to leave a comment. :)

Recently I’ve seen quite a large trend in customers that use alternative SSH ports. I like the idea behind this but as with most things, I don’t consider it a cure all.

Essentially, for those not in the know, when you have a public facing SSH Daemon on the standard port 22, you can just expect brute force attempts. It’s a fact of life. As is people using common usernames and common passwords. We have one issue, simply because we have the other.

Now, I’m fine with moving SSH to a different port, this avoids just about all types of standard SSH brute force scanning (with the exception of someone trolling the ports on your server and doing individual banner checks). My gripe is that people use non-priveleged ports for their SSH daemons. This, my friends, is an issue.

Linux has what’s known as “privileged ports”. These are ports from 0-1024 and what distinguishes these ports from the other 64512 ports you can bind to is that the linux kernel simply won’t let you open a socket on one of them unless you’re root. That’s it, nothing else. Any user can bind a socket on a port between 1025->64512 without an issue, but in that 0-1024 range, the kernel says NO.

Why? Well, security. With 1024 ports only bindable by the user root, we create a bit of a ‘haven’ for public facing services.

I’m a huge fan of analogies, so here’s one to try to illustrate the importance of privileged ports and more or less what they signify.

Let’s say your server is a Concert and the vendors, ports. At the concert, everyone wants to get their schwag, the T-shirts, the CDs, the posters, all the good stuff. Now the *quality* merchandise is inside the gates. The band can say “This stuff is quality and we stand behind it.” These shirts aren’t going to fall apart in three washes, posters aren’t going to be misprints, and the CDs not labeled with a sharpie. The vendors you buy this stuff from are your privilged ports. You know it’s authentic, good stuff.

Now, you go out to your car and pass the hippies selling hand made t-shirts out of a bag, posters printed at kinkos, and burned rips of the bands CD. It’s just not as good as the stuff the band stands behind, but anyone can open up shop out there in the parking lot. These, are your unprivileged ports.

So with basic services being under port 1024, you can gaurantee the authenticity of these services. That’s essentially it.

When server admins start binding SSH to port 9999, this raises an interesting security risk. Any standard user on the machine in question is capable of starting an app and binding to that port. The tricky part here, is getting the existing SSHd to drop it’s tie to the port so the users modified SSHd can attach to it. There are various methods of doing this, however I think the easiest way would be to symply synflood the port until the server operator triggers a restart. Given that your average hosting machine has a control panel that *will* still respond even if there’s a synflood on the SSHd port, it’s a easy enough to click a few buttons to restart SSH in order to see if that fixes the issue. Quite a few administrators have auto-restarts tied into their monitoring as well, so the server operator may not even know there was an issue with the daemon.

When that restart occurs, the trojaned SSHd is ready for it and grabs ahold of the port. Now, the fun starts. Basically we just have to answer with a standard banner and then force a downgrade of the SSH protocol to v1. After that, all of the authentication can be decrypted quite easily.

Now typically, Man in the Middle attacks through SSH are based around a separate node on the network ARP flooding with false information hoping to redirect an SSH session *through* the server with some form of hacked SSHd. They’ll then typically act in a passthrough state after downgrading the SSH connection to v1 in order to appear fairly seamless to the end user. In the above example, a hosting machine, a standard user account and a non-priv’d SSHd, you’re not going to be able to do successful passthrough as the trojaned daemon is going to be running as a non-priv’d user. You might be able to trick a remote user by throwing them into a sandbox, but that probably wont’ last too long. However they *will* have your password which can be retransmitted pretty quickly.

I’ll see if I can get a test environment setup for to be able to replicate this sort of behavior, but in the meantime there’s no shortage of information on implementing a MiTM attack via SSH on google. :) Stay tuned for part 2 where we’ll implement this in a test lab.

In closing, you should know what a privileged port is and you should now know to QUIT RUNNING SSHD ON NON PRIVILEGED PORTS. :)

Initially, as you may have noticed, this site was originally myself with several other admins trying to make a buck on what we do best. Just when we’d broken the profitability line, an awesome opportunity to work with one of the big names in hosting automation software popped up.

It wasn’t an easy choice by any stretch. Continue with fairly small margins on building my own business better and stronger? Take the opportunity to learn even more about hosting automation and work with some of the smartest brains in the industry.

I took B.

I learned a lot, running my own operation for those months. There’s no substitute for hard work, never turn down a job (no matter how small), and most importantly, never ever get comfortable.

So, that being said, I’d like to do something with this domain to help give back to the community a bit. In my day to day job, I run into some pretty crazy, one off issues that simply don’t have an answer on the interwebs. If I can document a few of these a week, in a proper, well worded and commented way, then I think that’s the biggest way that I can give back.

So with that being said, welcome to the new serveradmins.NET. :) There’s currently no services offered, or for sale here anymore, but hopefully I can help you along your way nonetheless. :)

-Chris

Here at serveradmins.NET, we’ve handled quite a few different tasks for customers in the past. Several of these have been automating day to day tasks in large scale hosting operations. The thought behind this is that if you can save a technician even two minutes on a task they day 60 times a day, you’ve just freed up two hours of their day to handle other things. Now if you apply that to an entire tech operation of 20+ employees, you start to see the advantages quite quickly. Two hours per day per employee multiplied by 20 employees is 40 hours of “free” time you just created. That’s a whole work week right there! Chances are that no matter how slick your operations run, there’s always an opportunity to do *something* better and this is where the experienced admin can step in.

This is the difference between an Admin and a true Senior Admin. A Senior admin has been in the industry long enough to see a good way to do things a bad way to do things and sometimes simply a better way to do things. A true Sr. Admin will be able to look at your operations from the top down, break down the individual components and analyze each one for weaknesses, make and prioritize a list and then act on it.

For example, at a prior clients site, we were brought in to streamline overall operations and “fix things”. We initially started off by looking at the public facing problems and digging down from there. After a bit of recon, we noticed that server restoration times were abysmal, server load averages were way too high across the board and a VERY high failure rate of machines. Way above the norm. This not only caused the obvious direct impact of un-happy customers and complaints, but quite a few side effects as well. Support technicians spent an inordinate amount of their time keeping customers happy. Admins spent way too much time watching server restorations. Billing had an insane spike in chargebacks and cancellations. Unaffected customers got caught up in the flood of support/billing requests and had their problem resolution time skyrocket. Loads were greatly increased on the backup servers on average, which meant normal backup operations for the non-broken machines went over into business hours which caused higher loads via i/o wait on the non-broken machines, etc.

We crafted our plan of attack by looking at the most frequent cause of full server crashes. In this case it was that there was no monitoring on any of the disk arrays in the hosting machines. One drive would fail and the machine would keep operating and then 6-7 months later, the next disk would go causing a full crash of the server and loss of all data. We audited the hardware, did a large scale sweep for broken arrays and array status and found quite a few alarming issues. At least 9 machines with a single failed disk in the array, 3 machines operating on raid 0 raid arrays and several machines with no raid and ailing hardisks! These were all catastrophes just waiting to happen so that’s where we started. Disks were replaced, NRPE RAID checking was put into place so we could be informed of drive failures and act on them immediately. One fix for quite a few problems.

Where I’m going with that example is that you should always be aware of not only the obvious effects a problem manifests, but all of the other problems that stem out from there. After that single fix, we moved onto server load and capacity guidelines, then onto properly defining what an ‘abusive’ customer was and putting a stop to that, etc.

These were all aspects of the tech side of things. For an operations perspective, let’s look at another problem we tackled. We noticed during the initial recon that there were thousands of suspended accounts on the machines. After asking around a bit, we discovered that old account removal was done by hand by an outsourced support staff that was paid monthly for every server they supported. These accounts had been building up for years, eating up backup server space, live production server space and preventing the re-use of pre-existing machines. With a few small shell scripts, we were able to fully remove cancelled, non-pay etc customers on a nightly basis without any manual intervention. A *very* simple fix that freed up a tech from “pruning” the servers only when they ran out of disk space, allowed thousands of new accounts to be placed on existing hardware dramatically cutting new hardware costs for several months and lowering new equipment costs for the entire timeline of the company. Small fix, big savings from a money standpoint.

Domain registration was another issue at this company. A high volume shared hosting operation was registering every domain by hand, one at a time. Sometimes signups exceeded 7-800 a day! This required several techs to do nothing but sit there and register domains all day which if you ask me is a pretty thankless job. With a simple modification to their Ubersmith instance, that was fully automated and 2-3 techs were freed up to alleviate workloads from the main support and phone support systems.

When you handle anything on a large scale, the best fixes are usually simple fixes. As I mentioned at the opening of this blog article, if you save a tech 2 hours a day and then scale that across a week, then a month then a year, you can start to see the cost savings this will net you.

We’re in an industry powered by incredibly intelligent people trying to show their skillset. All too many times I’ve seen an admin spend hours and hours of his day crafting a super complex fix to a simple problem as opposed to taking the quick fix and moving on. Sometimes an issue *does* require a very well crafted complex solution, but in my time doing this 95% of the issues that assault a company are very simple workflow or operations problems that tend to compound and build on eachother.

I hope you can take something from this and apply it to your operations. Remember an hour a day adds up pretty quickly. :)