Skip to content

ServerAdmins.NET

Stuff for Server Admins…

Archive

Category: Miscellaneous

Recently I’ve seen quite a large trend in customers that use alternative SSH ports. I like the idea behind this but as with most things, I don’t consider it a cure all.

Essentially, for those not in the know, when you have a public facing SSH Daemon on the standard port 22, you can just expect brute force attempts. It’s a fact of life. As is people using common usernames and common passwords. We have one issue, simply because we have the other.

Now, I’m fine with moving SSH to a different port, this avoids just about all types of standard SSH brute force scanning (with the exception of someone trolling the ports on your server and doing individual banner checks). My gripe is that people use non-priveleged ports for their SSH daemons. This, my friends, is an issue.

Linux has what’s known as “privileged ports”. These are ports from 0-1024 and what distinguishes these ports from the other 64512 ports you can bind to is that the linux kernel simply won’t let you open a socket on one of them unless you’re root. That’s it, nothing else. Any user can bind a socket on a port between 1025->64512 without an issue, but in that 0-1024 range, the kernel says NO.

Why? Well, security. With 1024 ports only bindable by the user root, we create a bit of a ‘haven’ for public facing services.

I’m a huge fan of analogies, so here’s one to try to illustrate the importance of privileged ports and more or less what they signify.

Let’s say your server is a Concert and the vendors, ports. At the concert, everyone wants to get their schwag, the T-shirts, the CDs, the posters, all the good stuff. Now the *quality* merchandise is inside the gates. The band can say “This stuff is quality and we stand behind it.” These shirts aren’t going to fall apart in three washes, posters aren’t going to be misprints, and the CDs not labeled with a sharpie. The vendors you buy this stuff from are your privilged ports. You know it’s authentic, good stuff.

Now, you go out to your car and pass the hippies selling hand made t-shirts out of a bag, posters printed at kinkos, and burned rips of the bands CD. It’s just not as good as the stuff the band stands behind, but anyone can open up shop out there in the parking lot. These, are your unprivileged ports.

So with basic services being under port 1024, you can gaurantee the authenticity of these services. That’s essentially it.

When server admins start binding SSH to port 9999, this raises an interesting security risk. Any standard user on the machine in question is capable of starting an app and binding to that port. The tricky part here, is getting the existing SSHd to drop it’s tie to the port so the users modified SSHd can attach to it. There are various methods of doing this, however I think the easiest way would be to symply synflood the port until the server operator triggers a restart. Given that your average hosting machine has a control panel that *will* still respond even if there’s a synflood on the SSHd port, it’s a easy enough to click a few buttons to restart SSH in order to see if that fixes the issue. Quite a few administrators have auto-restarts tied into their monitoring as well, so the server operator may not even know there was an issue with the daemon.

When that restart occurs, the trojaned SSHd is ready for it and grabs ahold of the port. Now, the fun starts. Basically we just have to answer with a standard banner and then force a downgrade of the SSH protocol to v1. After that, all of the authentication can be decrypted quite easily.

Now typically, Man in the Middle attacks through SSH are based around a separate node on the network ARP flooding with false information hoping to redirect an SSH session *through* the server with some form of hacked SSHd. They’ll then typically act in a passthrough state after downgrading the SSH connection to v1 in order to appear fairly seamless to the end user. In the above example, a hosting machine, a standard user account and a non-priv’d SSHd, you’re not going to be able to do successful passthrough as the trojaned daemon is going to be running as a non-priv’d user. You might be able to trick a remote user by throwing them into a sandbox, but that probably wont’ last too long. However they *will* have your password which can be retransmitted pretty quickly.

I’ll see if I can get a test environment setup for to be able to replicate this sort of behavior, but in the meantime there’s no shortage of information on implementing a MiTM attack via SSH on google. :) Stay tuned for part 2 where we’ll implement this in a test lab.

In closing, you should know what a privileged port is and you should now know to QUIT RUNNING SSHD ON NON PRIVILEGED PORTS. :)

Initially, as you may have noticed, this site was originally myself with several other admins trying to make a buck on what we do best. Just when we’d broken the profitability line, an awesome opportunity to work with one of the big names in hosting automation software popped up.

It wasn’t an easy choice by any stretch. Continue with fairly small margins on building my own business better and stronger? Take the opportunity to learn even more about hosting automation and work with some of the smartest brains in the industry.

I took B.

I learned a lot, running my own operation for those months. There’s no substitute for hard work, never turn down a job (no matter how small), and most importantly, never ever get comfortable.

So, that being said, I’d like to do something with this domain to help give back to the community a bit. In my day to day job, I run into some pretty crazy, one off issues that simply don’t have an answer on the interwebs. If I can document a few of these a week, in a proper, well worded and commented way, then I think that’s the biggest way that I can give back.

So with that being said, welcome to the new serveradmins.NET. :) There’s currently no services offered, or for sale here anymore, but hopefully I can help you along your way nonetheless. :)

-Chris

In our first installation in this blog, we went through a basic XCache installation with CentOS, Apache and PHP 5. Now, we’re going to look into how to make it work for your personal installation.

Much like every other tutorial out there, if you explicitly follow the instructions you find on the internet, you’re usually going to run into problems. One of the biggest problems I see in systems configurations, is getting someone that “tunes” the machine by copying and pasting configs straight from a forum or blog. Sure, it works, but that doesn’t mean it works *good*. It could also cause many more problems that it solves.

The best way to tune a machine will always be understanding what you’re doing. And that’s what we’re going to try to do here today with XCache. So, let’s take a look at that xcache.ini file we setup in the last post.


[xcache-common]

zend_extension = /usr/lib/php/modules/xcache.so

[xcache.admin]
xcache.admin.enable_auth = On
xcache.admin.user = "admin"
; xcache.admin.pass = md5(a5851d1f3a3cff5a42b3163a7c45e5ae)
xcache.admin.pass = "blahblahblahblahblah"

[xcache]
; ini only settings, all the values here is default unless explained

; select low level shm/allocator scheme implemenation
xcache.shm_scheme = "mmap"
; to disable: xcache.size=0
; to enable : xcache.size=64M etc (any size > 0) and your system mmap allows
xcache.size = 32M
; set to cpu count (cat /proc/cpuinfo |grep -c processor)
xcache.count = 2
; just a hash hints, you can always store count(items) > slots
xcache.slots = 8K
; ttl of the cache item, 0=forever
xcache.ttl = 0
; interval of gc scanning expired items, 0=no scan, other values is in seconds
xcache.gc_interval = 0

; same as aboves but for variable cache
xcache.var_size = 0M
xcache.var_count = 1
xcache.var_slots = 8K
; default ttl
xcache.var_ttl = 0
xcache.var_maxttl = 0
xcache.var_gc_interval = 300

xcache.test = Off
; N/A for /dev/zero
xcache.readonly_protection = Off
; for *nix, xcache.mmap_path is a file path, not directory.
; Use something like "/tmp/xcache" if you want to turn on ReadonlyProtection
; 2 group of php won't share the same /tmp/xcache
; for win32, xcache.mmap_path=anonymous map name, not file path
xcache.mmap_path = "/tmp/xcache"

; leave it blank(disabled) or "/tmp/phpcore/"
; make sure it's writable by php (without checking open_basedir)
xcache.coredump_directory = ""

; per request settings
xcache.cacher = On
xcache.stat = On
xcache.optimizer = Off

[xcache.coverager]
; per request settings
; enable coverage data collecting for xcache.coveragedump_directory and xcache_coverager_start/stop/get/clean() functions (will hurt executing performance)
xcache.coverager = Off

; ini only settings
; make sure it's readable (care open_basedir) by coverage viewer script
; requires xcache.coverager=On
xcache.coveragedump_directory = ""

The first thing we’ll look at is the xcache.count variable in the xcache.ini file. To put it simply, think of every one of these caches as a bucket. In these buckets you have marbles. Now, I’m going to hand you 200 marbles, all of different colors and dump them into your buckets, dispersed evenly. After that, you have to find the blue marble in your bucket(s). Now, if you have 8 buckets, each with a helper to look through it, you’re going to find that blue marble FAST. One bucket and one person searching, yeah, grab a snickers, it’ll be awhile. :)

In this analogy, the buckets are the caches, and the marbles are your PHP scripts (after they’ve been compiled and stored in memory). When your webserver gets a request for blah.php, it hands off to your PHP Handler (whatever that may be), which first references the xcache extension before trying to actually compile and serve the script. Your limit on buckets is the number of processors you can have that can execute an instruction at the same time. So, as the xcache.ini says, cat /proc/cpuinfo | grep -c processor will give you how many xcache queues you can have at once. Personally, I always subtract at least one proc in multicore/hyperthreaded setups, simply because there’s always more going on with your machine than processsing PHP/XCache requests. The reality is that you’ll *never* have all of your procs handling that request at the same time. Save some CPU cycles for other stuff. :)

Next up is the xcache.slots variable. This is basically a big hash table, or index of the items stored in your “buckets” from above. To visualize it, think about it as if you had a string tied to each marble in the above scenario, with a tag that says “blue, green, red”. The bigger the hash table, the more strings you have to tie to marbles. You’ll notice quicker seek times for the precompiled opcode by increasing this variable. Each “bucket” can hold a lot of marbles (opcode) and this is just a way of finding those pieces of opcode even faster. The hash tables don’t take up too much memory, so if you throw 32-64k at it, you’ll probably be good. Play around and see. :)

So, that’s how the how and why of your xcache.count and xcache.slots variables. Don’t set it too high, Don’t set it too low, set it just right.

Now, your next most important piece is going to be the Size. This is the total amount of ram you provide in the xcache.size variable. The sole purpose of this variable is to allocate a specific amount of memory strictly for storing pre-compiled PHP scripts. For the most part, these don’t take up much space in memory at all (as you’ll see), so throwing a bit of ram at this is a VERY important item.

If your site is live, and you’re serving out PHP, then you’ve already started collecting the necessary statistics to judge if you’ve allocated enough memory here. The two biggest tells are the “Avail” and “OOMs” columns. If you have 0.00M available in all of your queues, then you’re probably seeing increasing numbers in the OOMs column. What this means is that XCache has stored so many compiled PHP scripts in the chunk of memory that you gave it, that it just ran out of room. When this happens, any PHP requested by the webserver that’s not already in the cache is going to be interpreted normally. No Caching. This is bad. :)

In order to tune this variable correctly, throw a bit more memory than the initial 16M we threw at it in our initial config. Go ahead and double it, I’m sure you can justify the 32M of RAM here. :) Restart apache for the changes to take effect, and watch again. Does it fill up? Any OOMs? If not, just let it go and run for a few hours. You’re looking for the sweet spot to where you always have a bit of Available memory, and not hitting any OOMs, that’s when XCache is working at peak efficiency.

So, let’s say you have a HUGE site. Thousands upon Thousands of PHP scripts, all that need compiled and cached. You have the option of playing the ante up game, or looking down the config just a bit. :) The two next most important options as I see it are the xcache.ttl and the xcache.gc_interval config options.

These main goal of these two variables is to 1. Set an “expiration date” on your compiled code and 2. Clean it out of the memory after it expires. Both of these options are in seconds and disabled by default. In a perfect world, you could fit your entire website, post-compilation into 32-64M of ram and would never have to flush it. However, this isn’t a perfect world. If you’ve thrown all the memory that you can safely dedicate to this that you can, then this is the next options you should tune. The tuning here truly depends on how busy your site is. If you have 1000 PHP scripts, room to store 800 in memory and a busy site with fairly dispersed hits, then you might need to up the TTL to something like 60 seconds and th GC frequency up to every 4 minutes or so.

When an opcode cache is marked as expired, it sits in your queue until the next run of the GC (garbage collector, fwiw). The next time it is called, you’ll need to recompile it again and re-insert it into memory. So as you can see, you don’t want to set the TTL to low here otherwise you’ll see your box more loaded than it should be because it’s re-compiling PHP it shouldn’t be.

Now, the next issue isn’t so much of a performance concern (to a small degree I guess, but it’s negligible, really…) is the xcache.readonly_protection flag. By default this is set to Off. What’s going on here, is involving the mmap caching for the compiled php opcode. If set to Off, Xcache can modify files in memory rather than ditching the full opcode and recompiling if it needs to make a change.

The tradeoff here is security. With this set to On, you’re going to end up recompiling opcode for a slight adjustment, as opposed to making the adjustment. This also means that should there be some sort of hack or compromise that allows an attacker to modify any data in your caches (think someone editing your PHP on the server directly without your permission) that they simply can’t.

These updates don’t happen to often, hence the negligible performance hit. So please, enable this unless you really need to disable it. :)

With this in mind, you have the core essentials to XCache down. There are, of course, variables I didn’t discuss here, and as you see, I didn’t just give you a generic config to copy and paste. I’ve given you the information necessary to tune the heck out of your XCache installation.

Go forth and serve… (precompiled opcode. =P)