An easy way to see the existing runlevel is with the ‘who -r’ command.


root@server:~ # who -r
run-level 3 May 22 03:54 last=S


Here we show the last boot and current runlevel (3). In order to modify this, you can open up /etc/inittab and look for the following line.


id:3:initdefault:


This shows that runlevel 3 is the default one kicked off by init on a bootup.

For what it’s worth, this refers to the directories in /etc/rc.d/rcX.d/, where X is #. The number correlates to the runlevel of your machine, and all of the scripts stored in that directory are executed when that runlevel is enabled.

TMYK!

Fuser.

Why?

fuser is awesome. Not awesome in a “run it and it fixes everything” way, but awesome in a “What in the hell is binding to this port??” kind of way. Two classic scenarios where this is handy…

1. Apache won’t start, “Can’t bind to port ::80″ or “Can’t bind to port ::443″, etc. This typically means something else is already tied to that port, and won’t relinquish it…

2. A security scan of your machine shows something funny running on port 6667… You didn’t start this or know what it is.

What to do now? Well you can sift through netstat output, but that’s, well, boring and slightly annoying.

netstat output

[root@vps ~]# netstat -anp |grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3266/httpd
tcp 0 0 127.0.0.1:58725 127.0.0.1:80 TIME_WAIT -
tcp 0 0 10.10.10.10:2078 192.168.1.23:63024 ESTABLISHED 18088/cpdavd - acce
unix 3 [ ] STREAM CONNECTED 49222880 11574/dovecot-auth /var/run/dovecot/login/default
unix 2 [ ] DGRAM 6804658 14078/named


Okay now we see that 3266/httpd is running on 80. Then we do this to find the process..


[root@vps ~]# ps auxwww |grep 3266
nobody 3266 0.0 0.3 65704 3516 ? S 17:05 0:00 /usr/local/apache/bin/httpd -k start -DSSL
root 21665 0.0 0.0 6024 640 pts/2 S+ 18:51 0:00 grep 3266
[root@vps ~]#


Okay there we go…

Now for hacked systems, this could (and probably) is fully forged for a lot of remote shells. Going back to my previous post at http://serveradmins.net/ssh-on-nonstandard-ports-how-to-not-do-it/ which talks about priveleged ports, you could in theory have trojaned ‘ps’, top, etc masking that real process. It may *look* like httpd, but bound to a port like 23425… So dont’t trust that too much, but a bit on that in a second.

The fuser approach…


[root@vps ~]# fuser -n tcp 80
80/tcp: 3266 3267 3268 3269 3271 16078 18274
[root@vps ~]#


Oh look at that a list of all pids bound to that port. Nice, clean, to the point and easily parsable. fuser rocks.

Now a bit more about the masked processes… To run those down, here’s a quick tip. Forget ps/top and your other normal utilities, /proc/ is your friend here…

Proc looks like this on a linux box…

[root@vps ~]# cd /proc/
[root@vps proc]# ls -al
total 1
dr-xr-xr-x 78 root root 0 Jan 26 09:58 .
drwxr-xr-x 24 chrismm chrismm 1024 Feb 4 22:06 ..
dr-xr-xr-x 4 root root 0 Feb 5 05:05 1
dr-xr-xr-x 4 root root 0 Feb 5 05:05 11573
dr-xr-xr-x 4 root root 0 Feb 5 05:05 11574
dr-xr-xr-x 4 dovecot dovecot 0 Feb 5 05:05 11575
dr-xr-xr-x 4 dovecot dovecot 0 Feb 5 05:05 11576
...
...


These directories match the pids of the running process… So if you have something advertising itself as ‘httpd’ on port 234234 and you know it’s pid 3266, you’d just do the following…


[root@vps proc]# cd /proc/3266
[root@vps 3266]# ls -al
total 0
dr-xr-xr-x 4 nobody nobody 0 Feb 5 17:08 .
dr-xr-xr-x 78 root root 0 Jan 26 09:58 ..
-r-------- 1 root root 0 Feb 5 18:56 auxv
-r--r--r-- 1 root root 0 Feb 5 17:08 cmdline
-rw-r--r-- 1 root root 0 Feb 5 18:56 coredump_filter
-r--r--r-- 1 root root 0 Feb 5 18:56 cpuset
lrwxrwxrwx 1 root root 0 Feb 5 18:54 cwd -> /
-r-------- 1 root root 0 Feb 5 18:56 environ
lrwxrwxrwx 1 root root 0 Feb 5 17:10 exe -> /usr/local/apache/bin/httpd
dr-x------ 2 root root 0 Feb 5 18:49 fd
?r--r--r-- 1 root root 0 Feb 5 18:56 io
-r-------- 1 root root 0 Feb 5 18:56 limits
-rw-r--r-- 1 root root 0 Feb 5 18:56 loginuid
-r--r--r-- 1 root root 0 Feb 5 18:54 maps
-rw------- 1 root root 0 Feb 5 18:56 mem
-r--r--r-- 1 root root 0 Feb 5 18:56 mounts
-r-------- 1 root root 0 Feb 5 18:56 mountstats
-r--r--r-- 1 root root 0 Feb 5 18:56 numa_maps
-rw-r--r-- 1 root root 0 Feb 5 18:56 oom_adj
-r--r--r-- 1 root root 0 Feb 5 18:56 oom_score
lrwxrwxrwx 1 root root 0 Feb 5 18:54 root -> /
-r--r--r-- 1 root root 0 Feb 5 18:56 schedstat
-r-------- 1 root root 0 Feb 5 18:56 smaps
-r--r--r-- 1 root root 0 Feb 5 17:08 stat
-r--r--r-- 1 root root 0 Feb 5 17:10 statm
-r--r--r-- 1 root root 0 Feb 5 17:08 status
dr-xr-xr-x 3 nobody nobody 0 Feb 5 18:56 task
-r--r--r-- 1 root root 0 Feb 5 18:56 wchan
[root@vps 3266]#


Bam, there you go. cwd and exe are the things you’re looking for It shows you the dir it was spawned from (typically a users homedirectory on a shared hosting machine) and the full path/binary actually being executed (usually lame perl/php listeners)… Also the ./fd/ folder is kind of neat as it shows you all the open file handles tied up by that pid as well.

Anyway, /proc/ examination too, is for another day, I just wanted to ramble on about one of my favorite, neat little single use utilities that no one else seems to know about. fuser. Enjoy. =)