## 48.6. Beware of Sluggish Performance

Contrary to popular myth, systems don't just start to fail for no reason. If your system is starting to perform poorly, chances are it's because of something that's been initiated. In most cases, the cause has innocuous roots, such as a poorly designed script; however, sluggish performance could also mean an external attack. Regardless of the origin of the decreasing efficiency, you'll want to take steps to locate the problem and remove it before it takes your system down.

If you notice that your systems performance is degrading, there are several built-in utilities you can use to troubleshoot possible problems. Probably the most commonly used utility is ps (Section 24.5); however, there are other utilities that can provide useful information.

### 48.6.1. Check Processes

The first check to perform if you think that you have a destructive agent running on your machine is the processes currently in operation. You'll use the basic ps command to do this, after first checking to make sure that ps itself hasn't been replaced by a bogus program (check installation date, location, and size to see if the ps utility has been replaced).

Running the ps command with the flags -aux shows each user's processes, the CPU and memory usage, time started and command. Here's an example of output:

>     > ps -aux

>     root    6910  0.0  0.1  2088  516  ??  IsJ 30Apr02  1:04.80 /usr/sbin/sshd
>     root    6955  0.0  0.0  2600  384  ??  IsJ 30Apr02  0:06.67 /usr/local/sbin/xinetd -pidfile
>     /var/run/xinetd.pid
>     root    6970  0.0  0.0   624    0 #C1- IWJ -        0:00.00 /bin/sh /usr/virtual/share/
>     pkgs/installed/mysql-server/3.22.32/bin/
>     mysql   6994  0.0  0.0 11216  144 #C1- SJ  30Apr02  0:35.83 /usr/local/libexec/
>     mysqld --basedir=/usr/local --datadir=/var/db/my
>     root    7003  0.0  0.3 10028 2616  ??  SsJ 30Apr02  3:33.55 /usr/local/www/bin/httpd -DSSL
>     nobody 38060  0.0  0.3 10324 3116  ??  SJ  12:01PM  0:08.60 /usr/local/www/bin/httpd -DSSL
>     nobody 38061  0.0  0.3 10332 2612  ??  SJ  12:01PM  0:08.23 /usr/local/www/bin/httpd -DSSL
>     nobody 38062  0.0  0.3 11212 2656  ??  SJ  12:01PM  0:08.89 /usr/local/www/bin/httpd -DSSL
>     nobody 38117  0.0  0.2 10352 2580  ??  SJ  12:01PM  0:09.37 /usr/local/www/bin/httpd -DSSL
>     nobody 38314  0.0  0.2 10332 2596  ??  SJ  12:03PM  0:08.98 /usr/local/www/bin/httpd -DSSL
>     root   62104  0.0  0.0  2112  400  ??  SJ   9:57AM  0:00.16 sshd: shelleyp@ttyp2 (sshd)

In this listing, several processes are being run by root, but all are normal processes and accounted for. In addition, several processes are being run by "nobody," which is the generic user used with HTTP web page access. Using additional ps flags displays additional information, including -e for environment and -f for command-line and environment information of swapped-out processes.

### 48.6.2. Checking Swap Space

If your system is under DoS attack, your swap space is a vulnerable point. This hard disk space is reserved for use by the operating system and to provide space for temporary files. If your system is sluggish and you suspect a possible DoS attack -- or just a badly behaving script that results in a lot of temporary files -- the first thing you should check is how much swap space you have.

The pstat utility can be used to check swap space when using the -s option on the command line:

>     pstat -s

The result will be a listing of swap areas by device with available and used swap space. If the percentage of used space is much higher than normal, you probably have a bad script or external interference. Additional utilities can help you determine which.

Within FreeBSD and other Unix systems, swapinfo returns the same information as pstat -s. If you're running a Mac OS X system, instead of pstat, you'll use the ls command and check the contents of /var/vm:

>     ls -l /var/vm
>     -rw-------T   1   root      wheel     000000000  Jun    4    12:56    swapfile0

Since the system wasn't under load, the swap space didn't have any contents at the time this command was run.

### 48.6.3. Check Network Connections

Another check you can run if your system is running sluggishly and you think you might be under attack is netstat. This command will return activity on Unix sockets as well as all of the active Internet connections, including referrals if the connection occurs through HTTP.

Here's an example of netstat output:

>     Active Internet connections
>     Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
>     tcp4       0      0  burningbird.http       a25253.upc-a.che.3617  TIME_WAIT
>     tcp4       0      0  burningbird.http       pm66.internetsee.4301  TIME_WAIT
>     tcp4       0      0  burningbird.http       strider.ccs.neu..4492  TIME_WAIT
>     tcp4       0      0  burningbird.http       strider.ccs.neu..4491  TIME_WAIT
>     tcp4       0      0  burningbird.http       strider.ccs.neu..4490  TIME_WAIT
>     tcp4       0      0  burningbird.http       mailgate.ltsbfou.57600 FIN_WAIT_2
>     tcp4       0      0  burningbird.http       mailgate.ltsbfou.57595 FIN_WAIT_2
>     tcp4       0     20  burningbird.ssh        adsl-64-168-24-1.1076  ESTABLISHED
>     tcp4       0      0  burningbird.submission *.*                    LISTEN
>     tcp4       0      0  burningbird.smtp       *.*                    LISTEN
>     tcp4       0      0  burningbird.domain     *.*                    LISTEN
>     tcp4       0      0  burningbird.http       *.*                    LISTEN
>     tcp4       0      0  burningbird.https      *.*                    LISTEN
>     tcp4       0      0  burningbird.pop3s      *.*                    LISTEN
>     tcp4       0      0  burningbird.ssh        *.*                    LISTEN
>     udp4       0      0  burningbird.domain     *.*
>     udp4       0      0  burningbird.syslog     *.*
>     Active UNIX domain sockets
>     Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
>     e5ed4cc0 stream      0      0 e5f0cbc0        0        0        0 /tmp/mysql.sock
>     e5ed4d40 stream      0      0        0        0        0        0
>     e5e08380 dgram       0      0        0 e5ed4dc0        0 e5e083c0
>     e5e083c0 dgram       0      0        0 e5ed4dc0        0 e5ed4d80
>     e5ed4d80 dgram       0      0        0 e5ed4dc0        0        0
>     e5ed4dc0 dgram       0      0 e556c040        0 e5e08380        0 /var/run/log

Specifying netstat with the command line option -s provides a detailed report of per-protocol -- TCP, UDP, IP, and so on -- usage statistics.

The netstat program is helpful not only for determining if someone is trying to break into your system, but also for determining if your system is having basic communication problems.

### 48.6.4. Other Checks

You can use iostat to check I/O statistics on your various devices. For instance, to check to see what kind of activity is occurring on all devices every three seconds for nine runs, issue the following command:

>     # iostat -odICTw 2 -c 9
>           tty        mlxd0          acd0           fd0           md0             cpu
>      tin tout blk xfr msps  blk xfr msps  blk xfr msps  blk xfr msps  us ni sy in id
>        0    0   0   0  0.0    0   0  0.0    0   0  0.0    0   0  0.0   0  0  0  0  0
>        0    0 224  12  167    0   0  0.0    0   0  0.0    0   0  0.0   0  0  0  0  0
>        0    0 568  36 55.8    0   0  0.0    0   0  0.0    0   0  0.0   0  0  0  0  0
>        0    0 144   5  402    0   0  0.0    0   0  0.0    0   0  0.0   0  0  0  0  0
>        0    0 112   7  287    0   0  0.0    0   0  0.0    0   0  0.0   0  0  0  0  0
>        0    0  48   3  670    0   0  0.0    0   0  0.0    0   0  0.0   0  0  0  0  0
>        0    0 240  15  134    0   0  0.0    0   0  0.0    0   0  0.0   0  0  0  0  0
>        0    0 192  12  168    0   0  0.0    0   0  0.0    0   0  0.0   0  0  0  0  0
>        0    0  96   6  335    0   0  0.0    0   0  0.0    0   0  0.0   0  0  0  0  0

The result allows you to compare I/O over a period of time. Note that in some systems, iostat may be io_stat, instead.

Another check is vmstat (vm_stat), which displays the virtual memory statistics for a machine. As with iostat, running the command several times over a period of time can show if there is unusual activity within virtual memory. For instance, if the free memory unexpectedly decreases, no known user is running a process, the occurrence of the free memory use is consistent (occurring at a set time of day), and no other system processes or cron jobs are known to be running, you probably have an intruding application running somewhere on the system. Other tests can then be used to help you determine what the application is.

-- SP

Copyright © 2003 O'Reilly & Associates. All rights reserved.