HowTo Read vmstat / iostat and netstat output

vmstat

vmstat, as its name suggests, reports virtual memory statistics. It shows how much virtual memory there is, how much is free and paging activity. Most important, you can observe page-ins and page-outs as they happen. This is extremely useful.

To monitor the virtual memory activity on your system, it's best to use vmstat with a delay. A delay is the number of seconds between updates. If you don't supply a delay, vmstat reports the averages since the last boot and quit. Five seconds is the recommended delay interval.

To run vmstat with a five-second delay, type:
root@DCOS-71:~# vmstat 5


You also can specify a count, which indicates how many updates you want to see before vmstat quits. If you don't specify a count, the count defaults to infinity, but you can stop output with Ctrl-C.

To run vmstat with ten updates, five seconds apart, type:
root@DCOS-71:~# vmstat 5 10

root@DCOS-71:~# vmstat

procs -----------memory---------- --------swap-- -----io---- -system-- ----cpu----

 r   b   swpd   free          buff        cache       si   so     bi    bo       in   cs      us sy id wa

 1  0      0     13072876 112188 2778156       0    0     1     1         7    9        0  0 100  0

Proc
---
r: Processes actually running, waiting for some attention from the CPU
b: Uninterruptble sleeping processes (This I am yet to discover what does it mean)

Memory:
---
swpd: Virtual memory usage (swap areas are listed in /proc/swaps)
free: Idle memory
buff: Memory used as buffers, like before/after IO operations, I guess
cache: Memory used as cache.

Swap:
---
si: Memory swapped in from the disk
so: Memory swapped to the disk

IO:
---
bi: Blocks received from block device (like a hard disk)
bo: Blocks sent to a block device

System:
---
in: The number of interrupts per second, including the clock.
cs: The number of context switches per second.

CPU:
---
us: Time spent running non-kernel code. (user time, including nice time)
sy: Time spent running kernel code. (system time - network, IO interrupts, etc)
id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
wa: Time spent waiting for IO. 

iostat

The iostat command is used to monitor the load on server input/output (I/O) devices by observing the time the devices are active compared to the average transfer rate of the device. 

When your CPU seems to be spending a lot of time on iowait you need to make some changes. However an iowait can occur either because there is a lot of Disk/Network IO taking place, or because the disk subsystem is saturated and cannot provide greater throughput. iostat allows you to determine which one it is. A regular iostat output consists of the following fields -

root@DCOS-71:~# iostat -dx 5
Linux 2.6.31.9-scst (DCOS-71) 05/14/2012 _x86_64_ (4 CPU)

Device: rrqm/s  wrqm/s r/s w/s   rsec/s   wsec/s   avgrq-sz   avgqu-sz   await   svctm %util

sda      0.00     0.27    0.02 0.10 4.60      2.96      60.82        0.00         7.18    4.91    0.06

sdb      0.00     0.00    0.00 0.00 0.00       0.00      21.54       0.00         4.77    2.69   0.00

rrqm/s and wrqm/s

The number of merged read and write requests queued per second. “Merged” means the operating system took multiple logical requests and grouped them into a single request to the actual device.

r/s and w/s

The number of read and write requests sent to the device per second.

rsec/s and wsec/s

The number of sectors read and written per second. Some systems also output rkB/s and wkB/s, the number of kilobytes read and written per second. We omit those for brevity.

avgrq-sz

The request size in sectors.

avgqu-sz

The number of requests waiting in the device’s queue.

await

The number of milliseconds required to respond to requests, including queue time and service time. Unfortunately, iostat doesn’t show separate service time statistics for read and write requests, which are so different that they really shouldn’t be averaged together. However, you can probably chalk up high I/O waits to reads, because writes can often be buffered but reads usually have to be served directly from the spindles.

svctm

The number of milliseconds spent servicing requests, from beginning to end, including queue time and the time the device actually takes to fulfill the request.

%util

The percentage of CPU time during which requests were issued. This really shows the device utilization, as the name implies, because when the value approaches 100%, the device is saturated.

You can use the output to deduce some facts about a machine’s I/O subsystem. One important metric is the number of requests served concurrently. Because the reads and writes are per second and the service time’s unit is thousandths of a second, the dimensions in the following formula cancel out to show the number of concurrent requests the device is serving:*

concurrency = (r/s + w/s) * (svctm/1000)

netstat

netstat is a built-in, hidden utility that you can use to view and troubleshoot your network connection. It is a most useful and very versatile for finding connection to and from the host.

A typical result from netstat -an looks like this: (this is a slightly edited result of my (online) machine)

root@DCOS-71:~# netstat -an

Active Internet connections (servers and established)

Proto   Recv-Q Send-Q Local Address           Foreign Address                   State

tcp        0            0         0.0.0.0:46587           0.0.0.0:*                             LISTEN

tcp        0            0         0.0.0.0:2049             0.0.0.0:*                             LISTEN

tcp        0            0         10 .10.1.2:38728     202.148.200.136:80         ESTABLISHED

tcp        0            0         10.10.1.2:52641        74.125.236.206:80         ESTABLISHED

tcp        0            0         10.10.1.2:38625      202.148.200.136:80         TIME_WAIT

tcp        0            1         10.10.1.2:47811      192.168.1.254:8118         SYN_SENT

• In lines saying 'ESTABLISHED', you need the remote port to identify what has connected to the remote site.
• In lines saying 'LISTENING', you need the local port to identify what is listening there.
• Each outbound TCP connection also causes a LISTENING entry on the same port.
• Most UDP listening ports are duplicates from a listening TCP port. Ignore them unless they don't have a TCP twin.
• TIME_WAIT entries are not important.
• If it says 0.0.0.0 on the Local Address column, it means that port is listening on all 'network interfaces' (i.e. your computer, your modem(s) and your network card(s)).
• If it says 127.0.0.1 on the Local Address column, it means that port is ONLY listening for connections from your PC itself, not from the Internet or network. No danger there.
• If it displays your online IP on the Local Address column, it means that port is ONLY listening for connections from the Internet.
• If it displays your local network IP on the Local Address column, it means that port is ONLY listening for connections from the local network.
• In lines saying "SYN_SENT" means that your computer has sent a packet that, in a nutshell, is asking to create a connection.If you're seeing that as a state in netstat over a period of time, the general assumption is that the receiving computer is ignoring that packet, and will not make the connection.

Note that this document comes without warranty of any kind. But every effort has been made to provide the information as accurate as possible. I welcome emails from any readers with comments, suggestions, and corrections at webmaster_at admin@linuxhowto.in

Copyright © 2012 LINUXHOWTO.IN