From IT Mission Linux Tips, Hacks, Tutorials, Howtos - Itmission.org

Main: Linux-performance-monitoring


Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.


sar - Collect, report, or save system activity information.

Requirements: sysstat package.

Installing sysstat

sudo apt-get install sysstat

yum install sysstat

ubuntu@server3:~$ sudo apt-get install sysstat
[sudo] password for ubuntu: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
sysstat is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 240 not upgraded.
ubuntu@server3:~$ 

Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds

ubuntu@server3:~$ sudo sar -u 2 5

ubuntu@server3:~$ sudo sar -u 2 5
[sudo] password for ubuntu: 
Linux 2.6.38-8-generic (server3) 	10/31/2011 	_x86_64_	(1 CPU)

07:46:06 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:46:08 AM     all      3.52      0.00      3.52      5.03      0.00     87.94
07:46:10 AM     all      2.51      0.00      3.52      0.00      0.00     93.97
07:46:12 AM     all      2.50      0.00      3.50      0.00      0.00     94.00
07:46:14 AM     all      1.50      0.00      2.00      4.50      0.00     92.00
07:46:16 AM     all      1.50      0.00      2.00      0.00      0.00     96.50
Average:        all      2.30      0.00      2.91      1.90      0.00     92.89
ubuntu@server3:~$

The -u option specifies our interest in the CPU subsystem.
The -o option will create an output file that contains binary data. 3 samples at two-second intervals.

ubuntu@server3:~$sudo sar -u -o sardatafile 2 3

buntu@server3:~$ sar -u -f sardatafile 
Linux 2.6.38-8-generic (server3) 	10/31/2011 	_x86_64_	(1 CPU)

07:48:19 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:48:21 AM     all      0.00      0.00      1.50      0.00      0.00     98.50
07:48:23 AM     all      0.50      0.00      1.00      0.00      0.00     98.50
07:48:25 AM     all      1.00      0.00      0.50      7.50      0.00     91.00
Average:        all      0.50      0.00      1.00      2.50      0.00     96.00
ubuntu@server3:~$
The %iowait and %idle columns are of interest to us when doing performance analysis.
The %iowait column specifies the amount of time the %red%CPU spends waiting for I/O requests to complete%%.
The %idle column tells us how much useful work the CPU is doing.
A %idle time near zero indicates a CPU bottleneck.
A high %iowait value indicates unsatisfactory disk performance.

Additional information can be obtained by the sar -q command, which displays the run queue length, 
total number of processes,  and the load averages for the past one, five and fifteen minutes:
If you see more than one process is runnable at any given time, the the system will be busy.

ubuntu@server3:~$sar -q

runq-sz #Run queue length (number of processes waiting for run time).

Check if system has free memory and no swap usage, then memory is not the problem.

ubuntu@server3:~$ sar -r

To get swapping statistics

ubuntu@server3:~$ sar -W

Displays I/O and transfer rate statistics grouped for all block devices

ubuntu@server3:~$sar -b

sar -d it displays statistics data similar to those displayed by iostat:

ubuntu@server3:~$sar -d

Report paging statistics.

ubuntu@server3:~$sar-B ubuntu@server3:~$ sar -B -f /var/log/sysstat/sa09

sadf - Display data collected by sar in multiple formats

Extract memory, swap space and network statistics from system activity file 'sa21', and display them in a format that can be ingested by a database.

sadf -d /var/log/sysstat/sa21 -- -r -n DEV

Example:

ubuntu@server3:~$ sadf -d /var/log/sysstat/sa09 -- -B
server3;600;2011-10-09 09:55:01 UTC;6.29;18.12;43.23;0.10;266.73;0.00;0.00;0.00;0.00
server3;600;2011-10-09 10:05:01 UTC;4.29;4.00;24.69;0.02;209.58;0.00;0.00;0.00;0.00

Extract CPU statistics for processor 1 (the second processor) from current daily data file, and display them in a formatthat can easily be handled by a pattern processing command.

sadf -p -P 1

pidstat - Report statistics for Linux tasks.

The pidstat command is used to monitor processes and threads currently being managed by the Linux kernel. It can also monitor the children of those processes and threads.

With its -d option, pidstat can report I/O statistics

ubuntu@server3:~$ pidstat -d 2
Linux 2.6.38-8-generic (server3)     10/09/2011     _x86_64_    (1 CPU)
05:28:18 PM       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
05:28:20 PM      6082      0.00      4.00      0.00  gnome-terminal
05:28:20 PM      6727      0.00      4.00      0.00  rsyslogd

EXAMPLES

Display five reports of CPU statistics for every active task in the system at two second intervals.

pidstat 2 5

Display five reports of page faults and memory statistics for PID 1643 at two second intervals.

pidstat -r -p 1643 2 5

Display global page faults and memory statistics for all the processes whose command name includes strings.

pidstat -C "fox|bird" -r -p ALL

Display five reports of page faults statistics at two second intervals for the child processes of all tasks in the system.

pidstat -T CHILD -r 2 5

The following example reports CPU statistics for PID 6968 and all its threads:

ubuntu@server3:~$ pidstat -t -p 6968 3 2    

Linux 2.6.38-8-generic (server3)     10/09/2011     _x86_64_    (1 CPU)
05:36:00 PM      TGID       TID    %usr %system  %guest    %CPU   CPU  Command
05:36:03 PM      6968         -    0.33    0.00    0.00    0.33     0  soffice.bin
05:36:03 PM         -      6968    0.00    0.00    0.00    0.00     0  |__soffice.bin
05:36:03 PM         -      6970    0.00    0.00    0.00    0.00     0  |__soffice.bin
05:36:03 PM         -      6972    0.00    0.00    0.00    0.00     0  |__soffice.bin
05:36:03 PM         -      6973    0.00    0.00    0.00    0.00     0  |__soffice.bin
05:36:03 PM         -      6976    0.00    0.00    0.00    0.00     0  |__soffice.bin
05:36:03 PM         -      6983    0.00    0.00    0.00    0.00     0  |__soffice.bin

What is majflt/s ?

Number of major faults the system has made per second, those which have required loading a memory page from disk (post 2.5 kernels only). Page faults are not a bad thing. It's part of the way that the memory system works.

http://en.wikipedia.org/wiki/Page_fault

-r

Report page faults and memory utilization. When reporting statistics for individual tasks, the following values are displayed:

minflt/s Total number of minor faults the task has made per second, those which have not required loading a memory page from disk.

majflt/s Total number of major faults the task has made per second, those which have required loading a memory page from disk.

VSZ Virtual Size: The virtual memory usage of entire task in kilobytes.

RSS Resident Set Size: The non-swapped physical memory used by the task in kilobytes.

ubuntu@server3:~$ sudo pidstat -r 2
[sudo] password for ubuntu:
Linux 2.6.38-8-generic (server3)     10/09/2011     _x86_64_    (1 CPU)

08:31:52 PM       PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
08:31:54 PM      1011      0.50      0.00   48488   3392   0.43  fail2ban-server
08:31:54 PM      1320      0.50      0.00   45168    744   0.09  udisks-daemon
08:31:54 PM      7257    147.76      0.00   15704   1068   0.13  pidstat

08:31:54 PM       PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
08:31:56 PM      1320      0.50      0.00   45168    744   0.09  udisks-daemon
08:31:56 PM      7257    152.50      0.00   15704   1108   0.14  pidstat

vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity.

If there are lots of runnable processes, (the Procs r vmstat column), then that means there is more to do than the CPU can manage. If runnable processes hover much over 1 or 2, a faster computer will speed up the server significantly.

ubuntu@server3:~$ vmstat 5 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0  92152  34072  16144 300976    0    2    36    26  318  875  4 10 86  1
 0  0  92152  34064  16152 300976    0    0     0     7  329  876  2  6 92  1
 0  0  92152  34064  16160 300976    0    0     0     4  356 1020  2  4 93  1
 0  0  92152  34064  16168 300980    0    0     0     4  332  896  2  4 94  0
 0  0  92152  34064  16176 300980    0    0     0     8  338  949  1  3 96  0
ubuntu@server3:~$

Monitor with watch command and vmstat, memory usage

ubuntu@server3:~$ watch vmstat -sSM

Another userful script.

ubuntu@server3:~$ while (true); do clear; uname -n; echo ""; df -h /; echo ""; tail -5 /var/log/auth.log; echo ""; tail -5 /var/log/syslog; echo ""; vmstat 1 5; sleep 2; done

Process Monitorng

Monitoring file handles used by a particular process -r : repeat mode

lsof -c <process name> -r

ubuntu@server3:~$ lsof -c firefox -r

Monitoring a port connections

ubuntu@server3:~$ watch -n1 'netstat -tn | grep -P :22'

dstat - a mix of vmstat, iostat, netstat, ps, sar.

This is a very powerful command line tool to gather statistics for a Linux system.

http://dag.wieers.com/home-made/dstat/

dstat - versatile tool for generating system resource statistics'''

EXAMPLES

Using dstat to relate disk-throughput with network-usage (eth0), total CPU-usage and system counters:

dstat -dnyc -N eth0 -C total -f 5

Checking dstat’s behaviour and the system impact of dstat:

dstat -taf --debug

Using the time plugin together with cpu, net, disk, system, load, proc and top_cpu plugins:

dstat -tcndylp --top-cpu

this is identical to

dstat --time --cpu --net --disk --sys --load --proc --top-cpu

Using dstat to relate cpu stats with interrupts per device:

dstat -tcyif


Retrieved from http://www.itmission.org/Main/Linux-performance-monitoring
Page last modified on August 15, 2012, at 05:40 PM