|
Softwares          |
Main /
Linux-performance-monitoring
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.sar - Collect, report, or save system activity information.Requirements: sysstat package. Installing sysstat
ubuntu@server3:~$ sudo apt-get install sysstat [sudo] password for ubuntu: Reading package lists... Done Building dependency tree Reading state information... Done sysstat is already the newest version. 0 upgraded, 0 newly installed, 0 to remove and 240 not upgraded. ubuntu@server3:~$ Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds
ubuntu@server3:~$ sudo sar -u 2 5 [sudo] password for ubuntu: Linux 2.6.38-8-generic (server3) 10/31/2011 _x86_64_ (1 CPU) 07:46:06 AM CPU %user %nice %system %iowait %steal %idle 07:46:08 AM all 3.52 0.00 3.52 5.03 0.00 87.94 07:46:10 AM all 2.51 0.00 3.52 0.00 0.00 93.97 07:46:12 AM all 2.50 0.00 3.50 0.00 0.00 94.00 07:46:14 AM all 1.50 0.00 2.00 4.50 0.00 92.00 07:46:16 AM all 1.50 0.00 2.00 0.00 0.00 96.50 Average: all 2.30 0.00 2.91 1.90 0.00 92.89 ubuntu@server3:~$ The -u option specifies our interest in the CPU subsystem.
buntu@server3:~$ sar -u -f sardatafile Linux 2.6.38-8-generic (server3) 10/31/2011 _x86_64_ (1 CPU) 07:48:19 AM CPU %user %nice %system %iowait %steal %idle 07:48:21 AM all 0.00 0.00 1.50 0.00 0.00 98.50 07:48:23 AM all 0.50 0.00 1.00 0.00 0.00 98.50 07:48:25 AM all 1.00 0.00 0.50 7.50 0.00 91.00 Average: all 0.50 0.00 1.00 2.50 0.00 96.00 ubuntu@server3:~$ The %iowait and %idle columns are of interest to us when doing performance analysis. The %iowait column specifies the amount of time the %red%CPU spends waiting for I/O requests to complete%%. The %idle column tells us how much useful work the CPU is doing. A %idle time near zero indicates a CPU bottleneck. A high %iowait value indicates unsatisfactory disk performance. Additional information can be obtained by the sar -q command, which displays the run queue length, total number of processes, and the load averages for the past one, five and fifteen minutes: If you see more than one process is runnable at any given time, the the system will be busy.
runq-sz #Run queue length (number of processes waiting for run time). Check if system has free memory and no swap usage, then memory is not the problem.
To get swapping statistics
Displays I/O and transfer rate statistics grouped for all block devices
sar -d it displays statistics data similar to those displayed by iostat:
Report paging statistics.
sadf - Display data collected by sar in multiple formats Extract memory, swap space and network statistics from system activity file 'sa21', and display them in a format that can be ingested by a database.
Example: ubuntu@server3:~$ sadf -d /var/log/sysstat/sa09 -- -B server3;600;2011-10-09 09:55:01 UTC;6.29;18.12;43.23;0.10;266.73;0.00;0.00;0.00;0.00 server3;600;2011-10-09 10:05:01 UTC;4.29;4.00;24.69;0.02;209.58;0.00;0.00;0.00;0.00 Extract CPU statistics for processor 1 (the second processor) from current daily data file, and display them in a formatthat can easily be handled by a pattern processing command.
pidstat - Report statistics for Linux tasks. The pidstat command is used to monitor processes and threads currently being managed by the Linux kernel. It can also monitor the children of those processes and threads. With its -d option, pidstat can report I/O statistics ubuntu@server3:~$ pidstat -d 2 Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU) 05:28:18 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 05:28:20 PM 6082 0.00 4.00 0.00 gnome-terminal 05:28:20 PM 6727 0.00 4.00 0.00 rsyslogd EXAMPLES Display five reports of CPU statistics for every active task in the system at two second intervals.
Display five reports of page faults and memory statistics for PID 1643 at two second intervals.
Display global page faults and memory statistics for all the processes whose command name includes strings.
Display five reports of page faults statistics at two second intervals for the child processes of all tasks in the system.
The following example reports CPU statistics for PID 6968 and all its threads: ubuntu@server3:~$ pidstat -t -p 6968 3 2 Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU) 05:36:00 PM TGID TID %usr %system %guest %CPU CPU Command 05:36:03 PM 6968 - 0.33 0.00 0.00 0.33 0 soffice.bin 05:36:03 PM - 6968 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6970 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6972 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6973 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6976 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6983 0.00 0.00 0.00 0.00 0 |__soffice.bin What is majflt/s ? Number of major faults the system has made per second, those which have required loading a memory page from disk (post 2.5 kernels only). Page faults are not a bad thing. It's part of the way that the memory system works. http://en.wikipedia.org/wiki/Page_fault -r Report page faults and memory utilization. When reporting statistics for individual tasks, the following values are displayed: minflt/s Total number of minor faults the task has made per second, those which have not required loading a memory page from disk. majflt/s Total number of major faults the task has made per second, those which have required loading a memory page from disk. VSZ Virtual Size: The virtual memory usage of entire task in kilobytes. RSS Resident Set Size: The non-swapped physical memory used by the task in kilobytes. ubuntu@server3:~$ sudo pidstat -r 2 [sudo] password for ubuntu: Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU) 08:31:52 PM PID minflt/s majflt/s VSZ RSS %MEM Command 08:31:54 PM 1011 0.50 0.00 48488 3392 0.43 fail2ban-server 08:31:54 PM 1320 0.50 0.00 45168 744 0.09 udisks-daemon 08:31:54 PM 7257 147.76 0.00 15704 1068 0.13 pidstat 08:31:54 PM PID minflt/s majflt/s VSZ RSS %MEM Command 08:31:56 PM 1320 0.50 0.00 45168 744 0.09 udisks-daemon 08:31:56 PM 7257 152.50 0.00 15704 1108 0.14 pidstat vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity. If there are lots of runnable processes, (the Procs r vmstat column), then that means there is more to do than the CPU can manage. If runnable processes hover much over 1 or 2, a faster computer will speed up the server significantly. ubuntu@server3:~$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 92152 34072 16144 300976 0 2 36 26 318 875 4 10 86 1 0 0 92152 34064 16152 300976 0 0 0 7 329 876 2 6 92 1 0 0 92152 34064 16160 300976 0 0 0 4 356 1020 2 4 93 1 0 0 92152 34064 16168 300980 0 0 0 4 332 896 2 4 94 0 0 0 92152 34064 16176 300980 0 0 0 8 338 949 1 3 96 0 ubuntu@server3:~$ Monitor with watch command and vmstat, memory usage
Another userful script.
Process Monitorng Monitoring file handles used by a particular process -r : repeat mode
Monitoring a port connections
dstat - a mix of vmstat, iostat, netstat, ps, sar.This is a very powerful command line tool to gather statistics for a Linux system. http://dag.wieers.com/home-made/dstat/ dstat - versatile tool for generating system resource statistics'''EXAMPLES Using dstat to relate disk-throughput with network-usage (eth0), total CPU-usage and system counters:
Checking dstat’s behaviour and the system impact of dstat:
Using the time plugin together with cpu, net, disk, system, load, proc and top_cpu plugins:
this is identical to
Using dstat to relate cpu stats with interrupts per device:
|