|
Softwares          |
Main /
Linux-performance-monitoringMain.Linux-performance-monitoring HistoryHide minor edits - Show changes to output Changed lines 7-8 from:
to:
!!sar - Collect, report, or save system activity information. Added line 336:
Changed lines 1-2 from:
to:
(:Google1:) (:Googletxt:) Added lines 4-6:
!!Locate system bottlenecks and suggest some possible solutions to those annoying performance problems. ---- Deleted lines 28-33:
(:Google1:) (:Googletxt:) (:Googlemm:) Changed lines 1-2 from:
to:
!!Locate system bottlenecks and suggest some possible solutions to those annoying performance problems. Deleted lines 2-5:
---- !!Locate system bottlenecks and suggest some possible solutions to those annoying performance problems. ---- Added lines 25-30:
(:Google1:) (:Googletxt:) (:Googlemm:) Changed line 13 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 33 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 56 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 85 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 92 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 98 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 111 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 118 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 129 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 144 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 167 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 173 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 179 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 185 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 266 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 273 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 284 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 293 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed lines 298-299 from:
to:
!!!dstat - a mix of vmstat, iostat, netstat, ps, sar. Changed lines 304-305 from:
to:
!!dstat - versatile tool for generating system resource statistics''' Changed line 309 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 315 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 321 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 327 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Changed line 333 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :) Added lines 19-29:
[@ ubuntu@server3:~$ sudo apt-get install sysstat [sudo] password for ubuntu: Reading package lists... Done Building dependency tree Reading state information... Done sysstat is already the newest version. 0 upgraded, 0 newly installed, 0 to remove and 240 not upgraded. ubuntu@server3:~$ @] Changed line 6 from:
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems. to:
!!Locate system bottlenecks and suggest some possible solutions to those annoying performance problems. Deleted line 21:
Added lines 97-324:
(:tableend:) sar -d it displays statistics data similar to those displayed by iostat: (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) ubuntu@server3:~$sar -d (:tableend:) Report paging statistics. (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) ubuntu@server3:~$sar-B ubuntu@server3:~$ sar -B -f /var/log/sysstat/sa09 (:tableend:) '''sadf - Display data collected by sar in multiple formats''' Extract memory, swap space and network statistics from system activity file 'sa21', and display them in a format that can be ingested by a database. (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) sadf -d /var/log/sysstat/sa21 -- -r -n DEV (:tableend:) Example: [@ ubuntu@server3:~$ sadf -d /var/log/sysstat/sa09 -- -B server3;600;2011-10-09 09:55:01 UTC;6.29;18.12;43.23;0.10;266.73;0.00;0.00;0.00;0.00 server3;600;2011-10-09 10:05:01 UTC;4.29;4.00;24.69;0.02;209.58;0.00;0.00;0.00;0.00 @] Extract CPU statistics for processor 1 (the second processor) from current daily data file, and display them in a formatthat can easily be handled by a pattern processing command. (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) sadf -p -P 1 (:tableend:) '''pidstat - Report statistics for Linux tasks'''. The pidstat command is used to monitor processes and threads currently being managed by the Linux kernel. It can also monitor the children of those processes and threads. With its -d option, pidstat can report I/O statistics [@ ubuntu@server3:~$ pidstat -d 2 Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU) 05:28:18 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 05:28:20 PM 6082 0.00 4.00 0.00 gnome-terminal 05:28:20 PM 6727 0.00 4.00 0.00 rsyslogd @] ''EXAMPLES'' Display five reports of CPU statistics for every active task in the system at two second intervals. (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) pidstat 2 5 (:tableend:) Display five reports of page faults and memory statistics for PID 1643 at two second intervals. (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) pidstat -r -p 1643 2 5 (:tableend:) Display global page faults and memory statistics for all the processes whose command name includes strings. (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) pidstat -C "fox|bird" -r -p ALL (:tableend:) Display five reports of page faults statistics at two second intervals for the child processes of all tasks in the system. (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) pidstat -T CHILD -r 2 5 (:tableend:) The following example reports CPU statistics for PID 6968 and all its threads: [@ ubuntu@server3:~$ pidstat -t -p 6968 3 2 Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU) 05:36:00 PM TGID TID %usr %system %guest %CPU CPU Command 05:36:03 PM 6968 - 0.33 0.00 0.00 0.33 0 soffice.bin 05:36:03 PM - 6968 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6970 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6972 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6973 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6976 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6983 0.00 0.00 0.00 0.00 0 |__soffice.bin @] '''What is majflt/s''' ? Number of major faults the system has made per second, those which have required loading a memory page from disk (post 2.5 kernels only). Page faults are not a bad thing. It's part of the way that the memory system works. http://en.wikipedia.org/wiki/Page_fault -r Report page faults and memory utilization. When reporting statistics for individual tasks, the following values are displayed: ''minflt/s'' Total number of minor faults the task has made per second, those which have not required loading a memory page from disk. ''majflt/s'' Total number of major faults the task has made per second, those which have required loading a memory page from disk. ''VSZ'' Virtual Size: The virtual memory usage of entire task in kilobytes. ''RSS'' Resident Set Size: The non-swapped physical memory used by the task in kilobytes. [@ ubuntu@server3:~$ sudo pidstat -r 2 [sudo] password for ubuntu: Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU) 08:31:52 PM PID minflt/s majflt/s VSZ RSS %MEM Command 08:31:54 PM 1011 0.50 0.00 48488 3392 0.43 fail2ban-server 08:31:54 PM 1320 0.50 0.00 45168 744 0.09 udisks-daemon 08:31:54 PM 7257 147.76 0.00 15704 1068 0.13 pidstat 08:31:54 PM PID minflt/s majflt/s VSZ RSS %MEM Command 08:31:56 PM 1320 0.50 0.00 45168 744 0.09 udisks-daemon 08:31:56 PM 7257 152.50 0.00 15704 1108 0.14 pidstat @] ''vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity''. If there are lots of runnable processes, (the Procs r vmstat column), then that means there is more to do than the CPU can manage. If runnable processes hover much over 1 or 2, a faster computer will speed up the server significantly. [@ ubuntu@server3:~$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 92152 34072 16144 300976 0 2 36 26 318 875 4 10 86 1 0 0 92152 34064 16152 300976 0 0 0 7 329 876 2 6 92 1 0 0 92152 34064 16160 300976 0 0 0 4 356 1020 2 4 93 1 0 0 92152 34064 16168 300980 0 0 0 4 332 896 2 4 94 0 0 0 92152 34064 16176 300980 0 0 0 8 338 949 1 3 96 0 ubuntu@server3:~$ @] Monitor with watch command and vmstat, memory usage (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) ubuntu@server3:~$ watch vmstat -sSM (:tableend:) ''Another userful script''. (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) ubuntu@server3:~$ while (true); do clear; uname -n; echo ""; df -h /; echo ""; tail -5 /var/log/auth.log; echo ""; tail -5 /var/log/syslog; echo ""; vmstat 1 5; sleep 2; done (:tableend:) '''Process Monitorng''' Monitoring file handles used by a particular process -r : repeat mode (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) lsof -c <process name> -r ubuntu@server3:~$ lsof -c firefox -r (:tableend:) '''Monitoring a port connections''' (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) ubuntu@server3:~$ watch -n1 'netstat -tn | grep -P :22' (:tableend:) '''dstat - a mix of vmstat, iostat, netstat, ps, sar...''' This is a very powerful command line tool to gather statistics for a Linux system. http://dag.wieers.com/home-made/dstat/ '''dstat - versatile tool for generating system resource statistics''' EXAMPLES Using dstat to relate disk-throughput with network-usage (eth0), total CPU-usage and system counters: (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) dstat -dnyc -N eth0 -C total -f 5 (:tableend:) Checking dstat’s behaviour and the system impact of dstat: (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) dstat -taf --debug (:tableend:) Using the time plugin together with cpu, net, disk, system, load, proc and top_cpu plugins: (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) dstat -tcndylp --top-cpu (:tableend:) this is identical to (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) dstat --time --cpu --net --disk --sys --load --proc --top-cpu (:tableend:) Using dstat to relate cpu stats with interrupts per device: (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) dstat -tcyif Added lines 75-97:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) ubuntu@server3:~$sar -q (:tableend:) runq-sz #Run queue length (number of processes waiting for run time). Check if system has free memory and no swap usage, then memory is not the problem. (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) ubuntu@server3:~$ sar -r (:tableend:) To get swapping statistics (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) ubuntu@server3:~$ sar -W (:tableend:) Displays I/O and transfer rate statistics grouped for all block devices (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) ubuntu@server3:~$sar -b (:tableend:) Changed lines 20-22 from:
% Generated the CPU utilization stats with 5 lines /every 2 to:
%red%Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds%% %red%Generated the CPU utilization stats with 5 lines /every 2 seconds%% Changed lines 43-44 from:
The -u option specifies our interest in the CPU subsystem. to:
The -u option specifies our interest in the CPU subsystem.\\ Changed line 65 from:
The %iowait column specifies the amount of time the CPU spends waiting for I/O requests to complete. to:
The %iowait column specifies the amount of time the %red%CPU spends waiting for I/O requests to complete%%. Changed lines 71-72 from:
Additional information can be obtained by the sar -q command, which displays the run queue length, total number of processes, and the load averages for the past one, five and fifteen minutes: to:
Additional information can be obtained by the sar -q command, which displays the run queue length, total number of processes, and the load averages for the past one, five and fifteen minutes: Added line 78:
---- Changed line 2 from:
to:
Changed lines 6-7 from:
to:
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems. ---- Deleted lines 1-3:
(:Google1:) --- Added lines 3-6:
---- (:Google1:) ---- Changed line 2 from:
to:
--- Added lines 4-5:
--- Locate system bottlenecks and suggest some possible solutions to those annoying performance problems. Deleted line 0:
Deleted line 1:
Changed line 3 from:
to:
(:Google1:) Changed lines 4-6 from:
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems. %blue% to:
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems. '''sar - Collect, report, or save system activity information'''. Changed lines 41-42 from:
The -u option specifies our interest in the CPU subsystem. to:
The -u option specifies our interest in the CPU subsystem. Changed lines 73-231 from:
runq-sz #Run queue length (number of processes waiting for run time). # Check if system has free memory and no swap usage, then memory is not the problem. ubuntu@server3:~$ %red%sar -r%red% # To get swapping statistics ubuntu@server3:~$ %red%sar -W%red% '''#displays I/O and transfer rate statistics grouped for all block devices''' ubuntu@server3:~$sar -b #enables you to get more detailed information on a per device basis. #It displays statistics data similar to those displayed by iostat: ubuntu@server3:~$sar -d #Report paging statistics. ubuntu@server3:~$sar-B ubuntu@server3:~$ sar -B -f /var/log/sysstat/sa09 sadf - Display data collected by sar in multiple formats #Extract memory, swap space and network statistics from system activity file 'sa21', and display them in a format that can be ingested by a database. sadf -d /var/log/sysstat/sa21 -- -r -n DEV Example: ubuntu@server3:~$ sadf -d /var/log/sysstat/sa09 -- -B server3;600;2011-10-09 09:55:01 UTC;6.29;18.12;43.23;0.10;266.73;0.00;0.00;0.00;0.00 server3;600;2011-10-09 10:05:01 UTC;4.29;4.00;24.69;0.02;209.58;0.00;0.00;0.00;0.00 #Extract CPU statistics for processor 1 (the second processor) from current daily data file, and display them in a format that can easily be handled by a pattern processing command. sadf -p -P 1 pidstat - Report statistics for Linux tasks. The pidstat command is used to monitor processes and threads currently being managed by the Linux kernel. It can also monitor the children of those processes and threads. #With its -d option, pidstat can report I/O statistics ubuntu@server3:~$ pidstat -d 2 Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU) 05:28:18 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 05:28:20 PM 6082 0.00 4.00 0.00 gnome-terminal 05:28:20 PM 6727 0.00 4.00 0.00 rsyslogd EXAMPLES #Display five reports of CPU statistics for every active task in the system at two second intervals. pidstat 2 5 #Display five reports of page faults and memory statistics for PID 1643 at two second intervals. pidstat -r -p 1643 2 5 #Display global page faults and memory statistics for all the processes whose command name includes strings. pidstat -C "fox|bird" -r -p ALL #Display five reports of page faults statistics at two second intervals for the child processes of all tasks in the system. pidstat -T CHILD -r 2 5 #The following example reports CPU statistics for PID 6968 and all its threads: ubuntu@server3:~$ pidstat -t -p 6968 3 2 Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU) 05:36:00 PM TGID TID %usr %system %guest %CPU CPU Command 05:36:03 PM 6968 - 0.33 0.00 0.00 0.33 0 soffice.bin 05:36:03 PM - 6968 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6970 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6972 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6973 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6976 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6983 0.00 0.00 0.00 0.00 0 |__soffice.bin What is majflt/s ? Number of major faults the system has made per second, those which have required loading a memory page from disk (post 2.5 kernels only). Page faults are not a bad thing. It's part of the way that the memory system works. http://en.wikipedia.org/wiki/Page_fault -r Report page faults and memory utilization. When reporting statistics for individual tasks, the following values are displayed: minflt/s Total number of minor faults the task has made per second, those which have not required loading a memory page from disk. majflt/s Total number of major faults the task has made per second, those which have required loading a memory page from disk. VSZ Virtual Size: The virtual memory usage of entire task in kilobytes. RSS Resident Set Size: The non-swapped physical memory used by the task in kilobytes. ubuntu@server3:~$ sudo pidstat -r 2 [sudo] password for ubuntu: Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU) 08:31:52 PM PID minflt/s majflt/s VSZ RSS %MEM Command 08:31:54 PM 1011 0.50 0.00 48488 3392 0.43 fail2ban-server 08:31:54 PM 1320 0.50 0.00 45168 744 0.09 udisks-daemon 08:31:54 PM 7257 147.76 0.00 15704 1068 0.13 pidstat 08:31:54 PM PID minflt/s majflt/s VSZ RSS %MEM Command 08:31:56 PM 1320 0.50 0.00 45168 744 0.09 udisks-daemon 08:31:56 PM 7257 152.50 0.00 15704 1108 0.14 pidstat vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity. If there are lots of runnable processes, (the Procs r vmstat column), then that means there is more to do than the CPU can manage. If runnable processes hover much over 1 or 2, a faster computer will speed up the server significantly. ubuntu@server3:~$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 92152 34072 16144 300976 0 2 36 26 318 875 4 10 86 1 0 0 92152 34064 16152 300976 0 0 0 7 329 876 2 6 92 1 0 0 92152 34064 16160 300976 0 0 0 4 356 1020 2 4 93 1 0 0 92152 34064 16168 300980 0 0 0 4 332 896 2 4 94 0 0 0 92152 34064 16176 300980 0 0 0 8 338 949 1 3 96 0 ubuntu@server3:~$ #Monitor with watch command and vmstat, memory usage ubuntu@server3:~$ watch vmstat -sSM #Another userful script. ubuntu@server3:~$ while (true); do clear; uname -n; echo ""; df -h /; echo ""; tail -5 /var/log/auth.log; echo ""; tail -5 /var/log/syslog; echo ""; vmstat 1 5; sleep 2; done Process Monitorng #Monitoring file handles used by a particular process -r : repeat mode lsof -c <process name> -r ubuntu@server3:~$ lsof -c firefox -r Monitoring a port connections ubuntu@server3:~$ watch -n1 'netstat -tn | grep -P :22' dstat - a mix of vmstat, iostat, netstat, ps, sar... This is a very powerful command line tool to gather statistics for a Linux system. http://dag.wieers.com/home-made/dstat/ dstat - versatile tool for generating system resource statistics EXAMPLES Using dstat to relate disk-throughput with network-usage (eth0), total CPU-usage and system counters: dstat -dnyc -N eth0 -C total -f 5 Checking dstat’s behaviour and the system impact of dstat: dstat -taf --debug Using the time plugin together with cpu, net, disk, system, load, proc and top_cpu plugins: dstat -tcndylp --top-cpu this is identical to dstat --time --cpu --net --disk --sys --load --proc --top-cpu Using dstat to relate cpu stats with interrupts per device: dstat -tcyif to:
(:Google1:) (:Googlemm:) Changed line 3 from:
to:
Changed lines 12-14 from:
sudo apt-get install sysstat to:
sudo apt-get install sysstat yum install sysstat Deleted lines 70-71:
(:cellnr:) Changed lines 73-74 from:
(:cellnr:) to:
Added line 22:
Changed line 43 from:
ubuntu@server3:~$ to:
ubuntu@server3:~$sudo sar -u -o sardatafile 2 3 Changed lines 46-53 from:
The %iowait column specifies the amount of time the CPU spends waiting for I/O requests to complete.\\ The A A high %iowait value indicates unsatisfactory disk performance.\\ Additional information can be obtained by the sar -q command, which displays the run queue length, total number of processes, to:
[@ buntu@server3:~$ sar -u -f sardatafile Linux 2.6.38-8-generic (server3) 10/31/2011 _x86_64_ (1 CPU) 07:48:19 AM CPU %user %nice %system %iowait %steal %idle 07:48:21 AM all 0.00 0.00 1.50 0.00 0.00 98.50 07:48:23 AM all 0.50 0.00 1.00 0.00 0.00 98.50 07:48:25 AM all 1.00 0.00 0.50 7.50 0.00 91.00 Average: all 0.50 0.00 1.00 2.50 0.00 96.00 ubuntu@server3:~$ @] [@ The %iowait and %idle columns are of interest to us when doing performance analysis. The %iowait column specifies the amount of time the CPU spends waiting for I/O requests to complete. The %idle column tells us how much useful work the CPU is doing. A %idle time near zero indicates a CPU bottleneck. A high %iowait value indicates unsatisfactory disk performance. Additional information can be obtained by the sar -q command, which displays the run queue length, total number of processes, and the load averages for the past one, five and fifteen minutes: Changed lines 67-68 from:
to:
@] Changed lines 22-36 from:
to:
[@ ubuntu@server3:~$ sudo sar -u 2 5 [sudo] password for ubuntu: Linux 2.6.38-8-generic (server3) 10/31/2011 _x86_64_ (1 CPU) 07:46:06 AM CPU %user %nice %system %iowait %steal %idle 07:46:08 AM all 3.52 0.00 3.52 5.03 0.00 87.94 07:46:10 AM all 2.51 0.00 3.52 0.00 0.00 93.97 07:46:12 AM all 2.50 0.00 3.50 0.00 0.00 94.00 07:46:14 AM all 1.50 0.00 2.00 4.50 0.00 92.00 07:46:16 AM all 1.50 0.00 2.00 0.00 0.00 96.50 Average: all 2.30 0.00 2.91 1.90 0.00 92.89 ubuntu@server3:~$ @] Changed line 60 from:
(:cellnr: to:
(:cellnr:) Changed line 14 from:
to:
Changed line 17 from:
Generated the CPU utilization stats with 5 lines /every 2 to:
Generated the CPU utilization stats with 5 lines /every 2 seconds Changed line 20 from:
ubuntu@server3:~$ to:
ubuntu@server3:~$ sudo sar -u 2 5 Changed lines 24-26 from:
The -o option will create an output file that contains binary data. 3 samples at two-second intervals. to:
The -o option will create an output file that contains binary data. 3 samples at two-second intervals. Changed line 27 from:
(:cellnr: to:
(:cellnr:) Changed line 17 from:
to:
Generated the CPU utilization stats with 5 lines /every 2 seconds\\ Changed lines 23-24 from:
to:
The -u option specifies our interest in the CPU subsystem.\\ The -o option will create an output file that contains binary data. 3 samples at two-second intervals.\\ Added line 31:
Added line 6:
Added line 8:
Changed lines 15-16 from:
%blue%Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds to:
%blue%Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds%% Added line 22:
Deleted line 6:
Changed lines 12-15 from:
''' to:
\\ %blue%Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds/%blue%\\ #Generated the CPU utilization stats with 5 lines /every 2 seconds\\ (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) Changed lines 18-20 from:
to:
(:tableend:) #The -u option specifies our interest in the CPU subsystem.\\ #The -o option will create an output file that contains binary data. 3 samples at two-second intervals.\\ (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr: Changed lines 24-32 from:
The %iowait and %idle columns are of interest to us when doing performance analysis. The %iowait column specifies the amount of time the CPU spends waiting for I/O requests to complete. The %idle column tells us how much useful work the CPU is doing. A %idle time near zero indicates a CPU bottleneck. A high %iowait value indicates unsatisfactory disk performance. Additional information can be obtained by the sar -q command, which displays the run queue length, total number of processes, and the load averages for the past one, five and fifteen minutes: to:
(:tableend:) The %iowait and %idle columns are of interest to us when doing performance analysis.\\ The %iowait column specifies the amount of time the CPU spends waiting for I/O requests to complete.\\ The %idle column tells us how much useful work the CPU is doing.\\ A %idle time near zero indicates a CPU bottleneck.\\ A high %iowait value indicates unsatisfactory disk performance.\\ Additional information can be obtained by the sar -q command, which displays the run queue length, total number of processes, and the load averages for the past one, five and fifteen minutes:\\ Changed lines 34-35 from:
to:
\\ (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) ubuntu@server3:~$sar -q Changed lines 39-40 from:
to:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr: Changed line 3 from:
to:
\\ Changed line 5 from:
%blue%sar - Collect, report, or save system activity information.%% to:
%blue%sar - Collect, report, or save system activity information.%%\\ Changed lines 3-4 from:
-- Locate system bottlenecks and suggest some possible solutions to those annoying performance problems. to:
-- Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.\\ Changed line 3 from:
to:
--- Changed lines 5-6 from:
to:
%blue%sar - Collect, report, or save system activity information.%% Changed lines 8-9 from:
to:
%blue%Installing sysstat%% (:table border=1 bgcolor=#cccc99 cellspacing=0 :) (:cellnr:) sudo apt-get install sysstat / yum install sysstat (:tableend:) Deleted line 4:
Changed lines 7-8 from:
Requirements: sysstat package. to:
Requirements: ''sysstat package''. Deleted line 9:
Changed line 12 from:
Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds/ to:
'''Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds/''' Changed lines 14-15 from:
ubuntu@server3:~$ sudo sar -u 2 5 to:
ubuntu@server3:~$ %red%sudo sar -u 2 5%red% Changed lines 18-19 from:
ubuntu@server3:~$ sudo sar -u -o sardatafile 2 3 to:
ubuntu@server3:~$ %red%sudo sar -u -o sardatafile 2 3%red% Changed line 30 from:
ubuntu@server3:~$ sar - to:
ubuntu@server3:~$ %red%sar -q%red% Changed line 34 from:
ubuntu@server3:~$ sar - to:
ubuntu@server3:~$ %red%sar -r%red% Changed lines 36-38 from:
ubuntu@server3:~$ sar -W #displays I/O and transfer rate statistics grouped for all block to:
ubuntu@server3:~$ %red%sar -W%red% '''#displays I/O and transfer rate statistics grouped for all block devices''' Added line 3:
Changed lines 10-12 from:
'' ''' to:
''Installing sysstat'' %red%sudo apt-get install sysstat / yum install sysstat%red% Added lines 1-189:
(:Google1:) (:Googlemm:) Locate system bottlenecks and suggest some possible solutions to those annoying performance problems. '''sar - Collect, report, or save system activity information.''' Requirements: sysstat package. '''#Installing sysstat''' '''sudo apt-get install sysstat / yum install sysstat''' Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds/ #Generated the CPU utilization stats with 5 lines /every 2 seconds ubuntu@server3:~$ sudo sar -u 2 5 #The -u option specifies our interest in the CPU subsystem. #The -o option will create an output file that contains binary data. 3 samples at two-second intervals. ubuntu@server3:~$ sudo sar -u -o sardatafile 2 3 The %iowait and %idle columns are of interest to us when doing performance analysis. The %iowait column specifies the amount of time the CPU spends waiting for I/O requests to complete. The %idle column tells us how much useful work the CPU is doing. A %idle time near zero indicates a CPU bottleneck. A high %iowait value indicates unsatisfactory disk performance. Additional information can be obtained by the sar -q command, which displays the run queue length, total number of processes, and the load averages for the past one, five and fifteen minutes: If you see more than one process is runnable at any given time, the the system will be busy. ubuntu@server3:~$ sar -q runq-sz #Run queue length (number of processes waiting for run time). # Check if system has free memory and no swap usage, then memory is not the problem. ubuntu@server3:~$ sar -r # To get swapping statistics ubuntu@server3:~$ sar -W #displays I/O and transfer rate statistics grouped for all block devices ubuntu@server3:~$sar -b #enables you to get more detailed information on a per device basis. #It displays statistics data similar to those displayed by iostat: ubuntu@server3:~$sar -d #Report paging statistics. ubuntu@server3:~$sar-B ubuntu@server3:~$ sar -B -f /var/log/sysstat/sa09 sadf - Display data collected by sar in multiple formats #Extract memory, swap space and network statistics from system activity file 'sa21', and display them in a format that can be ingested by a database. sadf -d /var/log/sysstat/sa21 -- -r -n DEV Example: ubuntu@server3:~$ sadf -d /var/log/sysstat/sa09 -- -B server3;600;2011-10-09 09:55:01 UTC;6.29;18.12;43.23;0.10;266.73;0.00;0.00;0.00;0.00 server3;600;2011-10-09 10:05:01 UTC;4.29;4.00;24.69;0.02;209.58;0.00;0.00;0.00;0.00 #Extract CPU statistics for processor 1 (the second processor) from current daily data file, and display them in a format that can easily be handled by a pattern processing command. sadf -p -P 1 pidstat - Report statistics for Linux tasks. The pidstat command is used to monitor processes and threads currently being managed by the Linux kernel. It can also monitor the children of those processes and threads. #With its -d option, pidstat can report I/O statistics ubuntu@server3:~$ pidstat -d 2 Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU) 05:28:18 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command 05:28:20 PM 6082 0.00 4.00 0.00 gnome-terminal 05:28:20 PM 6727 0.00 4.00 0.00 rsyslogd EXAMPLES #Display five reports of CPU statistics for every active task in the system at two second intervals. pidstat 2 5 #Display five reports of page faults and memory statistics for PID 1643 at two second intervals. pidstat -r -p 1643 2 5 #Display global page faults and memory statistics for all the processes whose command name includes strings. pidstat -C "fox|bird" -r -p ALL #Display five reports of page faults statistics at two second intervals for the child processes of all tasks in the system. pidstat -T CHILD -r 2 5 #The following example reports CPU statistics for PID 6968 and all its threads: ubuntu@server3:~$ pidstat -t -p 6968 3 2 Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU) 05:36:00 PM TGID TID %usr %system %guest %CPU CPU Command 05:36:03 PM 6968 - 0.33 0.00 0.00 0.33 0 soffice.bin 05:36:03 PM - 6968 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6970 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6972 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6973 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6976 0.00 0.00 0.00 0.00 0 |__soffice.bin 05:36:03 PM - 6983 0.00 0.00 0.00 0.00 0 |__soffice.bin What is majflt/s ? Number of major faults the system has made per second, those which have required loading a memory page from disk (post 2.5 kernels only). Page faults are not a bad thing. It's part of the way that the memory system works. http://en.wikipedia.org/wiki/Page_fault -r Report page faults and memory utilization. When reporting statistics for individual tasks, the following values are displayed: minflt/s Total number of minor faults the task has made per second, those which have not required loading a memory page from disk. majflt/s Total number of major faults the task has made per second, those which have required loading a memory page from disk. VSZ Virtual Size: The virtual memory usage of entire task in kilobytes. RSS Resident Set Size: The non-swapped physical memory used by the task in kilobytes. ubuntu@server3:~$ sudo pidstat -r 2 [sudo] password for ubuntu: Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU) 08:31:52 PM PID minflt/s majflt/s VSZ RSS %MEM Command 08:31:54 PM 1011 0.50 0.00 48488 3392 0.43 fail2ban-server 08:31:54 PM 1320 0.50 0.00 45168 744 0.09 udisks-daemon 08:31:54 PM 7257 147.76 0.00 15704 1068 0.13 pidstat 08:31:54 PM PID minflt/s majflt/s VSZ RSS %MEM Command 08:31:56 PM 1320 0.50 0.00 45168 744 0.09 udisks-daemon 08:31:56 PM 7257 152.50 0.00 15704 1108 0.14 pidstat vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity. If there are lots of runnable processes, (the Procs r vmstat column), then that means there is more to do than the CPU can manage. If runnable processes hover much over 1 or 2, a faster computer will speed up the server significantly. ubuntu@server3:~$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 92152 34072 16144 300976 0 2 36 26 318 875 4 10 86 1 0 0 92152 34064 16152 300976 0 0 0 7 329 876 2 6 92 1 0 0 92152 34064 16160 300976 0 0 0 4 356 1020 2 4 93 1 0 0 92152 34064 16168 300980 0 0 0 4 332 896 2 4 94 0 0 0 92152 34064 16176 300980 0 0 0 8 338 949 1 3 96 0 ubuntu@server3:~$ #Monitor with watch command and vmstat, memory usage ubuntu@server3:~$ watch vmstat -sSM #Another userful script. ubuntu@server3:~$ while (true); do clear; uname -n; echo ""; df -h /; echo ""; tail -5 /var/log/auth.log; echo ""; tail -5 /var/log/syslog; echo ""; vmstat 1 5; sleep 2; done Process Monitorng #Monitoring file handles used by a particular process -r : repeat mode lsof -c <process name> -r ubuntu@server3:~$ lsof -c firefox -r Monitoring a port connections ubuntu@server3:~$ watch -n1 'netstat -tn | grep -P :22' dstat - a mix of vmstat, iostat, netstat, ps, sar... This is a very powerful command line tool to gather statistics for a Linux system. http://dag.wieers.com/home-made/dstat/ dstat - versatile tool for generating system resource statistics EXAMPLES Using dstat to relate disk-throughput with network-usage (eth0), total CPU-usage and system counters: dstat -dnyc -N eth0 -C total -f 5 Checking dstat’s behaviour and the system impact of dstat: dstat -taf --debug Using the time plugin together with cpu, net, disk, system, load, proc and top_cpu plugins: dstat -tcndylp --top-cpu this is identical to dstat --time --cpu --net --disk --sys --load --proc --top-cpu Using dstat to relate cpu stats with interrupts per device: dstat -tcyif |