Recent Changes - Search:

Softwares

.

Linux-performance-monitoring

Main.Linux-performance-monitoring History

Hide minor edits - Show changes to output

August 15, 2012, at 01:40 PM by 115.242.228.192 -
Added line 2:
(:Googletxt:)
Changed lines 335-337 from:
to:
----
August 15, 2012, at 12:55 PM by 115.242.228.192 -
Deleted line 1:
(:Googletxt:)
Deleted line 2:
August 15, 2012, at 12:54 PM by 115.242.228.192 -
Changed lines 338-339 from:
(:Google1:)
----
to:
August 15, 2012, at 12:54 PM by 115.242.228.192 -
Changed lines 7-8 from:
'''sar - Collect, report, or save system activity information'''.
to:
!!sar - Collect, report, or save system activity information.
Added line 336:
August 15, 2012, at 12:53 PM by 115.242.228.192 -
Changed lines 1-2 from:
!!Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.
to:
(:Google1:)
(:Googletxt:)
Added lines 4-6:

!!Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.
----
Deleted lines 28-33:

(:Google1:)
(:Googletxt:)

(:Googlemm:)
July 04, 2012, at 01:35 AM by 115.242.129.136 -
Changed lines 1-2 from:
(:Googlemm:)
to:
!!Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.
Deleted lines 2-5:
(:Google1:)
----
!!Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.
----
Added lines 25-30:

(:Google1:)
(:Googletxt:)

(:Googlemm:)
Changed line 13 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 33 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 56 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 85 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 92 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 98 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 111 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 118 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 129 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 144 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 167 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 173 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 179 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 185 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 266 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 273 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 284 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 293 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed lines 298-299 from:
'''dstat - a mix of vmstat, iostat, netstat, ps, sar...'''
to:
!!!dstat - a mix of vmstat, iostat, netstat, ps, sar.
Changed lines 304-305 from:
'''dstat - versatile tool for generating system resource statistics'''
to:
!!dstat - versatile tool for generating system resource statistics'''
Changed line 309 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 315 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 321 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 327 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
Changed line 333 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
to:
(:table border=1 width=50% bgcolor=#cccc99 cellspacing=0 :)
October 31, 2011, at 01:24 AM by 117.99.109.84 -
Added lines 19-29:

[@
ubuntu@server3:~$ sudo apt-get install sysstat
[sudo] password for ubuntu:
Reading package lists... Done
Building dependency tree
Reading state information... Done
sysstat is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 240 not upgraded.
ubuntu@server3:~$
@]
October 31, 2011, at 01:21 AM by 117.99.109.84 -
Changed line 6 from:
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.
to:
!!Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.
Deleted line 21:
%red%Generated the CPU utilization stats with 5 lines /every 2 seconds%%
October 31, 2011, at 12:21 AM by 117.99.109.84 -
Added line 277:
October 31, 2011, at 12:14 AM by 117.99.109.84 -
Added lines 97-324:
(:tableend:)

sar -d it displays statistics data similar to those displayed by iostat:

(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
ubuntu@server3:~$sar -d
(:tableend:)

Report paging statistics.

(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
ubuntu@server3:~$sar-B
ubuntu@server3:~$ sar -B -f /var/log/sysstat/sa09
(:tableend:)

'''sadf - Display data collected by sar in multiple formats'''

Extract memory, swap space and network statistics from system activity file 'sa21', and
display them in a format that can be ingested by a database.

(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
sadf -d /var/log/sysstat/sa21 -- -r -n DEV
(:tableend:)

Example:
[@
ubuntu@server3:~$ sadf -d /var/log/sysstat/sa09 -- -B
server3;600;2011-10-09 09:55:01 UTC;6.29;18.12;43.23;0.10;266.73;0.00;0.00;0.00;0.00
server3;600;2011-10-09 10:05:01 UTC;4.29;4.00;24.69;0.02;209.58;0.00;0.00;0.00;0.00
@]

Extract CPU statistics for processor 1 (the second processor) from current daily data file,
and display them in a formatthat can easily be handled by a pattern processing command.

(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
sadf -p -P 1
(:tableend:)

'''pidstat - Report statistics for Linux tasks'''.

The pidstat command is used to monitor processes and threads currently being managed by the Linux kernel.
It can also monitor the children of those processes and threads.

With its -d option, pidstat can report I/O statistics

[@
ubuntu@server3:~$ pidstat -d 2
Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU)
05:28:18 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command
05:28:20 PM 6082 0.00 4.00 0.00 gnome-terminal
05:28:20 PM 6727 0.00 4.00 0.00 rsyslogd
@]

''EXAMPLES''

Display five reports of CPU statistics for every active task in the system at two second intervals.
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
pidstat 2 5
(:tableend:)

Display five reports of page faults and memory statistics for PID 1643 at two second intervals.
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
pidstat -r -p 1643 2 5
(:tableend:)

Display global page faults and memory statistics for all the processes whose command name includes strings.
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
pidstat -C "fox|bird" -r -p ALL
(:tableend:)

Display five reports of page faults statistics at two second intervals for the child processes of all tasks in the system.
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
pidstat -T CHILD -r 2 5
(:tableend:)

The following example reports CPU statistics for PID 6968 and all its threads:
[@
ubuntu@server3:~$ pidstat -t -p 6968 3 2

Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU)
05:36:00 PM TGID TID %usr %system %guest %CPU CPU Command
05:36:03 PM 6968 - 0.33 0.00 0.00 0.33 0 soffice.bin
05:36:03 PM - 6968 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6970 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6972 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6973 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6976 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6983 0.00 0.00 0.00 0.00 0 |__soffice.bin
@]

'''What is majflt/s''' ?

Number of major faults the system has made per second,
those which have required loading a memory page from disk (post 2.5 kernels only).
Page faults are not a bad thing. It's part of the way that the memory system works.

http://en.wikipedia.org/wiki/Page_fault

-r

Report page faults and memory utilization.
When reporting statistics for individual tasks, the following values are displayed:

''minflt/s''
Total number of minor faults the task has made per second, those which have not required loading a memory page from disk.

''majflt/s''
Total number of major faults the task has made per second, those which have required loading a memory page from disk.

''VSZ''
Virtual Size: The virtual memory usage of entire task in kilobytes.

''RSS''
Resident Set Size: The non-swapped physical memory used by the task in kilobytes.


[@
ubuntu@server3:~$ sudo pidstat -r 2
[sudo] password for ubuntu:
Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU)

08:31:52 PM PID minflt/s majflt/s VSZ RSS %MEM Command
08:31:54 PM 1011 0.50 0.00 48488 3392 0.43 fail2ban-server
08:31:54 PM 1320 0.50 0.00 45168 744 0.09 udisks-daemon
08:31:54 PM 7257 147.76 0.00 15704 1068 0.13 pidstat

08:31:54 PM PID minflt/s majflt/s VSZ RSS %MEM Command
08:31:56 PM 1320 0.50 0.00 45168 744 0.09 udisks-daemon
08:31:56 PM 7257 152.50 0.00 15704 1108 0.14 pidstat
@]

''vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity''.

If there are lots of runnable processes, (the Procs r vmstat column),
then that means there is more to do than the CPU can manage.
If runnable processes hover much over 1 or 2, a faster computer will speed up the server significantly.

[@
ubuntu@server3:~$ vmstat 5 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 92152 34072 16144 300976 0 2 36 26 318 875 4 10 86 1
0 0 92152 34064 16152 300976 0 0 0 7 329 876 2 6 92 1
0 0 92152 34064 16160 300976 0 0 0 4 356 1020 2 4 93 1
0 0 92152 34064 16168 300980 0 0 0 4 332 896 2 4 94 0
0 0 92152 34064 16176 300980 0 0 0 8 338 949 1 3 96 0
ubuntu@server3:~$
@]

Monitor with watch command and vmstat, memory usage

(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
ubuntu@server3:~$ watch vmstat -sSM
(:tableend:)

''Another userful script''.

(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
ubuntu@server3:~$ while (true); do clear; uname -n; echo ""; df -h /; echo "";
tail -5 /var/log/auth.log; echo ""; tail -5 /var/log/syslog; echo ""; vmstat 1 5; sleep 2; done
(:tableend:)


'''Process Monitorng'''

Monitoring file handles used by a particular process -r : repeat mode

(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
lsof -c <process name> -r
ubuntu@server3:~$ lsof -c firefox -r
(:tableend:)

'''Monitoring a port connections'''

(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
ubuntu@server3:~$ watch -n1 'netstat -tn | grep -P :22'
(:tableend:)

'''dstat - a mix of vmstat, iostat, netstat, ps, sar...'''

This is a very powerful command line tool to gather statistics for a Linux system.

http://dag.wieers.com/home-made/dstat/

'''dstat - versatile tool for generating system resource statistics'''

EXAMPLES

Using dstat to relate disk-throughput with network-usage (eth0), total CPU-usage and system counters:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
dstat -dnyc -N eth0 -C total -f 5
(:tableend:)

Checking dstat’s behaviour and the system impact of dstat:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
dstat -taf --debug
(:tableend:)

Using the time plugin together with cpu, net, disk, system, load, proc and top_cpu plugins:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
dstat -tcndylp --top-cpu
(:tableend:)

this is identical to
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
dstat --time --cpu --net --disk --sys --load --proc --top-cpu
(:tableend:)

Using dstat to relate cpu stats with interrupts per device:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
dstat -tcyif
October 30, 2011, at 10:43 PM by 117.99.109.84 -
Added lines 75-97:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
ubuntu@server3:~$sar -q
(:tableend:)
runq-sz #Run queue length (number of processes waiting for run time).

Check if system has free memory and no swap usage, then memory is not the problem.
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
ubuntu@server3:~$ sar -r
(:tableend:)

To get swapping statistics
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
ubuntu@server3:~$ sar -W
(:tableend:)

Displays I/O and transfer rate statistics grouped for all block devices
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
ubuntu@server3:~$sar -b
(:tableend:)
October 30, 2011, at 10:35 PM by 117.99.109.84 -
Changed lines 20-22 from:
%blue%Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds%%

Generated the CPU utilization stats with 5 lines /every 2 seconds
to:
%red%Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds%%

%red%Generated the CPU utilization stats with 5 lines /every 2 seconds%%
Changed lines 43-44 from:
The -u option specifies our interest in the CPU subsystem.
to:
The -u option specifies our interest in the CPU subsystem.\\
Changed line 65 from:
The %iowait column specifies the amount of time the CPU spends waiting for I/O requests to complete.
to:
The %iowait column specifies the amount of time the %red%CPU spends waiting for I/O requests to complete%%.
October 30, 2011, at 10:33 PM by 117.99.109.84 -
Changed lines 71-72 from:
Additional information can be obtained by the sar -q command, which displays the run queue length, total number of processes, and the load averages for the past one, five and fifteen minutes:
to:
Additional information can be obtained by the sar -q command, which displays the run queue length,
total number of processes, and the load averages for the past one, five and fifteen minutes:
Added line 78:
----
October 30, 2011, at 10:32 PM by 117.99.109.84 -
Changed line 2 from:
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.
to:
Changed lines 6-7 from:
to:
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.
----
October 30, 2011, at 10:31 PM by 117.99.109.84 -
Deleted lines 1-3:
---
(:Google1:)
---
Added lines 3-6:
----
(:Google1:)
----
October 30, 2011, at 10:30 PM by 117.99.109.84 -
Changed line 2 from:
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.
to:
---
Added lines 4-5:
---
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.
October 30, 2011, at 10:29 PM by 117.99.109.84 -
Deleted line 0:
(:Google1:)
Deleted line 1:
Changed line 3 from:
to:
(:Google1:)
October 30, 2011, at 10:28 PM by 117.99.109.84 -
Changed lines 4-6 from:
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.\\
%blue%
sar - Collect, report, or save system activity information.%%\\
to:
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.

'''
sar - Collect, report, or save system activity information'''.
Changed lines 41-42 from:
The -u option specifies our interest in the CPU subsystem.\\
to:
The -u option specifies our interest in the CPU subsystem.
Changed lines 73-231 from:
ubuntu@server3:~$sar -q
runq-sz #Run queue length (number of processes waiting for run time).

# Check if system has free memory and no swap usage, then memory is not the problem.
ubuntu@server3:~$ %red%sar -r%red%
# To get swapping statistics
ubuntu@server3:~$ %red%sar -W%red%

'''#displays I/O and transfer rate statistics grouped for all block devices'''
ubuntu@server3:~$sar -b

#enables you to get more detailed information on a per device basis.
#It displays statistics data similar to those displayed by iostat:
ubuntu@server3:~$sar -d

#Report paging statistics.
ubuntu@server3:~$sar-B
ubuntu@server3:~$ sar -B -f /var/log/sysstat/sa09

sadf - Display data collected by sar in multiple formats

#Extract memory, swap space and network statistics from system activity file 'sa21', and display them in a format that can be ingested by a database.
sadf -d /var/log/sysstat/sa21 -- -r -n DEV

Example:
ubuntu@server3:~$ sadf -d /var/log/sysstat/sa09 -- -B
server3;600;2011-10-09 09:55:01 UTC;6.29;18.12;43.23;0.10;266.73;0.00;0.00;0.00;0.00
server3;600;2011-10-09 10:05:01 UTC;4.29;4.00;24.69;0.02;209.58;0.00;0.00;0.00;0.00

#Extract CPU statistics for processor 1 (the second processor) from current daily data file, and display them in a format
that can easily be handled by a pattern processing command.
sadf -p -P 1

pidstat - Report statistics for Linux tasks.

The pidstat command is used to monitor processes and threads currently being managed by the Linux kernel.
It can also monitor the children of those processes and threads.

#With its -d option, pidstat can report I/O statistics
ubuntu@server3:~$ pidstat -d 2
Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU)
05:28:18 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command
05:28:20 PM 6082 0.00 4.00 0.00 gnome-terminal
05:28:20 PM 6727 0.00 4.00 0.00 rsyslogd

EXAMPLES

#Display five reports of CPU statistics for every active task in the system at two second intervals.
pidstat 2 5

#Display five reports of page faults and memory statistics for PID 1643 at two second intervals.
pidstat -r -p 1643 2 5

#Display global page faults and memory statistics for all the processes whose command name includes strings.
pidstat -C "fox|bird" -r -p ALL

#Display five reports of page faults statistics at two second intervals for the child processes of all tasks in the system.
pidstat -T CHILD -r 2 5

#The following example reports CPU statistics for PID 6968 and all its threads:
ubuntu@server3:~$ pidstat -t -p 6968 3 2

Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU)
05:36:00 PM TGID TID %usr %system %guest %CPU CPU Command
05:36:03 PM 6968 - 0.33 0.00 0.00 0.33 0 soffice.bin
05:36:03 PM - 6968 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6970 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6972 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6973 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6976 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6983 0.00 0.00 0.00 0.00 0 |__soffice.bin

What is majflt/s ?

Number of major faults the system has made per second, those which have required loading a memory page from disk (post 2.5 kernels only). Page faults are not a bad thing. It's part of the way that the memory system works.

http://en.wikipedia.org/wiki/Page_fault

-r
Report page faults and memory utilization.
When reporting statistics for individual tasks, the following values are displayed:
minflt/s
Total number of minor faults the task has made per second, those which have not required loading a memory page from disk.
majflt/s
Total number of major faults the task has made per second, those which have required loading a memory page from disk.
VSZ
Virtual Size: The virtual memory usage of entire task in kilobytes.
RSS
Resident Set Size: The non-swapped physical memory used by the task in kilobytes.


ubuntu@server3:~$ sudo pidstat -r 2
[sudo] password for ubuntu:
Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU)

08:31:52 PM PID minflt/s majflt/s VSZ RSS %MEM Command
08:31:54 PM 1011 0.50 0.00 48488 3392 0.43 fail2ban-server
08:31:54 PM 1320 0.50 0.00 45168 744 0.09 udisks-daemon
08:31:54 PM 7257 147.76 0.00 15704 1068 0.13 pidstat

08:31:54 PM PID minflt/s majflt/s VSZ RSS %MEM Command
08:31:56 PM 1320 0.50 0.00 45168 744 0.09 udisks-daemon
08:31:56 PM 7257 152.50 0.00 15704 1108 0.14 pidstat

vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity.

If there are lots of runnable processes, (the Procs r vmstat column), then that means there is more to do than the CPU can manage. If runnable processes hover much over 1 or 2, a faster computer will speed up the server significantly.

ubuntu@server3:~$ vmstat 5 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 92152 34072 16144 300976 0 2 36 26 318 875 4 10 86 1
0 0 92152 34064 16152 300976 0 0 0 7 329 876 2 6 92 1
0 0 92152 34064 16160 300976 0 0 0 4 356 1020 2 4 93 1
0 0 92152 34064 16168 300980 0 0 0 4 332 896 2 4 94 0
0 0 92152 34064 16176 300980 0 0 0 8 338 949 1 3 96 0
ubuntu@server3:~$

#Monitor with watch command and vmstat, memory usage
ubuntu@server3:~$ watch vmstat -sSM

#Another userful script.
ubuntu@server3:~$ while (true); do clear; uname -n; echo ""; df -h /; echo ""; tail -5 /var/log/auth.log; echo ""; tail -5 /var/log/syslog; echo ""; vmstat 1 5; sleep 2; done


Process Monitorng

#Monitoring file handles used by a particular process -r : repeat mode
lsof -c <process name> -r
ubuntu@server3:~$ lsof -c firefox -r

Monitoring a port connections

ubuntu@server3:~$ watch -n1 'netstat -tn | grep -P :22'

dstat - a mix of vmstat, iostat, netstat, ps, sar...

This is a very powerful command line tool to gather statistics for a Linux system.

http://dag.wieers.com/home-made/dstat/

dstat - versatile tool for generating system resource statistics

EXAMPLES

Using dstat to relate disk-throughput with network-usage (eth0), total CPU-usage and system counters:
dstat -dnyc -N eth0 -C total -f 5

Checking dstat’s behaviour and the system impact of dstat:
dstat -taf --debug

Using the time plugin together with cpu, net, disk, system, load, proc and top_cpu plugins:
dstat -tcndylp --top-cpu

this is identical to
dstat --time --cpu --net --disk --sys --load --proc --top-cpu

Using dstat to relate cpu stats with interrupts per device:
dstat -tcyif
to:

(:Google1:)
(:Googlemm:)
October 30, 2011, at 10:24 PM by 117.99.109.84 -
Changed line 3 from:
\\
to:
Changed lines 12-14 from:
sudo apt-get install sysstat / yum install sysstat
to:
sudo apt-get install sysstat

yum install sysstat
Deleted lines 70-71:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
Changed lines 73-74 from:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
to:
October 30, 2011, at 10:22 PM by 117.99.109.84 -
Added line 22:
Changed line 43 from:
ubuntu@server3:~$ %red%sudo sar -u -o sardatafile 2 3%red%
to:
ubuntu@server3:~$sudo sar -u -o sardatafile 2 3
Changed lines 46-53 from:
The %iowait and %idle columns are of interest to us when doing performance analysis.\\
The %iowait column specifies the amount of time the CPU spends waiting for I/O requests to complete.\\
The
%idle column tells us how much useful work the CPU is doing.\\
A
%idle time near zero indicates a CPU bottleneck.\\
A high %iowait value indicates unsatisfactory disk performance.\\

Additional information can be obtained by the sar -q command, which displays the run queue length, total number of processes,
and the load averages for the past one, five and fifteen minutes:\\
to:
[@
buntu@server3:~$ sar -u -f sardatafile
Linux 2.6.38-8-generic (server3) 10/31/2011 _x86_64_ (1 CPU)

07:48:19 AM
CPU %user %nice %system %iowait %steal %idle
07:48:21 AM all 0.00 0.00 1.50 0.00 0.00 98.50
07:48:23 AM all 0.50 0.00 1.00 0.00 0.00 98.50
07
:48:25 AM all 1.00 0.00 0.50 7.50 0.00 91.00
Average: all 0.50 0.00 1.00 2.50 0.00 96.00
ubuntu@server3:~$
@]

[@
The %iowait and %idle columns are of interest to us when doing performance analysis.
The %iowait column specifies the amount of time the CPU spends waiting for I/O requests to complete.
The %idle column tells us how much useful work the CPU is doing.
A %idle time near zero indicates a CPU bottleneck.
A high %iowait value indicates unsatisfactory disk performance.

Additional information can be obtained by the sar -q command, which displays the run queue length, total number of processes, and the load averages for the past one, five and fifteen minutes:
Changed lines 67-68 from:
\\
to:
@]
October 30, 2011, at 10:17 PM by 117.99.109.84 -
Changed lines 22-36 from:
to:
[@
ubuntu@server3:~$ sudo sar -u 2 5
[sudo] password for ubuntu:
Linux 2.6.38-8-generic (server3) 10/31/2011 _x86_64_ (1 CPU)

07:46:06 AM CPU %user %nice %system %iowait %steal %idle
07:46:08 AM all 3.52 0.00 3.52 5.03 0.00 87.94
07:46:10 AM all 2.51 0.00 3.52 0.00 0.00 93.97
07:46:12 AM all 2.50 0.00 3.50 0.00 0.00 94.00
07:46:14 AM all 1.50 0.00 2.00 4.50 0.00 92.00
07:46:16 AM all 1.50 0.00 2.00 0.00 0.00 96.50
Average: all 2.30 0.00 2.91 1.90 0.00 92.89
ubuntu@server3:~$
@]
Changed line 60 from:
(:cellnr:
to:
(:cellnr:)
October 30, 2011, at 10:12 PM by 117.99.109.84 -
Changed line 14 from:
\\
to:
Changed line 17 from:
Generated the CPU utilization stats with 5 lines /every 2 seconds\\
to:
Generated the CPU utilization stats with 5 lines /every 2 seconds
Changed line 20 from:
ubuntu@server3:~$ %red%sudo sar -u 2 5%red%
to:
ubuntu@server3:~$ sudo sar -u 2 5
Changed lines 24-26 from:

The -o option will create an output file that contains binary data. 3 samples at two-second intervals.\\
to:
The -o option will create an output file that contains binary data. 3 samples at two-second intervals.
Changed line 27 from:
(:cellnr:
to:
(:cellnr:)
October 30, 2011, at 10:11 PM by 117.99.109.84 -
Changed line 17 from:
#Generated the CPU utilization stats with 5 lines /every 2 seconds\\
to:
Generated the CPU utilization stats with 5 lines /every 2 seconds\\
Changed lines 23-24 from:
#The -u option specifies our interest in the CPU subsystem.\\
#The -o option will create an output file that contains binary data. 3 samples at two-second intervals.\\
to:
The -u option specifies our interest in the CPU subsystem.\\

The -o option will create an output file that contains binary data. 3 samples at two-second intervals.\\
Added line 31:
October 30, 2011, at 10:09 PM by 117.99.109.84 -
Added line 6:
Added line 8:
Changed lines 15-16 from:
%blue%Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds/%blue%\\
to:
%blue%Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds%%
Added line 22:
October 30, 2011, at 10:07 PM by 117.99.109.84 -
Deleted line 6:
Changed lines 12-15 from:

'''
Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds/'''
#Generated the CPU utilization stats with 5 lines /every 2 seconds
to:
\\
%blue%
Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds/%blue%\\
#Generated the CPU utilization stats with 5 lines /every 2 seconds\\
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
Changed lines 18-20 from:
#The -u option specifies our interest in the CPU subsystem.
#The -o option will create an output file that contains binary data. 3 samples at two-second intervals.
to:
(:tableend:)
#The -u option specifies our interest in the CPU subsystem.\\
#The -o option will create an output file that contains binary data. 3 samples at two-second intervals.\\
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:
Changed lines 24-32 from:
The %iowait and %idle columns are of interest to us when doing performance analysis.
The %iowait column specifies the amount of time the CPU spends waiting for I/O requests to complete.
The %idle column tells us how much useful work the CPU is doing.
A %idle time near zero indicates a CPU bottleneck.
A high %iowait value indicates unsatisfactory disk performance.

Additional information can be obtained by the sar -q command, which displays the run queue length, total number of processes, and the load averages for the past one, five and fifteen minutes:
to:
(:tableend:)
The %iowait and %idle columns are of interest to us when doing performance analysis.\\
The %iowait column specifies the amount of time the CPU spends waiting for I/O requests to complete.\\
The %idle column tells us how much useful work the CPU is doing.\\
A %idle time near zero indicates a CPU bottleneck.\\
A high %iowait value indicates unsatisfactory disk performance.\\

Additional information can be obtained by the sar -q command, which displays the run queue length, total number of processes, and the load averages for the past one, five and fifteen minutes:\\
Changed lines 34-35 from:
ubuntu@server3:~$ %red%sar -q%red%
to:
\\
(
:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
ubuntu@server3:~$sar
-q
Changed lines 39-40 from:
to:
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:
October 30, 2011, at 10:04 PM by 117.99.109.84 -
Changed line 3 from:
--
to:
\\
Changed line 5 from:
%blue%sar - Collect, report, or save system activity information.%%
to:
%blue%sar - Collect, report, or save system activity information.%%\\
October 30, 2011, at 10:03 PM by 117.99.109.84 -
Changed lines 3-4 from:
---
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.
to:
--
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.\\
October 30, 2011, at 10:03 PM by 117.99.109.84 -
Changed line 3 from:
to:
---
Changed lines 5-6 from:
'''sar - Collect, report, or save system activity information.'''
to:
%blue%sar - Collect, report, or save system activity information.%%
Changed lines 8-9 from:
''Installing sysstat''
%red%sudo apt-get install sysstat / yum install sysstat%red%
to:
%blue%Installing sysstat%%
(:table border=1 bgcolor=#cccc99 cellspacing=0 :)
(:cellnr:)
sudo apt-get
install sysstat / yum install sysstat
(:tableend:)
October 30, 2011, at 01:05 PM by 117.192.154.209 -
Deleted line 4:
Changed lines 7-8 from:
Requirements: sysstat package.
to:
Requirements: ''sysstat package''.
Deleted line 9:
Changed line 12 from:
Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds/
to:
'''Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds/'''
Changed lines 14-15 from:
ubuntu@server3:~$ sudo sar -u 2 5
to:
ubuntu@server3:~$ %red%sudo sar -u 2 5%red%
Changed lines 18-19 from:
ubuntu@server3:~$ sudo sar -u -o sardatafile 2 3
to:
ubuntu@server3:~$ %red%sudo sar -u -o sardatafile 2 3%red%
Changed line 30 from:
ubuntu@server3:~$ sar -q
to:
ubuntu@server3:~$ %red%sar -q%red%
Changed line 34 from:
ubuntu@server3:~$ sar -r
to:
ubuntu@server3:~$ %red%sar -r%red%
Changed lines 36-38 from:
ubuntu@server3:~$ sar -W

#displays I/O and transfer rate statistics grouped for all block devices
to:
ubuntu@server3:~$ %red%sar -W%red%

'''#displays I/O and transfer rate statistics grouped for all block devices'''
October 30, 2011, at 12:33 PM by 117.192.154.209 -
Added line 3:
Changed lines 10-12 from:
'''#Installing sysstat'''

'''
sudo apt-get install sysstat / yum install sysstat'''
to:
''Installing sysstat''

%red%
sudo apt-get install sysstat / yum install sysstat%red%
October 30, 2011, at 12:31 PM by 117.192.154.209 -
Added lines 1-189:
(:Google1:)
(:Googlemm:)
Locate system bottlenecks and suggest some possible solutions to those annoying performance problems.

'''sar - Collect, report, or save system activity information.'''

Requirements: sysstat package.

'''#Installing sysstat'''

'''sudo apt-get install sysstat / yum install sysstat'''

Generate the CPU utilization report/Generated the CPU utilization stats with 5 lines /every 2 seconds/
#Generated the CPU utilization stats with 5 lines /every 2 seconds
ubuntu@server3:~$ sudo sar -u 2 5

#The -u option specifies our interest in the CPU subsystem.
#The -o option will create an output file that contains binary data. 3 samples at two-second intervals.
ubuntu@server3:~$ sudo sar -u -o sardatafile 2 3

The %iowait and %idle columns are of interest to us when doing performance analysis.
The %iowait column specifies the amount of time the CPU spends waiting for I/O requests to complete.
The %idle column tells us how much useful work the CPU is doing.
A %idle time near zero indicates a CPU bottleneck.
A high %iowait value indicates unsatisfactory disk performance.

Additional information can be obtained by the sar -q command, which displays the run queue length, total number of processes, and the load averages for the past one, five and fifteen minutes:

If you see more than one process is runnable at any given time, the the system will be busy.

ubuntu@server3:~$ sar -q
runq-sz #Run queue length (number of processes waiting for run time).

# Check if system has free memory and no swap usage, then memory is not the problem.
ubuntu@server3:~$ sar -r
# To get swapping statistics
ubuntu@server3:~$ sar -W

#displays I/O and transfer rate statistics grouped for all block devices
ubuntu@server3:~$sar -b

#enables you to get more detailed information on a per device basis.
#It displays statistics data similar to those displayed by iostat:
ubuntu@server3:~$sar -d

#Report paging statistics.
ubuntu@server3:~$sar-B
ubuntu@server3:~$ sar -B -f /var/log/sysstat/sa09

sadf - Display data collected by sar in multiple formats

#Extract memory, swap space and network statistics from system activity file 'sa21', and display them in a format that can be ingested by a database.
sadf -d /var/log/sysstat/sa21 -- -r -n DEV

Example:
ubuntu@server3:~$ sadf -d /var/log/sysstat/sa09 -- -B
server3;600;2011-10-09 09:55:01 UTC;6.29;18.12;43.23;0.10;266.73;0.00;0.00;0.00;0.00
server3;600;2011-10-09 10:05:01 UTC;4.29;4.00;24.69;0.02;209.58;0.00;0.00;0.00;0.00

#Extract CPU statistics for processor 1 (the second processor) from current daily data file, and display them in a format
that can easily be handled by a pattern processing command.
sadf -p -P 1

pidstat - Report statistics for Linux tasks.

The pidstat command is used to monitor processes and threads currently being managed by the Linux kernel.
It can also monitor the children of those processes and threads.

#With its -d option, pidstat can report I/O statistics
ubuntu@server3:~$ pidstat -d 2
Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU)
05:28:18 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command
05:28:20 PM 6082 0.00 4.00 0.00 gnome-terminal
05:28:20 PM 6727 0.00 4.00 0.00 rsyslogd

EXAMPLES

#Display five reports of CPU statistics for every active task in the system at two second intervals.
pidstat 2 5

#Display five reports of page faults and memory statistics for PID 1643 at two second intervals.
pidstat -r -p 1643 2 5

#Display global page faults and memory statistics for all the processes whose command name includes strings.
pidstat -C "fox|bird" -r -p ALL

#Display five reports of page faults statistics at two second intervals for the child processes of all tasks in the system.
pidstat -T CHILD -r 2 5

#The following example reports CPU statistics for PID 6968 and all its threads:
ubuntu@server3:~$ pidstat -t -p 6968 3 2

Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU)
05:36:00 PM TGID TID %usr %system %guest %CPU CPU Command
05:36:03 PM 6968 - 0.33 0.00 0.00 0.33 0 soffice.bin
05:36:03 PM - 6968 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6970 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6972 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6973 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6976 0.00 0.00 0.00 0.00 0 |__soffice.bin
05:36:03 PM - 6983 0.00 0.00 0.00 0.00 0 |__soffice.bin

What is majflt/s ?

Number of major faults the system has made per second, those which have required loading a memory page from disk (post 2.5 kernels only). Page faults are not a bad thing. It's part of the way that the memory system works.

http://en.wikipedia.org/wiki/Page_fault

-r
Report page faults and memory utilization.
When reporting statistics for individual tasks, the following values are displayed:
minflt/s
Total number of minor faults the task has made per second, those which have not required loading a memory page from disk.
majflt/s
Total number of major faults the task has made per second, those which have required loading a memory page from disk.
VSZ
Virtual Size: The virtual memory usage of entire task in kilobytes.
RSS
Resident Set Size: The non-swapped physical memory used by the task in kilobytes.


ubuntu@server3:~$ sudo pidstat -r 2
[sudo] password for ubuntu:
Linux 2.6.38-8-generic (server3) 10/09/2011 _x86_64_ (1 CPU)

08:31:52 PM PID minflt/s majflt/s VSZ RSS %MEM Command
08:31:54 PM 1011 0.50 0.00 48488 3392 0.43 fail2ban-server
08:31:54 PM 1320 0.50 0.00 45168 744 0.09 udisks-daemon
08:31:54 PM 7257 147.76 0.00 15704 1068 0.13 pidstat

08:31:54 PM PID minflt/s majflt/s VSZ RSS %MEM Command
08:31:56 PM 1320 0.50 0.00 45168 744 0.09 udisks-daemon
08:31:56 PM 7257 152.50 0.00 15704 1108 0.14 pidstat

vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity.

If there are lots of runnable processes, (the Procs r vmstat column), then that means there is more to do than the CPU can manage. If runnable processes hover much over 1 or 2, a faster computer will speed up the server significantly.

ubuntu@server3:~$ vmstat 5 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 92152 34072 16144 300976 0 2 36 26 318 875 4 10 86 1
0 0 92152 34064 16152 300976 0 0 0 7 329 876 2 6 92 1
0 0 92152 34064 16160 300976 0 0 0 4 356 1020 2 4 93 1
0 0 92152 34064 16168 300980 0 0 0 4 332 896 2 4 94 0
0 0 92152 34064 16176 300980 0 0 0 8 338 949 1 3 96 0
ubuntu@server3:~$

#Monitor with watch command and vmstat, memory usage
ubuntu@server3:~$ watch vmstat -sSM

#Another userful script.
ubuntu@server3:~$ while (true); do clear; uname -n; echo ""; df -h /; echo ""; tail -5 /var/log/auth.log; echo ""; tail -5 /var/log/syslog; echo ""; vmstat 1 5; sleep 2; done


Process Monitorng

#Monitoring file handles used by a particular process -r : repeat mode
lsof -c <process name> -r
ubuntu@server3:~$ lsof -c firefox -r

Monitoring a port connections

ubuntu@server3:~$ watch -n1 'netstat -tn | grep -P :22'

dstat - a mix of vmstat, iostat, netstat, ps, sar...

This is a very powerful command line tool to gather statistics for a Linux system.

http://dag.wieers.com/home-made/dstat/

dstat - versatile tool for generating system resource statistics

EXAMPLES

Using dstat to relate disk-throughput with network-usage (eth0), total CPU-usage and system counters:
dstat -dnyc -N eth0 -C total -f 5

Checking dstat’s behaviour and the system impact of dstat:
dstat -taf --debug

Using the time plugin together with cpu, net, disk, system, load, proc and top_cpu plugins:
dstat -tcndylp --top-cpu

this is identical to
dstat --time --cpu --net --disk --sys --load --proc --top-cpu

Using dstat to relate cpu stats with interrupts per device:
dstat -tcyif
Edit - History - Print - Recent Changes - Search
Page last modified on August 15, 2012, at 01:40 PM