The Unix Environment
Server monitoring can be done using a number of utilities. Those utilities are top, sar (system activitiy reporter),sadc, vmstat, the Unix (w)atch command, iostat, etc. Note that the results of these processes can parsed and placed into database tables on a scheduled basis for later statistical analysis.
top
last pid: 15713; load averages: 0.01, 0.01, 0.01 58 processes: 57 sleeping, 1 on cpu CPU states: 99.7% idle, 0.0% user, 0.3% kernel, 0.0% iowait, 0.0% swap Memory: 1024M real, 34M free, 598M swap in use, 2263M swap free PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND 15713 oracle 1 58 0 2136K 1664K cpu0 0:00 0.08% top 1732 build 26 58 0 195M 60M sleep 5:45 0.05% java 14478 ted 25 58 0 182M 40M sleep 1:08 0.01% java 15711 root 1 58 0 2152K 1728K sleep 0:00 0.00% sendmail 127 root 1 58 0 3208K 2672K sleep 0:38 0.00% in.named 235 root 1 58 0 1752K 1056K sleep 0:20 0.00% sshd1 9259 oracle 1 58 0 9144K 4544K sleep 0:18 0.00% tnslsnr 1609 build 20 0 0 167M 18M sleep 0:01 0.00% java 179 root 8 52 0 2376K 1944K sleep 0:01 0.00% nscd 15676 root 1 38 0 1568K 1264K sleep 0:00 0.00% in.rlogind 143 daemon 4 44 0 2184K 1568K sleep 0:00 0.00% statd 138 root 1 48 0 2008K 1584K sleep 0:00 0.00% inetd 231 root 1 48 0 2096K 1552K sleep 0:00 0.00% sendmail
sar and sadc
The sar utility is the System Activity Reporter. sar -u shows CPU activity, sar -w shows swapping activity,sar -gp shows paging activity, sar -r shows unused memory and sar -b show buffer activity. All the examples below execute five times every three seconds, ie. sar [-u|-w|-gp|-r|-b] t n where t determines how often (in seconds) a report is made and n determines the number of seconds between each report.
The sadc utility is used to collect server statistics. As opposed to the sar utility which displays system activity, the sadc utility is used to sample, save and process system activity statistics. These statistics are in general CPU utilization, buffers usage, disk I/O activity, TTY device activity, switching and system-call activity, file-access, queue activity, inter-process communications, and paging. The sar, sa1 and sa2 utilities are packaged in sar. sadc packages can be used to schedule cron jobs in the crontab file for collection and reporting of system activity statistics.
# sar -u 3 5 15:25:05 %usr %sys %wio %idle 15:25:08 1 0 1 98 15:25:11 1 2 9 88 15:25:14 0 0 1 99 15:25:17 0 0 1 99 15:25:20 1 0 1 98 Average 1 1 2 96
# sar -w 3 5 15:27:13 swpin/s bswin/s swpot/s bswot/s pswch/s 15:27:16 0.00 0.0 0.00 0.0 146 15:27:19 0.00 0.0 0.00 0.0 170 15:27:22 0.00 0.0 0.00 0.0 167 15:27:25 0.00 0.0 0.00 0.0 187 15:27:28 0.00 0.0 0.00 0.0 163 Average 0.00 0.0 0.00 0.0 167
# sar -gp 3 5 15:28:14 pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf atch/s pgin/s ppgin/s pflt/s vflt/s slock/s 15:28:17 1.33 1.33 1.33 0.00 0.00 1.33 0.00 0.00 1.67 1.67 0.00 15:28:20 1.33 1.33 1.33 0.00 0.00 1.33 0.00 0.00 0.00 0.00 0.00 15:28:23 1.33 1.33 1.33 0.00 0.00 1.33 0.00 0.00 0.00 0.00 0.00 15:28:26 1.67 1.67 1.67 0.00 0.00 1.33 0.00 0.00 0.00 1.00 0.00 15:28:29 1.33 1.33 1.33 0.00 0.00 1.33 0.00 0.00 0.00 0.00 0.00 Average 1.40 1.40 1.40 0.00 0.00 1.33 0.00 0.00 0.33 0.53 0.00
# sar -r 3 5 15:28:56 freemem freeswap 15:28:59 2097 3590815 15:29:02 2110 3614833 15:29:05 2089 3578925 15:29:08 2105 3602784 15:29:11 2113 3614866 Average 2103 3600390
# sar -b 3 5 15:29:31 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s 15:29:34 0 0 100 0 0 100 0 0 15:29:37 0 0 100 0 0 100 0 0 15:29:40 0 1 100 0 0 100 0 0 15:29:43 0 1 100 0 0 100 0 0 15:29:46 0 0 100 0 0 100 0 0 Average 0 0 100 0 0 100 0 0
vmstat
The vmstat utility is used to report virtual memory statistics.
# vmstat procs memory page disk faults cpu r b w swap free re mf pi po fr de sr m0 m1 m1 s0 in sy cs us sy id 0 0 0 54656 93832 0 7 2 1 3 0 0 1 1 1 1 423 74 92 0 0 99
# vmstat -cisS interrupt total rate -------------------------------- clock 19428784 100 hmec0 830039 4 hmec1 0 0 -------------------------------- Total 20258823 104
The Unix (w)atch Command
# w 4:28am up 2 day(s), 6 hr(s), 1 user, load average: 0.00, 0.00, 0.01 User tty login@ idle JCPU PCPU what oracle pts/2 4:08am w
iostat
iostat is used to report I/O statistics.
# iostat
tty md0 md1 md10 sd0 cpu
tin tout kps tps serv kps tps serv kps tps serv kps tps serv us sy wt id
0 5 17 1 17 17 1 21 23 1 26 17 1 17 0 0 1 99
# iostat -xtc
extended device statistics tty cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b tin tout us sy wt id
md0 0.1 1.0 3.4 13.5 0.0 0.0 17.1 0 1 0 5 0 0 1 99
md1 0.1 1.0 3.3 13.5 0.0 0.0 20.5 0 1
md10 0.3 1.0 6.7 16.3 0.0 0.0 26.4 1 1
sd0 0.1 1.0 3.4 13.6 0.0 0.0 17.1 0 1
sd1 0.1 1.6 3.3 13.9 0.0 0.0 21.2 0 1
sd6 0.0 0.0 1.4 0.0 0.0 0.0 26.0 0 0
nfs1 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0
# iostat -xnp
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.1 1.0 3.4 13.5 0.0 0.0 0.0 17.1 0 1 md0
0.1 1.0 3.3 13.5 0.0 0.0 0.0 20.5 0 1 md1
0.3 1.0 6.7 16.3 0.0 0.0 5.6 20.8 1 1 md10
0.1 1.0 3.4 13.6 0.0 0.0 0.0 17.0 0 1 c0t0d0
0.1 1.0 3.4 13.5 0.0 0.0 0.0 17.1 0 1 c0t0d0s0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 13.3 0 0 c0t0d0s1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0s2
0.1 1.6 3.3 13.9 0.0 0.0 0.0 21.2 0 1 c0t1d0
0.1 1.0 3.3 13.5 0.0 0.0 0.0 20.5 0 1 c0t1d0s0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t1d0s1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t1d0s2
0.0 0.7 0.0 0.3 0.0 0.0 0.0 22.5 0 1 c0t1d0s7
0.0 0.0 1.4 0.0 0.0 0.0 0.0 25.9 0 0 c0t6d0
Memory and CPUs
Display the top CPU users with the command ps -ef | sort +6 | tail -10.
# ps -ef | sort +6 | tail -10
root 28066 1 0 Jun 18 ? 2:25 /usr/sbin/nscd
root 23790 1 0 Jun 13 ? 8:31 /usr/local/sbin/sshd
root 3 0 1 May 18 ? 733:26 fsflush
network 9206 9196 0 Aug 09 pts/2 0:00 -sh
network 9237 9206 0 Aug 09 pts/2 0:00 tcsh
Display RAM size with the command prtconf | grep “Memory size”.
prtconf|grep -i "Memory Size" Memory size: 1024 Megabytes
Display allocated memory and semaphores using ipcs -mqs and ipcs -pmb as shown below. Use the ipcrm -m to deallocate memory segments. Execute oradebug ipc in svrmgrl to write Oracle used semaphores into a trace file.
# ipcs -mqs IPC status from <running system> as of Sat Jul 21 03:53:20 2001 T ID KEY MODE OWNER GROUP Message Queues: Shared Memory: m 300 0x5000a417 --rw-r--r-- root other m 101 0x7f8207c8 --rw-rw---- oracle dba Semaphores: s 458752 0x061761b5 --ra-ra---- oracle dba s 65537 0x061761b6 --ra-ra---- oracle dba s 65538 0x061761b7 --ra-ra---- oracle dba s 65539 0x061761b8 --ra-ra---- oracle dba s 65540 0x061761b9 --ra-ra---- oracle dba
# ipcs -pmb IPC status from <running system> as of Sat Jul 21 03:53:46 2001 T ID KEY MODE OWNER GROUP SEGSZ CPID LPID Shared Memory: m 300 0x5000a417 --rw-r--r-- root other 68 23372 23372 m 101 0x7f8207c8 --rw-rw---- oracle dba 249061376 7379 7367
Display the number of CPUs using the psrinfo -v command.
# psrinfo -v
Status of processor 0 as of: 07/21/01 03:58:02
Processor has been on-line since 07/18/01 22:28:48.
The sparc processor operates at 450 MHz,
and has a sparc floating point processor.
Status of processor 2 as of: 07/21/01 03:58:02
Processor has been on-line since 07/18/01 22:28:49.
The sparc processor operates at 450 MHz,
and has a sparc floating point processor.
Display system details using the uname -X command.
# uname -X System = SunOS Node = <node> Release = 5.6 KernelID = Generic_105181-26 Machine = sun4u BusType = Serial = Users = OEM# = 0 Origin# = 1 NumCPU = 2
Find the number CPUs and the speed of those CPUs or memory using dmesg | grep -i [cpu | mem].
# dmesg | grep -i cpu cpu0: SUNW,UltraSPARC-II (upaid 0 impl 0x11 ver 0xa0 clock 450 MHz) cpu1: SUNW,UltraSPARC-II (upaid 2 impl 0x11 ver 0xa0 clock 450 MHz) cpu2: SUNW,UltraSPARC-II (upaid 2 impl 0x11 ver 0xa0 clock 450 MHz) cpu3: SUNW,UltraSPARC-II (upaid 2 impl 0x11 ver 0xa0 clock 450 MHz)
# dmesg | grep -i mem mem = 1048576K (0x40000000) avail mem = 1039376384
System Log Files
Unix system log files are in /var/log, /var/adm and /var/cron. /var/log contains numbered sequences of /var/log/syslog[.n]. Examine the system logs using the tail command because log files can be large. /var/adm/messages[.n] files contain mostly login record histories. See Understanding system log files on a Solaris 2.x operating system for further details.
Related Postings:
Monitoring Oracle with Solaris Scripting
Monitoring Oracle Standby with Solaris Scripting