atsar

system activity report

atsar [ -flags ] seconds [ n ]

atsar [ -flags ] [ -s hh:mm ] [ -e hh:mm ] [ -i sec ] [ -f file | -n dd ]

Delivers statistics.

Samples cumulative activity counters n intervals of t seconds.

When no sampling interval specified, data is extracted from a previously recorded file, either :

  • the one specified by -f default: daily activity data file /var/log/atsar/atsadd for the current day dd (day of month).
  • -n can be used to specify the day of the month

    -i restricts records to those at sec second intervals.

    -S don't Supress times stamps if multiple lines are from same time period (easier for post-processing).
    -A All possible options.
    -u CPU utilization (average and per cpu).
    -P processes and load-averages.
    -d Limited utilization of disks
    -D utilization of disks and disk-partitions.
    -r memory and swap
    -p paging and swapping
    -I interrupts per second
    -v utilization of kernel-tables
    -y utilization of tty's (serial interfaces)
    -l network interfaces
    -L errors for network-interfaces
    -w IP network traffic
    -W errors for IP traffic
    -m -kv6 General ICMP layer activity
    -M -Kv6 Per-type ICMP layer activity
    -t -gv6 network traffic
    -T -Gv6 errors for traffic
    -U -hv6 UDP network traffic
    -jv6 TCPv6 socket-utilization
    -N NFS server and client requests
    -E errors for NFS server and client-requests
    -V behaviour of NFS-server
    -R RPC-calls for NFS-requests. A transfer-rate is given per type of RPC-request (% total num- ber of RPC-requests).
    -F Optional the ftp-traffic. This flag only produces relevant output if the script atsaftp is activated with regular intervals (by the script atsa1 ). Note that the names of the FTP-logfiles have to be specified in the /etc/atsar.conf configuration-file.
    -H Optional the http-traffic. This flag only produces relevant output if the script atsahttp is activated with regular intervals (by the script atsa1 ). Note that the names of the HTTP-logfiles have to be specified in the /etc/atsar.conf configuration-file.

    OUTPUT DESCRIPTION

    Depending on the flag, a columns with output-values are produced. The values are mostly presented as a number of events per second.
    -u per cpu:
    usr% user-mode with a nice-value of zero (default) or negative (i.e. higher priority).
    nice% user-mode with a nice-value larger than zero (i.e. lower priority ).
    sys% system-mode (kernel-text) for all active processes.
    A high percentage indicates a lot of system calls being issued. Interrupt-handling is also part of this percentage.
    irq% for interrupt-handling .
    softirq% for soft interrupt-handling .
    idle% unused because all processes are in a wait-state but not waiting for disk-I/O).
    wait% unused At least one of processes in wait-state for disk-I/O
    -P
    pswtch/s process-switches (aka context-switches) per second on all cpu's. A process-switch occurs at the moment that the active process (i.e. the process using a cpu) enters a wait-state or has used its time-slice completely; another process will then be chosen to use the cpu.
    runq processes which are currently waiting for CPU-scheduling (including the processes which are currently active on a CPU).
    nrproc Total processes present
    lavg1 lavg5 ,lavg15 Load-average reflecting the average processes in the run-queue during the mm minute(s).
    -d per active physical disk:
    device Disk-drive number (major-minor).
    read/s, rdKb/s
    write/s wrKb/s
    rdwr/s total of read/s + write/s)
    -D per used partition of a physical disk:
    partition Diskpartition) name.
    busy Busy-% the physical disk (i.e. the portion of time that the device was busy handling requests). only for entire physical disk.
    read/s Kbyt/r
    write/sKbyt/w
    avque Average number of disk-requests outstanding during the time that the disk is busy. for partitions which represent the entire physical disk.
    avserv Average milliseconds needed by a request on this physical disk (seek, latency and data-transfer) only entire physical disk.
    -r memory and swap utilization snapshot
    memtot, memfree Total, Available
    buffers, cached metadata-blocks, cache data-blocks
    slabmem dynamically allocated memory by the kernel
    swptot, swpfre Total, Available swap
    atsar -r |head
    
    Linux  pi93graf  4.14.98+  #1200 Tue Feb 12 20:11:02 GMT 2019  armv6l  04/11/2019
    
    21:16:16  memtot memfree buffers   cached  slabmem      swptot swpfree  _mem_
    21:17:27    433M     91M     21M     128M      13M       2047M   2047M
    21:17:32    433M     91M     21M     128M      13M       2047M   2047M
    21:20:01     linux restarts
    21:30:02    433M    155M     23M     103M      13M       2047M   2047M
    
    
    atsar -r  |sed "s/M/ /g"|uniq --skip-chars=8 |head -n50  # remove 'M' for clarity and uniquify by skipping the time  
    
    Linux  pi93graf  4.14.98+  #1200 Tue Feb 12 20:11:02 G T 2019  armv6l  05/20/2019
    
    00:00:01  memtot memfree buffers   cached  slabmem      swptot swpfree  _mem_
    00:00:16    433      39       8      155       14        2047    2017 
    00:00:46    433      39       8      156       14        2047    2017 
    00:01:16    433      38       8      156       14        2047    2017 some identical entries skipped
    00:01:31    433      37       8      156       14        2047    2017 
    00:02:16    433      36       8      157       14        2047    2017 
    00:03:16    433      34       8      158       14        2047    2017 
    
    
    atsar -r -s 12:00 -e 12:50  |sed "s/M/ /g"|uniq --skip-chars=8
    
    Linux  pi93graf  4.19.42+  #1219 Tue  ay 14 21:16:38 BST 2019  armv6l  06/28/2019
    
    12:00:02  memtot memfree buffers   cached  slabmem      swptot swpfree  _mem_
    12:01:02    432      33       6      122       15        2047    1997 
    12:03:02    432      23      11      120       16        2047    1998 
    12:10:01    432      27      11       79       16        2047    1998 
    12:11:01    432      27      11       80       16        2047    1998 
    12:12:01    432      30      11       81       16        2047    1998 
    12:13:01    432      29      11       82       16        2047    1998 
    12:20:01    432      64      12       87       16        2047    1998 
    12:21:01    432      61      12       87       16        2047    1998 
    12:22:01    432      61      12       88       16        2047    1998 
    12:23:01    432      60      12       88       16        2047    1998 
    12:30:01    432      54      12       93       16        2047    1998 
    12:31:01    432      51      12       94       16        2047    1998 
    12:32:01    432      46      12       94       16        2047    1998 
    12:33:01    432      46      12       95       16        2047    1998 
    12:40:01    432      43      12      100       16        2047    1998 
    12:41:01    432      44      12       99       16        2047    1998 
    12:42:01    432      43      12       99       16        2047    1998 
    12:43:01    432      43      12      100       16        2047    1998 
    12:47:40     linux restarts
    12:50:02    432      80      17      191       15        2047    2047 
    
    
    
    
    -p frequency of paging and swapping:
    pagein/s pageout/s implies normal data-access (so not necessarily indicating lack of memory).
    swapin/s swapout/s
    fork/s new processes started Swapping is copying the entire process address space, or at least , the non-shareable-text data segment .
    Paging is copying one or more pages of the address space. Paging is also requires per-page memory hardware attributes (dirty, no access, read-only, read-write, no execute, executable), and various virtual memory management tricks like demand-zero-filled pages, copy-on-write pages, paging requires page-granularity virtual memory page table hardware, which map virtual pages to the physical address, and includes flags like valid, read, write, written etc. The hardware must trigger a page fault exception if a vitrual page is unallocated. The OS then makes page accessible and/or schedule an I/O to load it to physical RAM.
    
    atsar -s 12:25 -e 12:40 -p
    
    Linux  pi93graf  4.14.98+  #1200 Tue Feb 12 20:11:02 GMT 2019  armv6l  05/24/2019
    
    12:25:01   pagein/s pageout/s     swapin/s swapout/s         fork/s      _page_
    12:25:16       0.00     18.67         0.00      0.00           0.00
    12:25:31       0.27     20.27         0.00      0.00           0.00
    12:25:46       0.00     12.80         0.00      0.00           0.00
    12:26:01    1953.60     36.80         0.00      2.13           3.07
    12:26:16       0.27     21.07         0.00      0.00           0.00
    12:26:31       0.00     15.20         0.00      0.00           0.00
    12:26:46       0.00     26.67         0.00      0.00           0.00
    12:27:01      22.13     10.40         0.00      0.00           0.00
    12:27:16     944.80    140.00         0.00      1.87           0.73
    12:27:31    1173.07      9.07         0.00      0.00           0.00
    12:27:46    1944.27     14.93         0.00      0.00           0.60
    12:28:01   15548.53    206.40         0.07     45.40           0.07
    12:28:16   19104.00     30.93         0.00      7.27           0.00
    12:28:31   18636.53     54.93         0.00     12.93           0.00
    12:28:46   18610.13     47.47         0.00      9.47           0.00
    …
    12:38:29   11808.80      9.07         0.13      0.13           0.00
    12:38:44    1210.67     22.67         0.00      0.00           0.53
    12:38:59     173.60     24.00         0.00      0.00           0.00
    12:39:14     266.67     25.87         0.00      0.00           0.00
    12:39:29     298.13     13.60         0.00      0.00           0.00
    12:39:44       6.13     12.27         0.00      0.00           0.00
    12:39:59     148.27     26.67         0.00      0.00           0.00
    12:40:01     472.00      0.00         0.00      0.00          14.00
    12:40:16       4.27     23.73         0.00      0.00           0.20
    12:40:31     157.87     12.80         0.00      0.00           0.00
    12:40:46      16.00     22.40         0.00      0.00           0.00
    
    
    
    
    atsar -p |head
    
    Linux  pi93graf  4.14.98+  #1200 Tue Feb 12 20:11:02 GMT 2019  armv6l  04/11/2019
    
    21:16:16   pagein/s pageout/s     swapin/s swapout/s         fork/s      _page_
    21:17:27       0.45     22.48         0.00      0.00           0.99
    21:17:32       0.00     32.80         0.00      0.00           0.00
    21:20:01     linux restarts
    21:30:02     581.67    185.21         0.00      0.00           3.03
    21:30:17       0.00     19.73         0.00      0.00           0.13
    21:30:32       0.00     25.33         0.00      0.00           0.00
    
    
    
    -I interrupt-frequency, reported per irq per cpu.
    iq.. interrupts per second for this interrupt-type. /proc/interrupts shows device
    -v utilization of limited kernel-resources: current and max
    superb-sz super-blocks which is maintained by the kernel for mounted filesystems.
    inode-sz incore-inodes maintained by the kernel. One entry is needed for every disk-file which is currently open (at least once).
    file-sz open-file entries maintained by the kernel. One entry is needed for every open-request for a disk-file. If a file is currently opened by five processes, one incore-inode and five open-file entries.
    dquota-sz occupation of the collection of disk-quota entries.
    flock-sz file-locks.
    -y utilization of tty's:
    port port-number
    xmit/s recv/s interrupts
    frer/s framing-errors
    parer/s parity-errors
    ovrun/s overrun-errors
    brk/s breaks

    Communications

    -l utilization of network-interfaces:
    inpck/s tpck/s packets
    inbyt/s otbyt/s bytes
    incmpr/s otcmpr/s compressed packets (SLIP).
    inmcst/s multicast-packets received from this interface
    iface Name of the interface.
    -L failures for network-interfaces:
    inerr/s bad packets received
    oterr/s packet-transmit problems
    coll/s collisions encountered while transmitting packets.
    indrop/s received packets dropped due to lack of buffer-space
    otdrop/s transmitted packets dropped due to lack of buffer-space
    infram/s frame alignment-errors on received packets.
    otcarr/s carrier-errors on transmitted packets.
    iface Name
    -w utilization of the IPv4-layer (formal snmp-names between brackets):
    inrecv/s input received from interfaces , including those received in error (ipInReceives).
    outreq/s local higher-layer protocols supplied to IP in requests for transmission (ipOutRequests).
    indeliver/s input succesfully delivered to higher protocol-layers (ipInDelivers).
    forward/s input for which this was not their final destination, attempted to forward (ipForwDatagrams).
    reasmok/s reassembled (ipReasmOKs).
    fragcreat/s fragments created (ipFragCreates).
    -W discards in the IPv4-layer
    indsc/s input discards for lack of buffer space (ipInDiscards).
    hder/s header error (ipInHdrErrors).
    ader/s address destination not valid to be received (ipInAddrErrors).
    unkp/s unknown or unsupported protocol inbound (ipInUnknownProtos).
    ratim/s reassembly timeout-situations while other fragments were expected (ipReasmTimeout).
    rfail/s reassembly algorithm (ipReasmFails).
    otdsc/s output for lack of buffer space (ipOutDiscards).
    nort/s no route found (ipOutNoRoutes).
    -t TCP-layer segments, ports
    insegs/s incomming segments, including those in error (tcpInSegs).
    otsegs/s output segments, excluding those containing only retransmitted octets (tcpOutSegs).
    actopen/s active opens (tcpActiveOpens).
    pasopen/s passive opens (tcpPassiveOpens).
    nowopen connections open now (snapshot), ESTABLISHED or CLOSE-WAIT (tcpCurrEstab).
    socknow sockets now open (snapshot).
    sockmax Maximum parallel ever open.
    -T failures which were detected in the TCP-layer
    inerr/s received segments per second received in error (tcpInErrs).
    retrans/s retransmitted segments per second (tcpRetransSegs).
    attfail/s failed connection attempts per second that have occurred at this entity (tcpAttemptFails).
    estabreset/s resets per second that have occurred at this entity (tcpEstabResets).
    outreset/s transmitted segments per second containing the RST flag (tcpOutRsts).
    -U utilization of UDPv4-layer
    indgram/s UDP datagrams per second delivered to UDP users (udpInDatagrams),
    otdgram/s UDP datagrams transmitted per second from this entity (udpOutDatagrams),
    inerr/s incomming UDP datagrams that could not be delivered for reasons other than the lack of application at the destination port (udpInErrors).
    noport/s received UDP datagrams without application at the destination port (udpNoPorts).
    socknow sockets now open (snapshot).
    sockmax Maximum parallel UDP-sockets ever open.
    -m ICMPv4-layer and some information per type of ICMP-message
    intot/s received at this entity (icmpInMsgs).
    outtot/s transmitted from this entity (icmpOutMsgs).
    inecho/s incomming Echo (request)s (icmpInEchos).
    inerep/s incomming Echo-Replys (icmpInEchoReps).
    otecho/s output Echo (request)s (icmpOutEchos).
    oterep/s output Echo-Replys (icmpOutEchoReps).
    -M incomming and output messages with errors
    ierr/s incomming received but determined to have ICMP-specific errors (icmpInErrors).
    isq/s incomming Source Quench messages received (icmpInSrcQuenchs).
    ird/s incomming Redirect messages received (icmpInRedirects).
    idu/s incomming Destination Unreachable messages received (icmpInDestUnreachs).
    ite/s incomming Time incomming Exceeded received (icmpOutTimeExcds).
    oerr/s output with ICMP-specific errors (icmpOutErrors).
    osq/s output Source Quench (icmpOutSrcQuenchs).
    ord/s output Redirect (icmpOutRedirects).
    odu/s output Destination Unreachable (icmpOutDestUnreachs).
    ote/s output Time Exceeded (icmpOutTimeExcds).
    -g utilization of the IPv6-layer
    inrecv/s input datagrams including those in error (ipv6IfStatsInRe- ceives).
    outreq/s output requests for transmission local higher-layer protocols supplied (ipv6IfStatsOutRequests). does not include forwarded datagrams.
    inmc/s incomming multicast packets received (ipv6IfStatsInMcastPkts).
    outmc/s output multicast packets transmitted (ipv6IfStatsOutMcastPkts).
    indeliv/s incomming delivers including ICMP (ipv6IfStatsInDelivers).
    reasmok/s reassembled OK(ipv6IfStatsReasmOKs).
    fragcre/s fragments generated (ipv6IfStatsOutFragCreates).
    -G failures detected at the IPv4-layer
    indsc/s input discards which no problems for lack of buffer space (ipv6IfStatsInDiscards).
    hder/s Header errors (ipv6IfStatsInHdrErrors).
    ader/s address destination not valid to be received by this entity (ipv6IfStatsInAddrErrors).
    unkp/s locally-addressed discards because of an unknown or unsupported protocol (ipv6IfStatsInUnknownProtos).
    ratim/s reassembly timeout-situations while other fragments were expected
    rfail/s reassembly failures (ipv6IfStatsReasmFails).
    otdsc/s outputs discarded for lack of buffer space (ipv6IfStatsOutDiscards).
    nort/s no route could be found (ipv6IfStatsInNoRoutes).
    -j utilization of the TCPv6-sockets:
    socknow currently open (snapshot).
    sockmax Maximum parallel ever open.
    -h utilization of UDPv6-layer
    indgram/s incomming delivered (udpInDatagrams),
    otdgram/s transmitted (udpOutDatagrams),
    inerr/s not be delivered for reasons other than the lack of an application at the destination port (udpInErrors).
    noport/s received datagrams for with no application at the destination port (udpNoPorts).
    socknow sockets now open (snapshot).
    sockmax Maximum parallel sockets ever open
    -k ICMPv6-layer per type of ICMP-message
    intot/s,outtot/s messages in, out (any type)
    inerr/s with ICMP-specific errors, such as bad ICMP checksums, bad length, etc
    innsol/s,otnsol/s Neighbor Solicit in, out
    innadv/s,otnadv/s Neighbor Advertisement in, out
    -K ICMP messages
    iecho/s incomming Echo (request) (ipv6IfIcmpInEchos).
    ierep/s incomming Echo-Reply (ipv6IfIcmpInEchoReplies).
    oerep/s output Echo-Reply (ipv6IfIcmpOutEchoReplies).
    idu/s incomming Destination Unreachable (ipv6IfIcmpInDestUnreachs).
    odu/s output Destination Unreachable (ipv6IfIcmpOutDestUnreachs).
    ird/s incomming redirects (ipv6IfIcmpInRedirects).
    ord/s output Redirect transmitted (ipv6IfIcmpOutRedirect).
    ite/s incomming Exceeded received (ipv6IfIcmpInTimeExcds).
    ote/s Oputput Time Exceeded (ipv6IfIcmpOutTimeExcds).
    -N utilization of NFS-layer, as client as well as server:
    svrpc/s RPC-requests handled
    clrpc/s RPC-requestsNFS-client
    sudp/s requests handled via UDP
    stcp/s requests handled via TCP
    stcon/s TCP-connections established
    cudp/s requests issued via UDP by client
    ctcp/s requests issued via TCP by client
    ctcon/s TCP-connections established by client
    -E failures detected in the NFS-layer:
    svbadfmt/s server RPC-requests received by the system with a
    bad format (version-number if RPC-packet not recognized).
    svbadauth/s bad authorization.
    svbadclnt/s Meaning unknown.
    clretrans/s client request retransmitted by the due to a timeout.
    clauthrefresh/s client credential refreshs due to rejection by server.
    -V NFS-server:
    rchit/s hits on the reply-cache .
    rcmiss/s missed
    %hit Hit-ratio
    fhstale/s stale-errors for file-handles.
    iord/s Number of bytes returned to read-requests .
    iowr/s bytes passed in write-requests per second.
    thrnow avaliable threads (nfsd-servers).
    tlast/s Number of times that the last available thread is activated per second.
    -R type of RPC-calls used by the NFS-layer (client as well as server).
    per RPC-type the usage-percentage is shown.
    gat Get Attribute.
    sat Set Attribute.
    lku Lookup.
    rln Read Link.
    rd Read.
    wrc Write Cache
    wr Write.
    cre Create.
    rm Remove.
    rnm Rename.
    lnk Link.
    sln Symbolic Link.
    mkd Mkdir.
    rmd Rmdir.
    rdd Read Directory.
    fst Statfs.
    -F FTP-traffic. atsaftp should be installed and called by atsa1 with intervals. ; only when viewing long-term counters from a history-file.
    xfers/s FTP-transfers per second.
    kbytes/s Kbytes transferred during the interval.
    avg_time avg_kbytes Average per transfer.
    direct transmitted (output) or received (input)
    name Symbolic name (as specified in the /etc/atsar.conf file) of the xfer-log
    -H Apache HTTP-traffic. atsahttp by the script atsa1 with regular intervals.
    only when viewing long-term counters from a history-file.
    head/s get/s put/s post/s delete/s
    name Symbolic name (as specified in the /etc/atsar.conf file) of the access-log file from which the usage-info is retrieved.

    EXAMPLES

    To see today's processor-activity so far: atsar

    To watch TCP activity for ten minutes (10 samples with sixty seconds interval):: atsar -t 60 10

    FILES

    /var/log/atsar/atsadd

    Daily data file, where dd are digits representing the day of the month.


    atsadc, atsa1, atsaftp, atsahttp

    counter-collection

    atsadc [ seconds ] [ ofile ] # Data collector

    atsa1 [ seconds ] [#]

    atsaftp

    atsahttp

    atsadc samples system data and writes to ofile or standard output.
    If seconds and # are omitted, a reset-record is written. Used when booting to a multi-user state, to mark the time at which the counters restart from zero. The reset-mark can be added by adding :

    /usr/local/bin/atsadc /var/log/atsar/atsa`date +%d` to /etc/rc.d/init.d/atsar # not included in RaspberryPi distribution.

    atsa1 collects and stores data in /var/log/atsar/atsadd.
    log-files older than a week are removed daily.

    Add to /etc/cron.d/atsar to produce records every 20 minutes during working hours and hourly otherwise:

    
           0     *    * * 0-6 root /usr/local/bin/atsa1 
           20,40 8-17 * * 1-5 root /usr/local/bin/atsa1
    

    atsaftp counts the new transfers registered in the FTP-logs since the previous time this script was activated; the new counters in /var/log/atsar/ftpstat . The names of the FTP-logfiles to be watched are specified in the /etc/atsar.conf

    atsahttp counts new transfers registered in the HTTP-logs since the previous time this script was activated; the new counters in /var/log/atsar/httpstat . The names of the HTTP-logfiles to be watched are specified /etc/atsar.conf

    Both scripts must be activated just before the program atsadc is started, which also collects these counters.

    FILES

    Daily data file, where dd are digits representing the day of the month. /var/log/atsar/atsadd