William Liu

Linux OS

#Summary

These are notes for the Linux OS.

##Linux Distributions

There’s a lot of Linux distributions. This image shows the various distributions:

###Package Managers

Each system has a package management program. They include:

###Favorite Programs

Apt Commands

sudo apt-get update - run this after changing /etc/apt/sources.list or /etc/apt/prefernces sudo apt-get upgrade - upgrades all installed packages sudo apt-get dist-upgrade - same as upgrade, except does a ‘smart upgrade’ (smart conflict resolution) sudo apt-get check - diagnostic tool, does an update of the package lists and checks for broken dependencies sudo apt-get -f install - Fixes broken packages (if you get “unmet dependencies”) sudo apt-get autoclean - removes .deb fiels for packages that are no longer installed on your system sudo apt-get clean - removes all packages from the package cache sudo apt-get remove <package_name> - removes an installed package, leaving configuration files sudo apt-get purge <package_name> - completely removes a package and any associated configuration files sudo apt-get autoremove <package_name> - removes packages that were installed by other packages that are no longer needed sudo apt-cache search <search_term> - searches package name and description contains <search_term>

##Overview

The Linux Operating System has three main levels:

  1. Physical Hardware, e.g. memory (lots of 1s and 0s) and CPU (reads and writes from and to memory)
  2. Kernel, the software sitting in memory that tells the CPU what to do (unrestricted access to hardware)
  3. User Processes, software that user runs (restricted acess to hardware)

##Kernel

The Kernel is the interface between hardware and the user software. In order to managem the CPU and memory, the Kernel has unrestricted access to hardware. Its main jobs are:

###Kernel’s MMU (Memory Management Unit)

The Kernel manages memory with the help of a memory management unit (MMU), that uses a memory management scheme (to enable virtual memory). Some details about memory:

###Kernel CPU Allocation

So what exactly is the relationship between the Kernel and the CPU? The Kernel helps with allocating CPU time for processes with these steps:

  1. Kernel passes User Processes over to the CPU
  2. CPU has an internal timer that interrupts the User Process once its used up its “time slice”
  3. CPU switches back from the User Process to Kernel Mode and hands control back to Kernel
  4. Kernel records the CPU and the process’ state in Memory kinda like a quick save
  5. Kernel does its own processes, e.g. handle input/output
  6. Kernel looks at a list of processes to see what’s ready
  7. Kernel prepares the CPU (how long should this process take) and prepares the Memory (where is there room to store state)
  8. Kernel passes the User Process the CPU to use

Rinse and Repeat.

###System Calls (syscall) with fork() and exec(program)

A system call (aka syscall) is when a User Process can’t do a process (either can’t access or have ability), so the process requests from the Kernel. There are two types of syscalls:

So what does this mean? Other than init, all User Processes start as a result of fork(). Most of the time, fork() then calls exec(program) to start a new program instead of running a copy of an existing program.

Note, system alls are written in C, which is why the syntax has parenthesis.

##Processes

There are two different types of processes in Linux:

###Daemon Processes

Daemon processes are special types of background processes that start at system startup and keep running forever as a service.

###Creating Processes

The init process is the mother (parent) of all processes on the system. You can see this with the following (see process id of 1)

ps aux | grep init
root         1  0.0  0.1 225680  9332 ?        Ss   Jan24   1:50 /sbin/init splash

A new process is normally created when an existing process makes an exact copy of itself in memory. The child process will have the same environment as its parent, but only the process ID number is different. You’ll see the Process ID (PID) and the Parent Process ID (PPID).

When creating a new process, you can use fork or exec.

##Users

Linux defines a User as ‘someone that can run processes and own files’. As a User:

##Passwords

Passwords are stored in /etc/shadow

###root is a superuser

The username root is a superuser, meaning it has access to escalated permissions to allow terminating other Users’ processes. You don’t want to normally run as root since it can be easy to make mistakes. Even though root is powerful, root still runs in ‘user space’ (and not in Kernel mode)

###Groups

Linux uses groups to organize users. Groups organize collections of accounts, mainly for handling permissions nicer. Control of each group is administered through the /etc/group file.

Every user has a default or primary group. You can change groups with the chgrp <newgroup> command.

##Shells

A shell is a program that runs commands. Shell Scripts are files that contain many shell commands. There are many types of shells, but they mainly originate from the bourne shell and the most common is the bourne again shell (aka bash).

You can usually find what shell is being used by going to /bin/sh and peaking in there to see the link to the shell.

chsh to change a shell.

###Bourne Shell

The original shell is the bourne shell. To see a modern shell, see below for Bourne Again Shell (aka bash)

###bash

bash is an enhanced version of the bourne shell and is the standard shell on most Linux distributions. Most of the below notes are for bash.

##Standard Streams

Linux processes use input and output steams to read and write data. For example, an input stream might be a device and the output stream might be a file or terminal. So what do you mean?

  1. Type in the command cat in bash
  2. Bash turns into an interactive stream; when you type in text, the text comes out to the terminal
  3. The reasoning is that since we didn’t specify either a standard input or standard output steam, it used the default.

There’s three types of Standard Streams:

###Standard Input (stdin)

Basic Input.

###Standard Output (stdout)

Where output goes. By default, the Stream ID is 1.

###Standard Error (stderror)

Where errors go. By default, the Stream ID is 2.

##Basic Unix Commands

Unix is a flavor of Linux. You can use unix in many other systems so they will be your staple commands.

###xargs

xargs is a good command to help chain other commands.

Say you have a set of directories that you want to copy over to another location. You can use xargs to help:

/tmp/yay/ls
dir1 dir2 dir3 dir4

/tmp/yay/ls | xargs -I % mkdir -p /tmp/another_dir/%
/tmp/another_dir/ls
dir1 dir2 dir3 dir

-I option

The -I option lets you take a string that will be replaced with the supplied input before the command is executed. Usually you use % as a common choice.

Shell Trick

Another trick is to use a shell script inside your command:

mkdir ~/backups
$ find /path -type f -name '*~' -print0 | xargs -0 bash -c 'for filename; do cp -a "$filename" ~/backups; done' bash

##Navigating Directories and Files

In order to navigate around Linux in the command line, you will need to know how directories and files are referenced and how to move across these paths.

These commands and terms are useful to navigate around paths:

##Searching Directories in Shell using Globbing (Wildcards)

Globbing (aka Wildcards) is the process of matching files and directories to a pattern in the shell.

###Globbing Expansion Steps

So what happens after a match is made? The shell does a process called expansion, which means:

  1. A glob matches a file or directory name
  2. The glob substitutes those names
  3. The glob then runs the rest of the shell command

Some Notes:

###Globbing with *

* is used in shell globbing to match any arbitrary set of characters. Here’s sample ways of using *:

###Globbing with ?

? is used in shell globbing to match one arbitrary character.

##Grep

Grep prints the lines that matches an expression/a pattern. Grep understands patterns called regular expressions. Example uses include:

##File Search

To search inside a file (not in the shell), use / and ?.

##Shell Basics

Let’s take a look at how our shell is configured and how to move around using shortcuts.

###Shell Keyboard Shortcuts

Shell keyboard shortcuts can be helpful:

###Dot (Configuration) Files

Dot files are files that start with a .. These configuration files and directories are normally hidden by default and usually hold config settings.

###Shell Variables

Shell variables are temporary variables being stored by the shell. These can contain simple strings or some variables to control the way the shell behaves.

###Environment Variables

Environment variables are slightly different than shell variables in that they get passed to the programs that the shell runs. These are used in configuration and option settings.

###PATH variable

The PATH variable is a special environment variable that contains the command path; its purpose is to list the directories that the shell searches to locate a command.

##man pages and info

man stands for Manual Pages, which are built in docts that are split into these 8 sections:

  1. User Commands
  2. System Calls
  3. Higher-level Unix programming library documentation
  4. Device Interface and Driver Information
  5. File descriptions (system configuration files)
  6. Games
  7. File formats, conventions, and encodings (ASCII, suffixes, and so on)
  8. System commands and Servers

###How to use man pages

Type in man <number> <search> where number is one of the sections above (1-8) and a search term

###

To get additional information, try out info <command>, e.g. info grep

##Redirecting Standard Streams

You can redirect standard input, standard output, and standard error streams. Normally we have:

stdin 0 - standard input typically carries data from a user to a program (e.g. receive input from a device like a keyboard). Normally stdin is terminated by EOF (end-of-file), meaning no more data to be read. stdout 1 - standard output writes the data that is generated by a program; when this is not redirected, it normally outputs text back to the terminal. stderr 2 - standard error writes the errors generated by a program that has failed at some point in its execution. the default destination for this stream is the terminal display

With redirection, we can:

You can combine some of the stdin, stdout, stderr together with redirection:

Overwrite

Overwrite with:

> standard output < standard input 2> standard error

Append

You can append with:

>> appends standard output << appends standard input 2>> appends standard error

Pipes

Use pipes | to move output from one program to the input of another program

##List and Manipulate Processes

A process is a running program. Here’s how to read process details:

Common process commands are:

###Kill Processes

kill to signal to kill a process

###More Job Control

As you saw in Kill, you can manage shell jobs by sending TSTP signals (similar to STOP) and CONT signals. You can also send a job to the background or foreground.

##Linux Directory Hierarchy

The details of the Linux Directory Hierarchy can be found in the Filesystem Hierarchy Standard (FHS) at http://www.pathname.com/fhs/

The overview looks like this for the main directories:

###usr

The /usr directory is where most of the user-space programs and data sites.

- `/usr/bin/`
- `/usr/man/` has man pages
- `/usr/info/` has info pages
- `/usr/include` holds header files used by the C compiler
- `/usr/lib/`
- `/usr/local/` where administrators can install their own software; should look like that of `/` and `/usr`
- `/usr/sbin/`
- `/usr/share/` has files that should work on other kinds of Unix machines; sometimes `/man` or `/info` are here

##File Permissions

Every Unix file has a set of permissions that say whether you can read, write, or execute the file.

###chmod to change file permissions

Use chmod to change file permissions. There’s also umask to set a predefined set of permissions for any new files you create.

###Common Permissions

Some of the most common modes:

##Group Permissions

You can see all the group permissions under /etc/group. If you want to check the members of a group, you can run getent group my_group. To see your own groups, run groups.

##Symbolic Links

A symbolic link is a file that points to another file or directory.

lrwxrwxrwx 1 ruser users 11 Feb 27 13:52 somedir -> /home/origdir

In this example, when you try to access somedir, you get /home/origdir, i.e. a shortcut.

Links can get pretty confusing because:

###Creating Symbolic Links

ln -s target linkname so that the linkname argument is the name of the symbolic link. The target arg is the path of the file or directory that the link points to. The -s flag specifies a symbolic link.

To update a link, you can run ln -sf /path/to/dir/that/exists/already/ /path/to/symlink/akamyshortcut/

Remember that symlink is like cp, where you have the source and the target.

###Delete Symbolic Links

To delete a symbolic link, just type in unlink followed by the link

You can see a symlink with something like ls -alh .

lrwxrwxrwx  1 will will 42 Apr  2  2016 my_stuff -> my_folder/my_stuff/

##Archiving and Compressing Files

We want to be able to zip/unzip files (i.e. do compression / uncompress files) and to handle archives (pack / unpack multiple files and directories into one file).

###Gzip

To compress and uncompress files, we’ll use the program gzip. Files end with .gz

###tar

To pack multiple files and directories into one file, we want to create an archive with tar. We can also unarchive (take one file and put back into multiple files and directories). Files end with .tar

A lot of this depends on the modes:

Then we can add additional flags:

Example Commands:

###Compressed Archives with .tar.gz

Archives are normally found compressed, so you’ll see a filename ending in .tar.gz. Work from the right side to the left. Uncompress, then Unarchive.

  1. Unzip first with gunzip my_file.tar.gz
  2. Tar to unarchive with tar xvf my_file.tar

###zcat

A faster and more efficient way (in terms of kernel I/O time, disk space), is to use zcat.

zcat my_file.tar.gz | tar xvf -

This zcat command is the same as gunzip -dc. The -d option decompresses and the -c option sends to standard output (in this case, to the tar commmand)

Note: a .tgz file is the same as a .tar.gz file

###bzip2

A .bz2 file is the result of another compression program called bzip2. The advantage of this format is that it comrpesses text files a little more.

bunzip2 to decompress.

###zip and unzip

zip and unzip are still available to help with zip archives that come from Windows systems.

##su and sudo

You can run commands as the superuser; one method is to use su. Issues include:

Instead, use sudo, e.g. sudo vipw

Note, you can configure list of users that can run commands as supervisor under /etc/sudoers file. You can set which Users can use sudo

##Devices

The kernel provides a way to interface with devices.

###Using /dev to identify devices

Let’s see how device ouptut might look like:

$ls -l

#Sample output
brw-rw----   1 root disk 8, 1 Sep 6 08:37 sda1
crw-cw-rw-   1 root disk 1, 3 Sep 6 08:37 null

Here’s how to read the output:

There are also two numbers before the date, indicating the major and minor device numbers that help the kernel identify the device.

###Using /sys to identify devices

###dd program for block and character devices

dd is useful for working with block and character devices because it allows you to read from an input file or stream and write to an output file or stream.

###Finding the name of a device

You can search for the name of a device a few ways:

###Hard Disks

Most hard disks are connected using the sd prefix, standing for SCSI disk (Small Computer System Interface). You might see:

####Terminals

Terminals are devices for moving character between a user process and an I/O device, usually for text output to a terminal screen. Pseudoterminal devices are emulated terminals that understand the I/O features of real terminals, but instead of talking to actual hardware, the kernel presents the I/O interface to software, like the shell terminal window.

Linux has two primary display modes: text mode and an X Window System server (graphics mode)

####Other Devices

####Creating Device Files with udevd

Use devtmpfs and udev to create device files

##Disks and Filesystems

When working with disks and filesystems, you should be able to partition disks, create and maintain filesystems inside of the partitions, and work with swap space.

###Understanding Partitions

There are a few kinds of partition tables.

When we break up the partitions, we see that there are primary, extended, and logical partitions:

When initially reading a table, the Linux kernel produces a debug output (can be found through dmesg) that shows these partitions, e.g.

sda: sda1 sda2 < sda5 >
# the `sda2 < sda5 >` output means that `/dev/sda2` is an extended partition containing a logical partition (`/dev/sda5`)
* We normally ignore extended partitions because we only want the logical partitions inside

###Partitioning Tools

Here are some different partitioning tools:

###Using a Partitioning Tool

You can view and change partition tables with parted and fdisk. We will use parted for system partitioning, but it’s not recommended to use parted for creating filesystems.

###Partition Considerations

There can be big performance consequences if you aren’t careful with laying out data based on the hardware.

Hard drives have a spinning platter on a spindle, with the head attached to a moving arm to read/write data on a cylinder; you can see sectors and cylinders, but the reported values are fiction so there’s not much you can do here; trust the Logical Block Addressing (LBA) to assign blocks correctly

Solid-State Disks (SSDs) don’t have moving parts, but one performance factor is partition alignment, which means read/writes are 4096 bytes at a time; this means if your partition data is outside the 4096 byte boundary, you have to do two reads instead of one

####Filesystems

The file-system are the files that you’re used to interacting with in user space. You can run ls and cd to these files and directories. Remember that the filesystem is a form of database that structures these simple block devices into a hierarchy of files and subdirectories that users can understand

There are different filesystem types, some optimized for Linux, and other foreign ones like Windows FAT family. Here are some common types:

You can also create filesystems with mkfs, e.g. mkfs -t ext4 /dev/sdf2 to automatically determine the number of blocks in a device and set reasonable defaults

####Mounting a Filesystem

Mounting is the process of attaching a filesystem. When the system boots, the kernel reads some configuration data, then mounts root / based on the config data.

You need to know the following in order to mount a filesystem:

####Mount Commands

Run mount to get the filesystem status on your system. Each line corresponds to one currently mounted filesystem and follows this format:

To mount and umount:

Finding mounts:

$ findmnt
TARGET                                SOURCE     FSTYPE          OPTIONS
/                                     /dev/sda1  ext4            rw,relatime,errors=remount-ro,data=ordered
├─/sys                                sysfs      sysfs           rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security              securityfs securityfs      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup                    tmpfs      tmpfs           ro,nosuid,nodev,noexec,mode=755
│ │ ├─/sys/fs/cgroup/systemd          cgroup     cgroup          rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd
│ │ ├─/sys/fs/cgroup/rdma             cgroup     cgroup          rw,nosuid,nodev,noexec,relatime,rdma
│ │ ├─/sys/fs/cgroup/cpu,cpuacct      cgroup     cgroup          rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
│ │ ├─/sys/fs/cgroup/memory           cgroup     cgroup          rw,nosuid,nodev,noexec,relatime,memory
│ │ ├─/sys/fs/cgroup/net_cls,net_prio cgroup     cgroup          rw,nosuid,nodev,noexec,relatime,net_cls,net_prio
│ │ ├─/sys/fs/cgroup/cpuset           cgroup     cgroup          rw,nosuid,nodev,noexec,relatime,cpuset
│ │ ├─/sys/fs/cgroup/pids             cgroup     cgroup          rw,nosuid,nodev,noexec,relatime,pids
│ │ ├─/sys/fs/cgroup/blkio            cgroup     cgroup          rw,nosuid,nodev,noexec,relatime,blkio
│ │ ├─/sys/fs/cgroup/devices          cgroup     cgroup          rw,nosuid,nodev,noexec,relatime,devices
│ │ ├─/sys/fs/cgroup/perf_event       cgroup     cgroup          rw,nosuid,nodev,noexec,relatime,perf_event
│ │ ├─/sys/fs/cgroup/freezer          cgroup     cgroup          rw,nosuid,nodev,noexec,relatime,freezer
│ │ └─/sys/fs/cgroup/hugetlb          cgroup     cgroup          rw,nosuid,nodev,noexec,relatime,hugetlb
│ ├─/sys/fs/pstore                    pstore     pstore          rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/debug                 debugfs    debugfs         rw,relatime
│ ├─/sys/kernel/config                configfs   configfs        rw,relatime
│ └─/sys/fs/fuse/connections          fusectl    fusectl         rw,relatime
├─/proc                               proc       proc            rw,nosuid,nodev,noexec,relatime
│ └─/proc/sys/fs/binfmt_misc          systemd-1  autofs          rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=480
├─/dev                                udev       devtmpfs        rw,nosuid,relatime,size=12283432k,nr_inodes=3070858,mode=755
│ ├─/dev/pts                          devpts     devpts          rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ ├─/dev/shm                          tmpfs      tmpfs           rw,nosuid,nodev
│ ├─/dev/hugepages                    hugetlbfs  hugetlbfs       rw,relatime,pagesize=2M
│ └─/dev/mqueue                       mqueue     mqueue          rw,relatime
└─/run                                tmpfs      tmpfs           rw,nosuid,noexec,relatime,size=2462716k,mode=755
  ├─/run/lock                         tmpfs      tmpfs           rw,nosuid,nodev,noexec,relatime,size=5120k
  └─/run/user/1000                    tmpfs      tmpfs           rw,nosuid,nodev,relatime,size=2462716k,mode=700,uid=1000,gid=1000
    └─/run/user/1000/gvfs             gvfsd-fuse fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000

####Mount NFS File Systems with fstab

Another way you can mount an NFS share from another machine is to add a line to /etc/fstab file. The line needs to start the hostname of the NFS server, the directory on the server being exported, and the directory on the local machine where the NFS share is to be mounted.

####Mount NFS File Systems with autofs

You can mount an NFS share using the autofs service. Autofs uses the automount daemon to manage your mount points by only mounting them dynamically when they are accessed. Autofs uses the master map configuration file over in /etc/auto.master to determine which mount points are defined. It then starts an automount process with the appropriate parameters for each mount point. Each line in the master map defines a mount point and a separate map file that defines the file systems to be mounted under this mount point. You might see:

/etc/auto.misc file that defines the mount points in the /misc directory /etc/auto.master the master file might define the relationship for this with a line like:

/misc and /etc/auto.misc --timeout 60

#####Autofs Commands

You can start the autofs service with:

/sbin/service autofs restart /sbin/service autofs status /sbin/service autofs reload - you must reload autofs when you modify /etc/auto.master

##Load Averages

Load Averages are a good indication on how your server is being used. You can see this when you type uptime or from htop. You see three numbers, like:

$ uptime
Tasks: 137, 554 thr, 1 running
Load average: 0.03 0.54 1.24
Uptime: 9 days

There are three numbers for load averages, and they represent the 1 min, 5 min, 15 min marks. With these, you can tell if the system load is idle, increasing, decreasing, or if its higher than your CPU count, you probably have a performance problem. Note that this is system load, not necessarily just CPU so it might include CPU, disk, uninterruptible locks.

You can look up bcc-tools (https://github.com/iovisor/bcc) for analysis of what the CPU is doing (e.g. create a CPU Flame Graph).

##Better CPU Metrics

Some better metrics (especially for CPUs) might be:

per-CPU utilization

$ mpstat -P ALL 1
09:45:30 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
09:45:31 PM  all    1.27    0.00    0.76    0.00    0.00    0.00    0.00    0.00    0.00   97.97
09:45:31 PM    0    1.04    0.00    1.04    0.00    0.00    0.00    0.00    0.00    0.00   97.92
09:45:31 PM    1    2.02    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   97.98
09:45:31 PM    2    1.02    0.00    1.02    0.00    0.00    0.00    0.00    0.00    0.00   97.96
09:45:31 PM    3    1.00    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00   98.00

per-process CPU utilization

$ top, pidstat 1
09:52:14 PM   UID       PID    %usr %system  %guest    %CPU   CPU  Command
09:52:15 PM   123      1123    1.00    0.00    0.00    1.00     2  beam.smp
09:52:15 PM     0      1271    1.00    0.00    0.00    1.00     1  docker-containe
09:52:15 PM     0      1332    1.00    0.00    0.00    1.00     3  Xorg
09:52:15 PM  1001      2288    0.00    1.00    0.00    1.00     0  i3status
09:52:15 PM  1001     10390    1.00    0.00    0.00    1.00     2  pidstat
09:52:15 PM  1001     16999    2.00    0.00    0.00    2.00     1  chrome

per-thread run queue (scheduler) latency

E.g. in /proc/PID/schedstats, delaystats, perf sched

$ perf sched

CPU run queue latency

E.g. in /proc/schedstat, perf sched

CPU run queue length

E.g. vmstat 1 and the ‘r’ column

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
0  0  48348 224424 385352 3696108    0    0    28   149  205  156 36  8 57  0  0
0  0  48348 224268 385352 3696148    0    0     0     0  275  674  1  1 98  0  0
1  0  48348 277900 385352 3642484    0    0     0     0 3792 1269  1  1 98  0  0
0  0  48348 277912 385352 3642432    0    0     0     0 5856 1545  1  1 98  0  0
0  0  48348 277820 385356 3642428    0    0     0    60 7907 1987  2  1 97  0  0

##IO Stat

You can get IO stats from iostat.

$iostat
Linux 4.10.0-20-generic (xps) 	08/08/2017 	_x86_64_	(4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          36.21    0.09    7.78    0.08    0.00   55.84

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
nvme0n1           1.10         5.03       254.27    4160554  210132300
dm-0              0.02         0.01         0.06       9600      52532

Run iostat -m 5 to show in megabytes per second and 5 to recalculate every 5 seconds. Another good option is -x to show extended statistics.

$iostat -mx 5
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          16.43   13.11    3.13    0.96    0.72   65.65

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00   246.45  279.18  235.57     2.04     2.77    19.15     0.04    0.07    0.28    0.21   0.08   3.90
dm-0              0.00     0.00    1.49    5.55     0.01     0.04    14.11     0.02    2.73    4.83    2.17   0.43   0.31
dm-1              0.00     0.00    2.28  119.85     0.04     0.53     9.68     0.02    0.19    7.62    0.05   0.19   2.26
dm-2              0.00     0.00   48.27   33.16     0.33     0.18    12.83     0.06    0.69    1.34    2.47   0.47   3.80
dm-3              0.00     0.00  225.28  278.88     1.65     2.01    14.87     0.04    0.09    0.33    0.21   0.01   0.31

You can check out the man pages to see what each of these mean, but in summary:

Overall IOPS are with:

iostat -d <your disk name> | grep <your disk name> | awk '{ print $2; }'

e.g. iostat -dm sda | grep sda | awk '{ print $2; }'
     16.41

Below this is the Disk Utilization Report, which shows stats per physical device or partition basis.

##sysstat

sysstat is good for checking out disk I/O. On Ubuntu, you can install with:

/etc/cron.d/sysstat

# The first element of the path is a directory where the debian-sa1
# script is located
PATH=/usr/lib/sysstat:/usr/sbin:/usr/sbin:/usr/bin:/sbin:/bin

# Activity reports every 10 minutes everyday
*/2 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1

# Additional run at 23:59 to rotate the statistics file
59 23 * * * root command -v debian-sa1 > /dev/null && debian-sa1 60 2

##Use Method

For additional metrics / troubleshooting tips, see more from Brendan Gregg (like his USE Method)[http://www.brendangregg.com/usemethod.html].

USE Method - For every resource, check utilization, saturation, and errors.

Terminology:

resource - all physical server functional components (CPUs, disks, buses,
etc)
utilization - average time that the resource was busy servicing work
(expressed as a percent over time, e.g. one disk is running at 90%
utilization)
saturation - degree to which the resource has extra work which it can't
service, often queued (as a queue length, e.g. the CPU has an average run
queue length of four)
errors - count of error events (e.g. this network interface has had fifty
late collisoins)

Things to look for:

##Systemd

Systemd is an init system and system manager that is managed through systemctl (command line utility for starting and stopping system-level services and checking their status)

Files

We have these files living in /etc/systemd/system/ directory and in directories like /run/systemd/system/ (for run-time unit definitions) and /lib/systemd/system/ (where the system has a copy of unit files).

# List the services that can be started or stopped
ls /etc/systemd/system/*.service
ls /run/systemd/system/*.service
ls /lib/systemd/system/*.service

Where to store files:

Overriding conf files for

If you were modifying a unit in say /etc/systemd/system, you want to do the following so you don’t overwrite the original file:

Running Commands

You can run commands like:

systemctl status my_name.service
systemctl status ntpd.service

# list unit files
systemctl list-unit-files --type=service
systemctl list-units --type [unit-name]

systemctl start my_name.service  # activate/start a service immediately
systemctl stop my_name.service  # deactivate/stop a service immediately
systemctl restart my_name.service  # restart a service (stop then start)
systemctl enable my_name.service  # Enable a service to start on bootup
systemctl disable my_name.service  # Disable a service to NOT start on bootup
systemctl reload my_name.service  # When supported, reloads a config file without interrupting pending operations
systemctl is-enabled my_name.service; echo $?  # Check if service is enabled (0 is enabled, 1 is disabled)
systemctl daemon-reload  # if you've modified a .service and need to reload it

Units

In systemd, a unit refers to any resource that the system knows how to operate on and manage. These resources are defined using configuration files called unit files. Basically, units are the objects that systemd knows how to manage. These units can be abstract services like network resources, devices, and filesystem mounts.

Types of Units

Systemd categories units according to the type of resource they describe. You can tell the type of a unit with its suffix (e.g. .service, .socket).

##Log Rotate

Logrotate is a program on Linux systems that rotates, compresses, and mails system logs. Logrotate is used with ease of administration of systems that generate a large number of log files. Each log file can be handled daily, weekly, monthly, or when it grows too large.

Normally logrotate is setup as a daily cron job and is setup through a series of configuration files.

Log Rotate Config file

On Ubuntu, logrotate files are usually in:

For example, apt’s logrotate would be in cat /etc/logrotate.d/apt:

/var/log/apt/term.log {
  rotate 12
  monthly
  compress
  missingok
  notifempty
}

/var/log/apt/history.log {
  rotate 12
  monthly
  compress
  missingok
  notifempty
}

Another logrotate configuration would be:

# sample logrotate configuration file
compress

/var/log/messages {
    rotate 5
    weekly
    postrotate
        /usr/bin/killall -HUP syslogd
    endscript
}

"/var/log/httpd/access.log" /var/log/httpd/error.log {
    rotate 5
    mail www@my.org
    size 100k
    sharedscripts
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
}

/var/log/news/* {
    monthly
    rotate 2
    olddir /var/log/news/old
    missingok
    postrotate
        kill -HUP `cat /var/run/inn.pid`
    endscript
    nocompress
}

~/log/*.log {}

William Liu

github.com/williamqliu