Full lesson by Shawn Powers on YouTube at freeCodeCamp.

• Linux Server- System Configuration and Operation
  • Table of Content
  • System Information
  • Kernel and Boot Concepts
  • Configure and Verify Network Connections
    • veth Pair
    • ipvlan
  • Manage Storage
    • GUID Partition Table (GPT) & Master Boot Record (MBR)
    • Filesystem Hierarchy
      • Navigating the Filesystem
    • Creating Partitions
    • Formatting Filesystems
    • Mounting Partitions
    • Logical Volume Manager (LVM)
  • Install, Update, and Configure Software
    • Installing Tarballs
    • Managig .deb
    • Configuring APT Repositories
    • Managing RPM Packages
    • Configuring YUM Repositories
  • Managing Users and Groups
    • Common tools
    • User Profiles
  • Manipulating Files
    • less
    • stdin, stdout, stderr
    • /dev/null, tee, xargs
    • Text Manipulation
    • awk and sed Basic Usage
    • Looking for Files with find
    • Copying Files over Network Using scp and rsync
    • Creating tomporary files with mktemp
  • Managing Services
    • SysV and SystemD Initialization
    • Managing Services
      • With SysV
      • With SystemD
  • Automate and Schedule Jobs
    • System-Wide Scheduling
    • Personal Scheduling
      • Personal crontabs
      • at Daemon
    • Foreground and Background Processes
      • Using nohup to keep a Process Running even if User Logs Out
  • Linux Devices and Filesystem
    • Finding Local Devices
    • Linux Virtual Filesystems
  • Shell Scripting
    • Special Variables
    • Tests
    • if Syntax
    • for Syntax
    • while Syntax
    • trap Syntax
  • Development Tools
    • C Compiler
    • Library Locations
    • Dynamic Linking
      • The runtime dynamic linker/loader - ld.so
      • List Shared Libraries
  • Getting Help
  • Shortcuts
    • Command Line Editing
  • Miscellaneous Commands
  • Desktop Environments
  • Utilities
  • References

• /etc/*release: OS Info.
• uptime: Check how long machine has been on
• free: Check memory information (free/used)

• /etc/modules: Kernel modules to load at boot time.
• /etc/modprobe.d/blacklist.conf: Kernels to not load at boot time.
• Insert & remove modules from running kernel:

  # insert module
  modprobe
  # list modules
  lsmod
  # remove modules
  rmmod

• ip addr: List network connections.
• ip link: List network interfaces.
• ip route: List routing information.
• dig: DNS lookup.
• /etc/hosts: Hosts file. Local DNS.
• /etc/nsswitch.conf: DNS resolution preference.
• /etc/netplan folder holds yaml files for manual network configuration. Run netplan apply.
• nmtui: (in Ubuntu) to invoke Network Manager on the terminal.
• CentOS & RedHat Linux network configuration files are held in the /etc/sysconfig. To reflect changes, run service network restart.
• netstat: basic network debugging. Use -lunt flags to print open TCP & UDP ports.
• lsof -i: list programs currently listening on or using sockets. You can filter by protocol, host, or/and port using the following syntax: lsof -i protocol@host:port. Any one of the parts can be left out. E.g., filtering for host only would be: lsof -i @127.0.0.1 or port only would be: lsof -i :9889. lsof -i 4 lists IPv4 only entries. To filter for Unix domain sockets, run lsof -U.
• tcpdump: can be used to report on every packet that flows through a network interface. E.g., tcpdump -i lo tcp port 7777 prints header information about every TCP IP packet through port 7777. heck the pcap-filter(76) manual page for more usage information.
• netcat: can connect to local / remote TCP/UDP ports, listen on ports, redirect stdio to/fro network connetions, etc. Use netcat -l PORT to listen on a port or netcat HOST PORTto connect to a port.
• nmap: scan machine / network for open ports. E.g., nmap 10.1.2.2. More informationat https://namp.org.
• lsusb & lspci; List USB & PCI connected devices. Listed devices may / may not have drivers installed.
• lshw -C network: List connected network devices. I.e., drivers properly installed & configured.

Create a virtual network interface in a network namespace which can reach the host network.

ip netns add ns1
ip link add veth0 type veth peer name veth1
ip link set veth1 netns ns1
ip addr add 10.200.1.1/24 dev veth0
ip link set veth0 up
ip netns exec ns1 ip addr add 10.200.1.2/24 dev veth1
ip netns exec ns1 ip link set veth1 up
ip netns exec ns1 ip route add default via 10.200.1.1

# Allow ns0 to reach outside world via host
sysctl -w net.ipv4.ip_forward=1

Example ipvlan implementation according to IPVLAN Driver HOWTO.

# Create two network namespaces - ns0, ns1
ip netns add ns0
ip netns add ns1
# Create two ipvlan slaves on eth0 (master device)
ip link add link eth0 ipvl0 type ipvlan mode l2
ip link add link eth0 ipvl1 type ipvlan mode l2
# Assign slaves to the respective network namespaces
ip link set dev ipvl0 netns ns0
ip link set dev ipvl1 netns ns1
# Now switch to the namespace (ns0 or ns1) to configure the slave devices
# - For ns0 (perform similar steps for ns1 with the corresponding ipvl1 device)
ip netns exec ns0 bash
ip link set dev ipvl0 up
ip link set dev lo up
ip -4 addr add 127.0.0.1 dev lo
ip -4 addr add $IPADDR dev ipvl0
ip -4 route add default via $ROUTER dev ipvl0

GPT is essentially a part of UEFI. It is newer, and better performant than MBR (which is attached to the older BIOS). It allows creation of an unlimited number of partitions, while MBR only allows 4. Due to backward compatibility ennhancements, some BIOS systems can read GPT drives.

Linux uses a single filesystem. Say we have linux running on a drive /dev/sda. The root of the filesystem is at /. On this single filesystem, we can have a variety of files:

• Files that actually live on /dev/sda, e.g. /var/log/syslog
• Virtual Files, e.g. files that represent an interface to the kernel /proc/, /sys/
• USB Drives (mounted onto the filesystem), e.g. /media/USBdrive1
• Other drives attached to the computer (mounted onto the filesystem), e.g. /dev/sdb mounted at /mnt/data
• Remote Files (mounted onto the filesystem). Source from e.g. NFS or Samba, e.g. /workspace/

• tree: Shows a tree representation.

• List block devices: lsblk or cat /proc/partitions
• List block devices with extra info (e.g. UUID): blkid
• Creating partitions: parted, gparted (GUI), fdisk. fdisk is older and available in all linux systems
• Deleting partitions: sgdisk

• ext family:
  • Most popular
  • later versions support journals
• xfs:
  • Used on CentOS & RedHat
  • Has its own set of tools (doesn't use the general linux tools)
• btrfs:
  • Newer & feature rich
  • abandoned
• DOS file systems: exFAT, ntfs, fat32, vfat, etc.

Creating a filesystem, use mkfs.

Use mount to mount partitions into the root filesystem and unmount using umount. To mount on boot, we edit /etc/fstab. In /etc/fstab, 'pass' represents wether or not we want to check the filesystem of the drive in the entry on boot. For drive holding root filesystem, specify 1 to check, for all other partitions to be mounted, specify 2 tocheck, or 0 otherwise.

Combine physical devices into volume groups and split the volume groups into logical volumes for use.

Steps:

• ./config: if it exists
• make: If Makefile exists as the default name 'Makefile'. make install may be used to copy program binaries into executable paths (e.g. /usr/local/bin).

Use apt because it's newer than aptitude which is in turn newer than apt-get. dpkg is the program that's used underground, it doesn't install dependencies so can be a pain to use manually.

/etc/apt/sources.list is the system default file. /etc/apt/sources.list.d is a folder in which files can be placed and then they get read like the default file. After adding a repository, e.g. deb <repo-url> <folder/version> <repo-type>, we need to add the Repo's GPG key to prevent man-in-the-middle attacks, e.g.:

# Download key and add to system key list
wget -q0- <repo-key-path> | apt-key add -

ANother way to add a repo is using PPA. e.g.:

add-apt-repository ppa:<user-name>/<repo-name>

This adds the repo to sources.list and adds the key to our list of keys also in one step.

CentOS, Fedora, and RedHat distributions use RPM packages. As with dpkg in Debain, RPM is the low level tool used under the hood, also like dpkg, it doesn't manage dependencies.

• YUM: Updates repo info while installing or upgrading, no 2 steps required (as in apt). Handles dependencies.
• DNF: Replacing YUM in Fedora. Similarly structured. Also handles dependencies.

Main config file is at /etc/yum.conf. All repos are stored in /etc/yum.repos.d. We add epel repo by running: yum install epel-release. This creates an epel.repo file in /etc/yum.repos.d.

• useradd: Home ddirectory and shell need to be manually specified. Password needs to be manually added using passwd. e.g.:

  # Create user
  useradd -d /home/faaizz -s /bin/bash faaizz
  # Create password
  passwd faaizz

• adduser: High-level script that gives an interactive session to create new user.
• userdel: Delete a user
• usermod: Modify a user
• groupadd, groupmod, & groupdel add, modify, & delete groups respectively.
• groups <user-name>: List groups that a user belongs to.
• whoami: Display current user.
• who: Display all users that are currently logged in.
• w: Display all users with more info, including what they're doing.
• pinky: Display all users with more info, including when & from where the user logged in.
• id <user-name>: Info about user, e.g. UID.
• last: Gives history of recent logins.

/etc/environment (if present) and /etc/profile (which includes all files in /etc/profile.d) are loaded on login. /etc/bash.bashrc (for mostly Debian) or /etc/bashrc (for CentOS) is executed whenever a terminal is opened. For user-specific profiles which override the system-wide settings, /home/user/.profile or /home/user/.bash-profile is loaded on login, while /home/user/.bashrc is loaded whenever a terminal is opened.

• dmesg is a command that lets us view logs. E.g. dmesg | grep "search-term".
• df -h: get information about all mounted filesystems

Lets us use Pg Up and Pg Dn. We can also search forward by typing /search-term and pushing Enter, type / and press enter to find more occurence.

# -N flag gives line numbering
less -N /path/to/file

• stdin: Pass data into stdin of a program using | or <.
• stdout: Redirect a program's stdout using >
• stderr: Redirect a program's stderr using 2>

To redirect stdoutandstderrto the same location, we can do:ls Docs ff > /dev/null 2>&1. 2>&1redirectsstderr (2>) into stdout (&1`).

• Whatever you write to /dev/null vanishes forever. It's useful to get rid of output you don't need, ever.
• tee takes in data stdin and pipes it to its stdout while also writing it to a file.
• xargs takes in data via stdin, and runs a program passing the data as arguments. E.g.: ls | xargs mkdir will create a folder for each directroy/file name listed.

• sort sorts the lines of text, by default in alphabetical order. It can also sort in numerical or random order, etc.
• wc counts the words, lines and characters in a file. Gives its output in the format: num-words num-lines num-chars. Flags can be used to output only one of those.
• cut lets us cut out characters with respect to their position on each line. E.g.: cut -c 3,4,5 file1.txt cuts out the 3rd, 4th, & 5th characters on each line of text in file1.txt.
• paste lets us join 2 files line by line seperated with a tab.

# Sample File
cat file1.txt
>Hello Lanre
>Aisha Adekunle
>Yetunde Ademola

# sed
# Substitute (the 's') "Faaizz" for all instances (global 'g') of Lanre term
sed s/Lanre/Faaizz/g file1.txt
# Gives
cat file1.txt
>Hello Faaizz
>Aisha Adekunle
>Yetunde Ademola

# awk
# Replicate the 2nd term ($2), and put it in front of the 1st term ($1) and seperate them with space
# The 'terms' may be seperated by space or tab relative to each line
awk '{ print $2 " " $1}'
# Gives
>Faaizz Hello
>Adekunle Aisha
>Ademola Yetunde

find /path -name *filename* searches for file/directory that contains 'filename' in its name (note the wildcard character '*'). We can also perform actions on the found file. E.g.: find /path -name *filename* -delete finds the file and deletes it.

# Copy file1 to home folder on centos machine via scp
scp ./file1 user@centos:/home
# Copy file2 from home directory on centos machine
scp user@centos:/home/file2 ./
# rsync can recursively copy directories
rsync -av user@centos:/home/Desktop ./

# Create a temporary file in /tmp with a random name starting with 'tempfile'
TMPFILE=$(mktemp /tmp/tempfile.XXXXXX)

cat /proc/interrupts > $TMPFILE

SysV Runlevels and SystemD Boot Target equivalents

• 0: poweroff
• 1: rescue mode
• 3: multi-user
• 5: graphical
• 6: reboot

To check default mode: systemctl get-default. To change the default: systemctl set-default "new-default". To switch between boot targets: systemctl isolate "new-runlevel".

Service files located in:/etc/init.d

• service sshd start
• service sshd status
• service sshd stop
• chkconfig --list sshd: Show startup on boot info for each runlevel
• chkconfig sshd on: Start on boot for all runlevels
• chkconfig -- level 3 sshd on: Start on boot for runlevel 3

Service files scattered around: /etc/systemd/system and other possible locations such as /usr/lib/systemd/system.

• systemctl status sshd
• systemctl start sshd: Start
• systemctl stop sshd
• systemctl disable sshd
• systemctl enable sshd: Start on boot

You must be logged in as root (or prefixed with sudo) to perform system-wide scheduling. \etc\crontab file is loaded by the cron daemon. Every file in /etc/cron.d gets loaded by the cron daemon. Scripts/programs places in auxilliary cron folders e.g. /etc/cron.daily, /etc/cron.weekly, etc. are run as specified by the auxiliary folder's name.

Cron jobs take a scheduling format: minute hour day of month month day of week user to run as command to run.

• * means every
• Minutes go from 0-59
• Hours go from 0-23
• */5 can be used to represent "every 5". E.g. every 5 minutes
• Days of the week go from 0-6, with 0 Sunday and 6 Saturday
• 1-5 can be specified to represent every weekday
• Months go from 1-12, with 1 January and 12 December
• Multiples values can be specified seperated by a comma. E.g. 1,2,3 can be used to specify every 1st, 2nd, and 3rd day of the month

The user to run as field is missing in personal crontabs since we already know which user to run as.

# View Personal crontab
crontab -e

Allows you to run one-off job(s).

at now +5 minute
# Brings up at prompt
at> echo "This is a one-off" >> /home/faaizz/log.txt
# Press ctrl+d to exit prompt

# View queued jobs, displays job ID and other job info
atq

# Cancel job with ID 6
atrm 6

# Run in background
sleep 10000 &

# View running jobs
jobs

# Bring jod with ID 2 (JOB ID not process ID) to foreground
fg 2

# Pressing ctrl+Z on a running job suspends it and puts it in the background
# View the job
jobs
# Assuming the job has ID 3, continue running in background
bg 3

nohup myprogram &

# Show info about plugging in a device
dmesg

# List hardware things
lsblk
lscpu
lspci
lsusb
# etc...

The following directories are a part of the linux virtual filesystem. Information about running processes, the kernel, attached devices etc. can be found in them.

• /proc/
• /sys/
• /dev/

Here'a a list of some special variables that can be used in shell scripts:

Bash and some other shell environments have in-built support for the test command via [ <TEST> ]. Here's a reference to some common test operators:

if CONDITION
then
    # Code to run if CONDITION is true
else
    # Code to run if CONDITION is false
fi

for VARIABLE in LIST
do
    # Code to run for each item in LIST
done

CONDITION below is tested based on it's exit status. If it returns 0, the loop continues, otherwise it stops.

while CONDITION
do
    # Code to run while CONDITION is true
done

trap command allows you to "trap" signals and run a command when the signal is received. It has the syntax trap 'COMMAND' SIGNAL

#!/bin/bash
TMPFILE1=$(mktemp /tmp/tempfile1.XXXXXX)
TMPFILE2=$(mktemp /tmp/tempfile2.XXXXXX)
# Clean up temporary files on interrupt
trap 'rm -f $TMPFILE1 $TMPFILE2; exit 1' INT
# Simulate a long-running process
cat /proc/interrupts > $TMPFILE1
sleep 10000 # Simulate long-running process
cat /proc/interrupts > $TMPFILE2
diff $TMPFILE1 $TMPFILE2
rm -f $TMPFILE1 $TMPFILE2

Most distributions don;t include the tools necessary to compile C code by default. Install build-essential on Debian-based systems or the "Developer Tools" yum groupinstall for Fedora/CentOS systems.

# Compiling C code
cat <<EOF > aux.c
#include <stdio.h>
int hello_call() {
  printf("Hello, World!\n");
}
EOF

cat <<EOF > main.c
void hello_call();
int main() {
  hello_call();
}
EOF

# Compile into object files
cc -c main.c aux.c
# Link object files into an executable
cc -o hello main.o aux.o

# Link against a static library located in one of the default locations
cc -o hello main.o aux.o -lsample

Standard library (& include) locations, include:

• /lib/
• /usr/lib/
• /usr/include/

To make the preprocessor search for header files in a custom location, use the -I flag:

# Compile with a custom header file location
cc -c -I/custom/dir main.c aux.c

Further, includes with double-quote "" indicate that the header file is not in a system directory, usually indicating that it's in the same directory as the source file. E.g.,:

#include "aux.h"  // This will look for aux.h in the same directory as the source file
#include <stdio.h> // This will look for stdio.h in the system include directories

To link against a library in a custom location, use the -L flag:

# Link against a shared library located in /custom/dir/libsample.so
cc -o hello main.o aux.o -L/custom/dir -lsample

Programs don't often know where to find shared libraries, they only often know the library names and some hint at where they might be located. The dynamic linker/loader ld.so - also dynamically linked to the program with its own location explicitly specified - is responsible for finding the shared libraries and loading them into memory when a program is run. You can get help about the dynamic linker via man 8 ld.so.

It uses the following locations to find shared libraries:

• The LD_LIBRARY_PATH environment variable: first place to look for shared libraries. Do not set this at compile time or in startup scripts. Use this only as a last resort, and if it must be used, prefer to set this in a wrapper script that runs the target program.
• Program's runtime library search path (rpath) - if specified. To specify rpath, use the -Wl,-rpath=/path/to/lib flag when linking. E.g., cc -o hello main.o aux.o -Wl,-rpath=/path/to/lib -L/path/to/lib -lsample. Note that L is still required for the linker to know where to find the library during linking. Although not preferred, the patchelf program can be used to update rpath on an existing executable.
• dynamic linker system cache at /etc/ld.so.cache: this is a cache of shared libraries found in directories listed in /etc/ld.so.conf - & correspondingly files contained in /etc/ld.so.conf.d. If modifications are made to these files, run ldconfig -v to update the cache.

To list the shared libraries that an executable is linked against, use ldd:

ldd /bin/bash

# Display command manual
man COMMAND
# Search manual for term
man -k SEARCH_TERM

Manual Sections

# Query DNS hsotname Start of Authority (SOA) Record
nslookup -type=SOA cdcrk.com

# Searching + globbing
grep TERM /path/to/files/*  # lists the files that contain the search term
find /path/to/search -name FILE_TO_FIND -print  # find files that match and print their corresponding paths

• Wayland protocol is the "new cool"
• Check the value of $WAYLAND_DISPLAY to know if you're runninig wayland.
• libinput is a library used by Wayland compositors (& other window managers) to collect messages from peripherials through the linux kernel. It presents an utility (also named libinput) to inspect input devices and events. libinput list-devices: provides information about connected input devices libinput debug-events --show-keycodes: listens for input events

Consider installing the GNU Coreutils on a Linux server for enhanced command-line utilities.

• Linux Server Course - System Configuration and Operation