Linux Part 2 — the productivity boosters

Text processing (grep / sed / awk)

grep "error" app.log                  # lines containing "error"
grep -i "Error" app.log               # case-insensitive
grep -r "TODO" src/                   # recursive
grep -v "healthcheck" app.log         # invert (exclude)
grep -c "500" app.log                 # just count matches

sed 's/foo/bar/g' file                # print with foo → bar
sed -i 's/foo/bar/g' file             # in-place edit

awk -F: '{print $1}' /etc/passwd      # first field, ':' separator
awk '$3 > 100 {print $1}' data.txt    # rows where col3 > 100

Diffing & counting

diff old.txt new.txt                  # line-by-line diff
wc -l file                            # count lines
wc -w file                            # count words

Networking

ping google.com                       # is DNS+network alive?
netstat -pnltu                        # who is listening?
ss -tulpn                             # newer alternative to netstat
ssh alice@server                      # remote login
scp file alice@server:/tmp/           # copy over SSH
curl https://api.github.com           # HTTP request
curl -X POST -d '{"x":1}' -H "Content-Type: application/json" $URL
wget https://…/file.zip               # download a URL

Archives

tar cf archive.tar dir/               # create tar
tar xf archive.tar                    # extract tar
tar czf archive.tar.gz dir/           # gzip'd tar (very common)
tar xzf archive.tar.gz                # extract .tar.gz
tar jcvf f.tar.bz2 dir/               # bz2 tar
zip -r stuff.zip folder               # create zip
unzip stuff.zip                       # extract zip
gzip file    /    gzip -d file.gz     # single-file gzip

Disk / memory / uptime

df -h                                 # disk free per mount
du -h --max-depth=1                   # size of each subfolder
free -h                               # RAM + swap
uptime                                # load average + uptime
date                                  # current time
uname -a                              # kernel info

Package installs

sudo apt-get update
sudo apt-get install pkgname
sudo apt-get remove pkgname

Bash scripting basics

#!/usr/bin/env bash
name=${1:-World}                      # arg 1, or default World

if [[ $name == "root" ]]; then
  echo "hi root"
elif [[ $name == "" ]]; then
  echo "no name"
else
  echo "Hello, $name!"
fi

for f in *.log; do
  gzip "$f"
done

i=0
while (( i < 3 )); do
  echo "$i"
  ((i++))
done

Golden rule: always quote your variables ("$name") and run shellcheck script.sh.

Cron

crontab -e            # edit
# ┌────── minute (0-59)
# │ ┌──── hour (0-23)
# │ │ ┌── day of month (1-31)
# │ │ │ ┌ month  ┌ dow (0=Sun)
# │ │ │ │ │
0 2 * * *   /usr/local/bin/backup.sh   # 2 AM daily
@reboot     /usr/local/bin/warmcache.sh

Chaining commands

cmd1 ; cmd2               # cmd2 always
cmd1 && cmd2              # cmd2 only if cmd1 succeeded
cmd1 || cmd2              # cmd2 only if cmd1 failed
cmd &                     # background

Real-world example

You want to alert on failing logins across the fleet:

grep "Failed password" /var/log/auth.log | awk '{print $9,$11}' | sort | uniq -c | sort -rn | head

One line → username × source IP × count. Beautiful.