Linux Administration

TL;DR

• The Linux OS layer (filesystem, processes, systemd, permissions, package management) underlies every orchestration platform and CI/CD pipeline — production incidents are diagnosed fastest at this layer, not at the abstraction on top of it.
• A filesystem can be “full” (df -h) for three distinct reasons: block exhaustion, inode exhaustion (df -i), or space held by a deleted-but-open file (lsof | grep deleted) — each needs a different fix.
• A zombie process (Z state) is a dead process whose exit status hasn’t been reaped by its parent; it cannot be killed directly and consumes no real resources. A D-state process is blocked on I/O and inflates load average without inflating CPU%.
• systemd + journalctl is the primary service-management and log-triage toolchain; StartLimitBurst/StartLimitIntervalSec caps crash-loop restarts.
• chmod 777 is a near-automatic security-review finding; a dedicated group with setgid (2770) is the correct pattern for shared write access.
• Package pinning (apt-mark hold, yum versionlock) must happen at the package-manager level to survive unattended upgrades — avoiding a manual upgrade command isn’t sufficient.

Core Concepts

Filesystem: blocks, inodes, and deleted-but-open files

The filesystem is a tree, but two operational details matter more than the tree structure itself:

• Inodes vs. data blocks: a filesystem tracks free space in two independent pools — data blocks and inodes (metadata structures, one per file). A directory tree with millions of tiny files can exhaust inodes while blocks still show free space. df -h reports block usage only; df -i reports inode usage, and it is the only command that reveals inode exhaustion.
• Deleted-but-open files: on Unix filesystems, a file’s data isn’t freed until both its directory entry is removed and its last open file descriptor is closed. A process that has a file open when it’s deleted (e.g., a log file removed by a cleanup job while the writing process is still running) keeps consuming disk space invisibly. du walks the directory tree and cannot see the file (no directory entry exists), but df still counts the space against the filesystem, so the two disagree. lsof +L1 (or lsof | grep deleted) lists open files with a link count of 0 — the direct way to find the process holding the space, which is freed only when that process closes the descriptor or is restarted.

Process lifecycle and states

• Zombie (Z): the process has exited and released its resources, but its entry remains in the process table until the parent calls wait()/waitpid() to collect the exit status. A zombie is not consuming CPU or memory beyond a table slot; it cannot be killed with kill -9 because it is already dead — the fix targets the parent (fixing its signal handling, or restarting/killing the parent so init/systemd reparents and reaps the zombie).
• Uninterruptible sleep (D): the process is blocked on a low-level I/O operation (disk, some NFS calls) and cannot be interrupted by signals, including SIGKILL, until the I/O completes or times out. Many D-state processes are the direct explanation for a load average that looks high relative to CPU usage.
• Load average counts both runnable processes and uninterruptible-sleep processes over 1/5/15-minute windows. A high load average with moderate CPU% in top is the classic signature of an I/O bottleneck, not a CPU capacity problem — iostat or the %wa column in top confirms it.

systemd service management and log triage

systemd is the init system and service manager on most modern distributions. Core operational commands:

• systemctl status <service> — quick health snapshot (active/failed/state, recent log lines, main PID).
• journalctl -u <service> -f — live tail of a unit’s logs.
• journalctl -u <service> --since "10 min ago" — recent history without tailing.
• Unit-file restart control: Restart=on-failure, RestartSec=, and critically StartLimitBurst= / StartLimitIntervalSec=, which cap how many restart attempts are allowed within a time window. Without a sane limit, a broken deploy can crash-loop indefinitely, hammering the host and any downstream dependency it calls on each restart attempt, instead of systemd giving up and reporting a clear failed state.

Users, groups, and permissions

The rwx model is simple in isolation but requires care for shared access:

• chmod 777 grants read/write/execute to owner, group, and all other users — any local account or compromised process on the host gains full access to the directory’s contents. This is flagged in virtually every security review because it removes any access boundary at all.
• The correct pattern for a directory that a specific team needs shared write access to is a dedicated group plus the setgid bit (chmod 2770 dir): setgid on a directory makes new files/subdirectories inherit the directory’s group automatically, and 770 restricts read/write/execute to the owner and that group only — narrower and auditable, while still functionally supporting shared writes.
• umask controls default permission bits for newly created files/directories and interacts with setgid-based directory schemes; worth checking when new files unexpectedly land with the wrong group-write bit.

Package management and version pinning

apt/dpkg (Debian/Ubuntu) and yum/dnf/rpm (RHEL/CentOS/Fedora) manage installed software and dependencies. The operational risk they introduce is an unattended or routine upgrade silently bumping a package whose exact version a service depends on for compatibility.

• Debian/Ubuntu: apt-mark hold <package> prevents the package manager (including unattended-upgrades) from upgrading it until unheld with apt-mark unhold.
• RHEL/CentOS: yum versionlock add <package> (via the versionlock plugin) achieves the same effect.
• Pinning must happen at this level — relying on convention (“nobody run apt upgrade on this box”) does not protect against automated unattended-upgrade services or a teammate running a routine patch cycle across a fleet.

Process monitoring

• top/htop — live resource snapshot; sortable by CPU/memory.
• ps aux --sort=-%mem / --sort=-%cpu — one-shot sorted process listings, useful for scripting and non-interactive checks.
• A high load average with low CPU% in either tool points to I/O wait, not a CPU shortage — adding compute does not fix a storage bottleneck.

systemd vs. SysV init

Soft link vs. hard link

Command / Configuration Reference

df -h                          # disk usage by block, human-readable
df -i                          # inode usage — catches "too many small files" full disks
du -sh /*                      # directory sizes at top level (won't see deleted-but-open files)
lsof +L1                       # files with link count 0 (deleted but still open)
lsof | grep deleted            # alternate way to find deleted-but-open file handles

ps aux | grep ' Z '            # find zombie processes
ps -eo pid,ppid,stat,cmd | grep D   # find processes in uninterruptible sleep

systemctl status <service>     # health snapshot
journalctl -u <service> -f     # live log tail
journalctl -u <service> --since "10 min ago"   # recent history
systemctl restart <service>    # restart a unit
# unit file:
# [Service]
# Restart=on-failure
# RestartSec=5
# StartLimitBurst=5
# StartLimitIntervalSec=60

chmod 2770 /shared/dir         # setgid + rwx for owner/group, nothing for others
chgrp teamgroup /shared/dir    # set the directory's group ownership

apt-mark hold <package>        # pin package version (Debian/Ubuntu)
apt-mark unhold <package>      # remove the pin
yum versionlock add <package>  # pin package version (RHEL/CentOS)
yum versionlock delete <package>

top                             # live process/resource view
ps aux --sort=-%mem | head      # top memory consumers
iostat -x 1                     # I/O wait / device utilization, refreshed every second

Common Pitfalls

• Pitfall: A “disk full” alert is investigated with du alone and the discrepancy against df is never explained. Why: du only sees files reachable through the directory tree; a deleted-but-open file has no directory entry. Fix: Cross-check df against du, and run lsof | grep deleted when they disagree.
• Pitfall: chmod 777 is applied to resolve a permissions error and left in place. Why: It’s the fastest way to make an error disappear, but it removes any access boundary. Fix: Use a dedicated group with setgid (2770) scoped to the users who actually need write access.
• Pitfall: A systemd service crash-loops for an extended period, hammering a downstream dependency. Why: No StartLimitBurst/StartLimitIntervalSec was configured, so systemd keeps retrying indefinitely. Fix: Set restart limits in the unit file so systemd gives up and reports a failed state after a bounded number of attempts.
• Pitfall: A high load average is treated as a CPU capacity problem and more compute is provisioned. Why: Load average includes uninterruptible-sleep (D-state) processes, which are blocked on I/O, not CPU. Fix: Check iostat/%wa before assuming the CPU is the bottleneck.
• Pitfall: A package believed to be “pinned” gets upgraded by an automated process anyway. Why: The pin existed only as a convention (“don’t run apt upgrade on this”), not as an actual package-manager hold/versionlock. Fix: Use apt-mark hold / yum versionlock, which unattended-upgrade tooling itself respects.

Interview Questions

• A host shows 100% disk usage but du -sh /* only accounts for 60% of it — walk through the investigation. — Tests whether the deleted-but-open-file mechanism (not just the symptom) is understood.
• Explain the difference between a zombie process and a process stuck in D state, and what action (if any) resolves each. — Tests precise process-state knowledge versus vague familiarity.
• A load average of 8 on a 4-core box shows 20% CPU usage in top. What’s the first hypothesis, and how is it confirmed? — Tests whether I/O wait is the immediate, correct instinct over a CPU-scaling reflex.
• Why does chmod 777 get flagged in security reviews, and what’s the correct alternative for a genuinely shared directory? — Tests understanding of least-privilege sharing patterns (group + setgid) versus a blanket “never use 777” rule without an alternative.
• What’s the difference between systemd’s restart handling and SysV init’s, and why does it matter operationally? — Tests whether the candidate can reason about crash-loop protection, not just recite that systemd is “newer.”