Shell Scripting

TL;DR

• Shell scripts get less review scrutiny than application code (“just glue”), yet a single missing safety flag or unquoted variable can cause a silent, partial production failure.
• set -euo pipefail at the top of every script is the single highest-leverage habit: it converts silent failures into loud, immediate ones.
• Cron runs with a minimal environment (no login shell PATH/aliases) — scripts that work interactively can fail silently under cron without -e and an explicit PATH.
• Unquoted variable expansion ($var instead of "$var") is subject to word-splitting and globbing; quote by default, not as a fix after a bug appears.
• awk and jq outperform grep/cut/text-matching for anything involving field extraction, accumulation, or structured (JSON) data — they operate on structure, not raw text.
• Overlapping cron runs (a job that occasionally runs longer than its schedule interval) are a common, easily prevented source of “unpredictable” behavior; flock or a lock file fixes it.

Core Concepts

Bash safety flags

set -euo pipefail should appear at the top of essentially every production Bash script:

• -e (errexit): stops execution immediately when any command exits non-zero, instead of continuing with subsequent commands as if nothing failed. Exceptions: commands in a conditional (if cmd; then), part of &&/|| chains, or whose exit status is explicitly checked are exempt.
• -u (nounset): treats referencing an undefined variable as an error rather than silently substituting an empty string — catches typos in variable names immediately.
• -o pipefail: makes a pipeline’s exit status the last non-zero status among its stages (or zero if all succeed), rather than only the last command’s status. Without it, false | true reports success because only true’s exit code is checked.

This matters more under cron than interactively because cron executes with a minimal, non-login environment: no shell profile, no inherited PATH customizations, no aliases. A command that resolves correctly in an interactive shell can silently resolve to a different binary, or not be found at all, under cron — and without -e, the script continues past that failure instead of stopping.

Quoting and word-splitting

Unquoted variable expansion ($var) undergoes word-splitting (on IFS, whitespace by default) and pathname expansion (globbing). A value containing a space or a glob character (*, ?) silently splits into multiple shell “words” or expands as a pattern when unquoted. Quoting ("$var") preserves the value as a single, literal token regardless of its contents. This should be a default habit, not a fix applied after a filename-with-a-space bug surfaces — code that “worked in every test” commonly breaks the first time a real-world value contains whitespace.

Loops, glob patterns, and parsing ls

for file in $(ls *.txt) performs command substitution first, producing a single string, then word-splits that string on whitespace — any filename containing a space breaks into multiple loop iterations. for file in *.txt expands the glob pattern directly into separate, correctly-delimited tokens without an intermediate string stage, handling spaces and special characters correctly. The general rule — “don’t parse ls output” — extends to any case where a glob, an array, or find -print0 / read -d '' accomplishes the same iteration more robustly.

Functions, conditionals, and when to leave Bash

Bash functions and conditionals (if/case/[[ ]]) are straightforward for linear, sequential logic. The practical limit is reached when a script needs real data structures (nested structures beyond flat arrays/associative arrays), structured error handling beyond exit-code checks, or non-trivial control flow (retries with backoff, concurrent work). Past that point, a script rewritten in Python (or another general-purpose scripting language) is typically more maintainable and less fragile than continuing to extend Bash logic.

Cron scheduling and overlapping runs

Cron executes a command on a fixed schedule (* * * * * = minute, hour, day-of-month, month, day-of-week) regardless of whether the previous invocation has finished. A job scheduled every 5 minutes that occasionally takes longer than 5 minutes to complete will have two instances running concurrently, racing on the same resources — this produces behavior that looks like unpredictable scheduling but is actually overlap. A lock file, or wrapping the command in flock, ensures a new invocation checks whether the lock is held and exits immediately (rather than running concurrently) if a previous instance is still active.

Text-processing toolkit: grep, sed, awk, jq

• grep: line-based pattern filtering; outputs matching lines (or with -o, matching substrings).
• sed: stream-oriented line editing — substitution (s/pattern/replacement/), deletion, insertion — operates line by line without loading structure.
• awk: field-splitting (on whitespace or a specified delimiter, -F) plus a programming model with variables, conditionals, and per-record processing — critically, it can accumulate state (running sums, counts, per-key aggregation) across records in a single pass. A grep | cut pipeline can filter and extract fields but has no native way to accumulate a sum across lines; that requires an additional loop or tool that awk avoids.
• jq: a structural JSON processor — it parses JSON into its actual data model and lets filters (.field, [], select(), pipes) operate on that structure. Grepping raw JSON text matches key/value substrings textually and breaks the moment formatting, key order, or nesting changes even slightly; jq is immune to those cosmetic differences because it never treats JSON as flat text.

Bash vs. POSIX sh

$() vs. backticks

Command / Configuration Reference

#!/usr/bin/env bash
set -euo pipefail                 # stop on error, undefined var, or pipeline failure

"$var"                             # always quote expansions
"${arr[@]}"                        # quote array expansions to preserve element boundaries

for file in *.txt; do              # glob expansion, not `ls` parsing
  echo "$file"
done

find . -print0 | while IFS= read -r -d '' f; do   # null-delimited, handles any filename
  echo "$f"
done

flock -n /tmp/job.lock -c "/path/to/script.sh"     # prevent overlapping cron runs
# or inside the script:
exec 200>/tmp/job.lock
flock -n 200 || exit 1

*/5 * * * * /path/to/script.sh >> /var/log/script.log 2>&1   # crontab entry, every 5 minutes

awk -F'|' '{sum += $3} END {print sum}' file.log   # sum a pipe-delimited numeric field
grep 'ERROR' file.log | wc -l                       # count matching lines

jq -r '.items[] | select(.status=="failed") | .id' response.json   # structural JSON filter + extract

Common Pitfalls

• Pitfall: A deployment script continues past a failed step and completes a partial, broken deployment. Why: No set -e (or equivalent), so a non-zero exit status is silently ignored. Fix: Add set -euo pipefail at the top of every production script.
• Pitfall: for file in $(ls ...) breaks the first time a filename contains a space. Why: Command substitution followed by word-splitting breaks whitespace-containing filenames into multiple tokens. Fix: Use for file in *.txt (direct glob expansion) or find ... -print0 with read -d ''.
• Pitfall: An unquoted variable causes a command to fail or operate on the wrong target when it holds a path with a space. Why: Unquoted expansion is subject to word-splitting and globbing. Fix: Quote all variable expansions ("$var") by default.
• Pitfall: A cron job produces confusing, seemingly random behavior. Why: A slow run overlaps with the next scheduled invocation, and two instances race on shared resources. Fix: Wrap the job in flock or use a lock file to prevent concurrent execution.
• Pitfall: JSON parsing via grep/text-matching breaks after an upstream API change. Why: Text-matching depends on formatting, key order, and structure staying constant, none of which JSON guarantees. Fix: Use jq, which parses structurally and is unaffected by cosmetic formatting changes.
• Pitfall: A script “just works” when run manually but fails or behaves differently under cron. Why: Cron executes with a minimal, non-login environment — different PATH, no aliases, no shell profile. Fix: Set an explicit PATH (and any other required environment variables) inside the script rather than relying on the invoking shell’s environment.

Interview Questions

• A production script worked fine manually but did the wrong thing under cron — walk through the debugging process. — Tests whether cron’s minimal environment and a missing set -e are the immediate hypotheses.
• Why should shell variables always be quoted? — Tests whether word-splitting/globbing is understood as a mechanism, not just followed as a rule.
• Extract and sum a numeric column from a multi-gigabyte pipe-delimited log file from the command line — which tool, and why? — Tests whether awk’s single-pass field-splitting-plus-accumulation is understood versus reaching for a slower multi-tool pipeline.
• Why does for file in $(ls *.txt) behave differently from for file in *.txt? — Tests understanding of command substitution, word-splitting, and glob expansion as distinct mechanisms.
• What’s the difference between set -e and set -o pipefail, and why are they often used together? — Tests precise understanding of exit-status propagation through pipelines versus single commands.