4.3 Practical Shell Scripting & Automation

Practical Shell Scripting & Automation

In modern enterprise IT environments managing thousands of client workstations, virtual machines, and cloud server instances, manual point-and-click administration does not scale. Repetitive tasks—such as provisioning new employee user accounts, auditing dormant logins, monitoring free disk thresholds, rotating log archives, and enforcing security configurations—must be automated. Shell scripting provides the bridge between manual interactive commands and fully autonomous system administration.

---

1. Scripting Foundations for Enterprise IT Support

+-----------------------------------------------------------------------------+
|                   WHY SCRIPT? THE POWER OF IT AUTOMATION                    |
|                                                                             |
|   - REPEATABILITY:     Executes identical steps deterministically every time.|
|   - ERROR REDUCTION:   Eliminates typos and missed steps in manual workflows.|
|   - SCALABILITY:       Manages 10,000 servers as easily as managing 1.      |
|   - AUDITABILITY:      Scripts can be version-controlled in Git repositories.|
|   - UNATTENDED:        Executes off-hours maintenance via background timers. |
+-----------------------------------------------------------------------------+

Scripting vs. Compiled Programming

• Shell Scripts (Interpreted): Plaintext files containing sequences of commands executed line-by-line by a shell interpreter (e.g., /bin/bash or pwsh). Shell scripts require no compilation step, have instant development turnaround, and directly invoke underlying operating system binaries and APIs.
• Compiled Programs (C, C++, Rust, Go): Source code compiled by a compiler into standalone binary machine code. Compiled languages offer raw execution speed for intensive algorithms, but are heavier and less agile for routine OS administration and pipeline glue tasks.

---

2. Linux Bash Scripting Fundamentals

Bash (Bourne Again Shell) is the industry-standard scripting language for Linux systems administration.

+-----------------------------------------------------------------------------+
|                         ANATOMY OF A BASH SCRIPT                            |
|                                                                             |
|   #!/bin/bash                                 <-- 1. Shebang Line           |
|   # Automated Maintenance Script              <-- 2. Comment / Metadata     |
|   set -euo pipefail                           <-- 3. Defensive Shell Flags  |
|                                                                             |
|   LOG_DIR="/var/log/custom_app"               <-- 4. Variable Declaration   |
|   THRESHOLD=85                                                              |
|                                                                             |
|   if [ ! -d "$LOG_DIR" ]; then                <-- 5. Conditional Logic      |
|       echo "Error: Log dir missing!" >&2                                    |
|       exit 1                                  <-- 6. Exit Status Code       |
|   fi                                                                        |
+-----------------------------------------------------------------------------+

The Shebang Line & File Permissions

• The Shebang (#!): The very first line of an executable script. It informs the operating system kernel's execve() system call which interpreter binary to load:
  • #!/bin/bash: Standard Bash path on Linux.
  • #!/usr/bin/env bash: Portable shebang that searches the user's $PATH for the Bash binary.
• Execution Permissions: Before a script can be executed directly as ./script.sh, it must be granted execution permissions via chmod +x script.sh.
• Execution Invocation Methods:
  • ./script.sh: Runs the script in a dedicated child subshell. Requires chmod +x.
  • bash script.sh: Explicitly launches Bash to run the script in a subshell (does not require execute bit).
  • . script.sh or source script.sh: Executes the script directly inside the current active shell environment. Any variable changes or directory changes (cd) persist in the user's active session.

Variables, Quotes & Positional Parameters

• Variable Assignment Rules: No spaces are permitted around the = assignment operator (BACKUP_DIR="/opt/backup" is valid; BACKUP_DIR = "/opt/backup" throws a command-not-found error).
• Variable Expansion: Reference variables using $VAR_NAME or ${VAR_NAME} (echo "Target: ${BACKUP_DIR}_old").
• Quoting Mechanics:
  • "Double Quotes": Perform variable expansion and command substitution (echo "User is $USER on $(hostname)").
  • 'Single Quotes': Treat all characters strictly as raw literals with zero expansion (echo '$USER' outputs $USER).
  • $(command): Command Substitution—executes a command and captures its stdout into a variable (TODAY=$(date +%Y-%m-%d)).

Positional Parameters & Automatic Variables

Conditionals & Test Operators

Conditionals evaluate expressions using [ condition ] (standard POSIX test) or [[ condition ]] (extended Bash test supporting regular expressions and boolean && / ||).

# Bash If-Else Syntax:
if [ condition ]; then
    # commands
elif [ condition ]; then
    # commands
else
    # commands
fi

Bash vs. PowerShell Comparison Operators

Practical Linux IT Administration Scripts

1. Automated Disk Space Alert Script

#!/bin/bash
# Disk Space Monitoring & Alerting Script
set -euo pipefail

THRESHOLD=85
CURRENT_USAGE=$(df / | grep / | awk '{ print $5 }' | sed 's/%//g')

if [ "$CURRENT_USAGE" -ge "$THRESHOLD" ]; then
    MESSAGE="WARNING: Root filesystem disk usage is at ${CURRENT_USAGE}% on $(hostname)!"
    logger -t DISK_MONITOR "$MESSAGE"
    echo "$MESSAGE" | mail -s "Disk Space Alert: $(hostname)" sysadmin@example.com
else
    logger -t DISK_MONITOR "Disk space normal at ${CURRENT_USAGE}%."
fi

2. Bulk User Provisioning Script

#!/bin/bash
# Provisions users from a CSV file (Format: username,group,fullname)
set -euo pipefail

CSV_FILE="/root/new_users.csv"

if [ ! -f "$CSV_FILE" ]; then
    echo "Error: User list $CSV_FILE not found." >&2
    exit 1
fi

while IFS=',' read -r USERNAME GROUP FULLNAME; do
    # Skip empty lines or header
    [ -z "$USERNAME" ] || [ "$USERNAME" == "username" ] && continue

    # Create group if missing
    if ! getent group "$GROUP" > /dev/null 2>&1; then
        groupadd "$GROUP"
    fi

    # Create user with home directory and primary group
    if ! id "$USERNAME" > /dev/null 2>&1; then
        useradd -m -s /bin/bash -c "$FULLNAME" -g "$GROUP" "$USERNAME"
        echo "User $USERNAME created successfully."
    else
        echo "Notice: User $USERNAME already exists. Skipping."
    fi
done < "$CSV_FILE"

3. Log Rotation & Compression Script

#!/bin/bash
# Compresses and archives logs older than 30 days
set -euo pipefail

LOG_SRC="/var/log/custom_app"
ARCHIVE_DST="/backup/log_archives"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

mkdir -p "$ARCHIVE_DST"

# Find and package log files older than 30 days
OLD_LOGS=$(find "$LOG_SRC" -name "*.log" -type f -mtime +30)

if [ -n "$OLD_LOGS" ]; then
    tar -czf "${ARCHIVE_DST}/logs_archived_${TIMESTAMP}.tar.gz" $OLD_LOGS
    find "$LOG_SRC" -name "*.log" -type f -mtime +30 -delete
    logger -t LOG_ARCHIVER "Archived and purged old logs successfully."
fi

---

3. Windows PowerShell Scripting

PowerShell is an object-oriented shell and scripting framework built on Microsoft .NET.

+-----------------------------------------------------------------------------+
|                      POWERSHELL EXECUTION & OBJECT MODEL                    |
|                                                                             |
|   [Execution Policy Security Filter]                                        |
|   Get-ExecutionPolicy / Set-ExecutionPolicy RemoteSigned -Scope CurrentUser |
|                                                                             |
|   [Structured Object Pipeline]                                              |
|   Get-Service | Where-Object Status -eq 'Stopped' | Start-Service          |
|   (Passes System.ServiceProcess.ServiceController .NET Objects)             |
+-----------------------------------------------------------------------------+

PowerShell Execution Policy

To prevent users from inadvertently executing malicious scripts downloaded from the web, PowerShell enforces an Execution Policy. Note: The Execution Policy is a safety barrier against accidental execution, not a cryptographic security boundary.

• Restricted: Default on client Windows. Blocks all script (.ps1) execution; permits interactive commands only.
• AllSigned: Only scripts digitally signed by a trusted certificate publisher can execute.
• RemoteSigned: Default on Windows Server. Scripts written locally run without signatures; scripts downloaded from the Internet (bearing an alternate data stream mark of the web) must be digitally signed.
• Unrestricted: Runs all scripts; prompts a warning before executing downloaded internet scripts.
• Bypass: Nothing is blocked and no warning prompts are shown (commonly used in automated deployment pipelines).
• Configuring Policy:

  # Check current execution policy across scopes:
  Get-ExecutionPolicy -List
  
  # Set policy to RemoteSigned for the current user:
  Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser -Force
  

PowerShell Script Syntax, Variables & Collections

• Variables: Prefixed with $ ($ComputerName = $env:COMPUTERNAME).
• Arrays: Defined using @(item1, item2, item3) or comma-separated lists ($Servers = @("DC01", "WEB01", "DB01")). Access via index: $Servers[0].
• Hashtables (Dictionaries): Defined using @{ Key = 'Value' } ($User = @{ Name = 'Alice'; Role = 'Sysadmin'; Department = 'IT' }). Access via $User.Role.
• Environment Variables: Accessed via $env:VARIABLE_NAME (e.g., $env:USERPROFILE, $env:TEMP, $env:PATH).

Pipeline Filtering: Where-Object & ForEach-Object

• Where-Object (Alias: ?): Filters objects in the pipeline based on conditional evaluations.
• ForEach-Object (Alias: %): Iterates over every individual object in the pipeline, referencing the current item using $_ or $PSItem.

Practical Windows Administration Scripts

1. Bulk CSV User Creation Script

# Bulk Local User Provisioning from CSV
# CSV Columns: Username, FullName, Department, Description

$CsvPath = "C:/Scripts/NewUsers.csv"

if (-not (Test-Path -Path $CsvPath)) {
    Write-Error "CSV input file not found at $CsvPath"
    exit 1
}

$Users = Import-Csv -Path $CsvPath

foreach ($User in $Users) {
    if (-not (Get-LocalUser -Name $User.Username -ErrorAction SilentlyContinue)) {
        $TempPassword = ConvertTo-SecureString "P@ssw0rd1234!" -AsPlainText -Force
        New-LocalUser -Name $User.Username `
                      -FullName $User.FullName `
                      -Description $User.Description `
                      -Password $TempPassword `
                      -PasswordNeverExpires:$false
        
        Write-Host "Created user: $($User.Username)" -ForegroundColor Green
    } else {
        Write-Warning "User $($User.Username) already exists. Skipping."
    }
}

2. Inactive User Account Audit & Remediation

# Audits local accounts inactive for over 90 days and disables them
$ThresholdDate = (Get-Date).AddDays(-90)

$InactiveUsers = Get-LocalUser | Where-Object {
    $_.Enabled -eq $true -and 
    $_.LastLogon -ne $null -and 
    $_.LastLogon -lt $ThresholdDate
}

foreach ($Account in $InactiveUsers) {
    Disable-LocalUser -Name $Account.Name
    Write-EventLog -LogName Application -Source "IT_Audit" -EventId 2001 `
                   -EntryType Warning `
                   -Message "Disabled inactive local account: $($Account.Name)"
    Write-Host "Disabled dormant account: $($Account.Name)" -ForegroundColor Yellow
}

3. Service Auto-Recovery Watchdog

# Watchdog script to monitor and auto-restart critical Windows services
$CriticalServices = @("Spooler", "wuauserv", "W32Time")

foreach ($ServiceName in $CriticalServices) {
    $Service = Get-Service -Name $ServiceName -ErrorAction SilentlyContinue
    if ($Service -and $Service.Status -ne 'Running') {
        Write-Warning "Service $ServiceName is stopped. Attempting restart..."
        try {
            Start-Service -Name $ServiceName -ErrorAction Stop
            Write-Host "Successfully restarted $ServiceName." -ForegroundColor Green
        } catch {
            Write-Error "Failed to start $ServiceName. Reason: $($_.Exception.Message)"
        }
    }
}

---

4. Task Scheduling & Automation (Cron, Systemd Timers & Task Scheduler)

Automating scripts requires scheduling daemons that execute jobs at precise calendar times, recurring intervals, or upon specific system events.

+-----------------------------------------------------------------------------+
|                        CRONTAB 5-FIELD SYNTAX FORMAT                        |
|                                                                             |
|   *       *       *       *       *       /path/to/script.sh                |
|   |       |       |       |       |                                         |
|   |       |       |       |       +---- Day of Week (0 - 7) (0/7 = Sunday)  |
|   |       |       |       +------------ Month (1 - 12)                      |
|   |       |       +-------------------- Day of Month (1 - 31)               |
|   |       +---------------------------- Hour (0 - 23)                       |
|   +------------------------------------ Minute (0 - 59)                     |
+-----------------------------------------------------------------------------+

Linux cron Daemon & Crontab Syntax

The crond daemon wakes up every 60 seconds, evaluates user and system crontabs, and executes any scheduled commands matching the current minute.

• Crontab Commands:
  • crontab -e: Edits the current user's crontab file using the default editor.
  • crontab -l: Displays the current user's crontab entries.
  • crontab -r: Purges and removes the current user's crontab file.
  • sudo crontab -u alice -e: Allows administrators to edit another user's crontab.

Crontab Schedule Expression Examples

Linux systemd Timers

Modern enterprise Linux systems increasingly utilize systemd Timers as a robust alternative to cron:

• Monotonic Timers: Can trigger tasks relative to boot time (OnBootSec=15min) or relative to previous task completion (OnUnitActiveSec=1h).
• Real-Time Calendar Timers: Trigger at exact calendar dates (OnCalendar=*-*-* 04:00:00).
• Benefits over Cron: Tight integration with systemd cgroups, CPU/memory resource throttling, automatic logging to journalctl, and failure dependency tracking.
• Management: systemctl list-timers.

Windows Task Scheduler (taskschd.msc)

Windows Task Scheduler automates tasks across four core components:

1. Triggers: When the task executes (Schedule: Daily/Weekly; System Event: At Startup, On User Logon, On specific Event ID).
2. Actions: What the task does (e.g., Program: powershell.exe, Arguments: -ExecutionPolicy Bypass -File C:/Scripts/audit.ps1).
3. Conditions: Safeguards (Run only if computer is connected to AC power, Start only if network is available).
4. Settings: Operational behaviors (Restart if task fails, Stop task if it runs longer than 2 hours).

Command-Line Scheduling with schtasks.exe and PowerShell

:: Create a daily task running at 3:00 AM under the SYSTEM account:
schtasks /create /tn "DailyLogCleanup" /tr "powershell.exe -ExecutionPolicy Bypass -File C:/Scripts/cleanup.ps1" /sc daily /st 03:00 /ru "SYSTEM"

:: Query task status:
schtasks /query /tn "DailyLogCleanup" /fo list /v

# PowerShell Scheduled Task Creation:
$Action = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-File C:/Scripts/backup.ps1"
$Trigger = New-ScheduledTaskTrigger -Daily -At 2:00AM
$Principal = New-ScheduledTaskPrincipal -UserId "NT AUTHORITY\SYSTEM" -LogonType ServiceAccount
Register-ScheduledTask -TaskName "EnterpriseBackup" -Action $Action -Trigger $Trigger -Principal $Principal

---

5. Defensive Scripting, Structured Logging & Exit Codes

Unattended enterprise scripts must handle unexpected errors gracefully, validate inputs, and report standard exit codes.

Standard POSIX Exit Codes

Defensive Bash Header

Enterprise Bash scripts should always begin with strict safety flags:

#!/bin/bash
# -e: Exit immediately if any command returns a non-zero exit status
# -u: Treat unset/uninitialized variables as an error and exit immediately
# -o pipefail: Pipeline returns exit status of the rightmost command to fail
set -euo pipefail