3.1 Adversarial Evasion Attacks and Perturbations

3.1 Adversarial Evasion Attacks and Perturbations

Adversarial machine learning represents a fundamental shift in how security professionals analyze software vulnerabilities. While traditional software security identifies flaws in control flow, memory safety, or input validation logic, machine learning security addresses vulnerabilities inherent in high-dimensional statistical optimization. Adversarial evasion attacks (often called exploratory attacks) occur exclusively at inference time (test time). In an evasion attack, the target model's training process has completed, its parameters $\theta$ are frozen, and the attacker crafts a maliciously modified input $x_{adv} = x + \delta$ designed to force an erroneous prediction while keeping the perturbation $\delta$ imperceptible or semantically inconsequential.

Evasion attacks fall into two operational categories:

1. Untargeted Evasion: The attacker seeks any incorrect classification: $f(x_{adv}) \neq y_{true}$. For example, forcing an intrusion detection system (IDS) to classify a malicious command-and-control beacon as any benign protocol.
2. Targeted Evasion: The attacker forces the model to output a specific malicious label selected in advance: $f(x_{adv}) = y_{target}$ where $y_{target} \neq y_{true}$. For example, inducing an autonomous vehicle's vision model to classify a Stop Sign specifically as a Speed Limit 45 sign.

---

Adversarial Threat Models: Access and Knowledge

Security engineers evaluate adversarial robustness against formalized threat models that define what an attacker knows and what interfaces they can access:

In score-based black-box attacks, adversaries exploit output confidence scores to approximate gradients numerically using finite differences ($[f(x + h \cdot u) - f(x)] / h$). In decision-based (label-only) attacks, adversaries observe only discrete class labels, walking along the classification boundary (e.g., using HopSkipJump) to find the minimal perturbation crossing the decision threshold.

---

Mathematical Bounding: $L_p$ Norms in Adversarial ML

To ensure an adversarial input remains valid, functional, or stealthy, perturbations are constrained within an allowable perturbation set $\Delta$ defined by an $L_p$ norm. Given a perturbation vector $\delta = x_{adv} - x \in \mathbb{R}^d$, the $L_p$ norm measures its magnitude:

$\|\delta\|_p = \left( \sum_{i=1}^d |\delta_i|^p \right)^{1/p}$

CompTIA SecAI+ emphasizes three primary $L_p$ norms used in security evaluations:

1. $L_0$ Norm (Sparsity Metric)

$\|\delta\|_0 = \sum_{i=1}^d \mathbb{I}(\delta_i \neq 0)$

• Mechanism: The $L_0$ pseudo-norm counts the number of altered features or dimensions, regardless of how large each individual alteration is. Here, $\mathbb{I}$ is the indicator function.
• Security Application: Ideal for attacks where an adversary can only manipulate a few specific inputs. Examples include modifying only 5 pixels in a high-resolution biometric image, changing 3 specific words in a phishing email to evade an NLP spam filter, or altering 4 specific registry key calls in an endpoint detection and response (EDR) behavioral vector.

2. $L_2$ Norm (Euclidean Distance / Energy Metric)

$\|\delta\|_2 = \sqrt{\sum_{i=1}^d \delta_i^2}$

• Mechanism: Computes standard geometric distance, measuring the total accumulated energy of the perturbation across all dimensions.
• Security Application: Common in physical signal processing, audio speech-to-text evasion (e.g., adding low-amplitude white noise across acoustic frequencies to trick voice assistants), and biometric facial verification where total structural distortion must remain small.

3. $L_\infty$ Norm (Chebyshev Distance / Worst-Case Peak Metric)

$\|\delta\|_\infty = \max_{1 \le i \le d} |\delta_i|$

• Mechanism: Measures the maximum absolute change in any single feature dimension. If $|\delta|\infty \le \epsilon$, every single feature $i$ satisfies $|x{adv, i} - x_i| \le \epsilon$.
• Security Application: The most widespread benchmark in computer vision and sensory machine learning. When images are normalized to pixel values between $0.0$ and $1.0$ (or $0$ to $255$), choosing $\epsilon = 8/255 \approx 0.031$ ensures that no color channel of any pixel changes by more than 8 intensity levels out of 255—rendering the perturbation completely imperceptible to human visual inspection.

---

Canonical Evasion Attack Algorithms

Adversarial perturbations are computed by optimizing the model's loss function $\mathcal{L}(\theta, x, y)$ with respect to the input $x$. Rather than adjusting weights $\theta$ to minimize loss (standard training), the adversary freezes $\theta$ and adjusts input $x$ to maximize loss.

Standard ML Training:     min_θ  L(θ, x, y)       [Updates weights θ]
Adversarial Evasion:      max_δ  L(θ, x + δ, y)   [Updates input x, subject to ||δ||_p ≤ ε]

Fast Gradient Sign Method (FGSM)

Introduced by Goodfellow et al. (2014), FGSM is a fast, one-step white-box attack designed for $L_\infty$ bounded perturbations:

$x_{adv} = x + \epsilon \cdot \text{sign}\left( \nabla_x \mathcal{L}(\theta, x, y) \right)$

• Mechanism: The attacker calculates the analytical gradient of the classification loss with respect to the input vector $x$. The sign() function extracts the polarity ($+1$ or $-1$) of each dimension's gradient. Multiplying by $\epsilon$ takes a single uniform step of magnitude $\epsilon$ in the direction that most rapidly increases the model's loss.
• Computational Complexity: $\mathcal{O}(1)$ backward passes (extremely fast, requires only a single backpropagation).
• Limitations: Because it relies on a first-order linear approximation of the loss surface around $x$, FGSM frequently under-optimizes on complex, highly curved loss surfaces. It represents a coarse attack and is easily defended by adversarial training.

Projected Gradient Descent (PGD)

Introduced by Madry et al. (2017), PGD is an iterative multi-step attack that represents the standard empirical benchmark for worst-case first-order adversaries under $L_\infty$ or $L_2$ bounds:

$x^{(t+1)} = \Pi_{x + \mathcal{S}}\left( x^{(t)} + \alpha \cdot \text{sign}\left( \nabla_x \mathcal{L}(\theta, x^{(t)}, y) \right) \right)$

• Mechanism:
  1. Random Initialization: Rather than starting at $x$, PGD initializes $x^{(0)}$ with a uniform random perturbation within the $\epsilon$-ball $\mathcal{S} = { \delta : |\delta|_\infty \le \epsilon }$. This random restart prevents the attack from becoming trapped in flat local loss plateaus.
  2. Iterative Steps: At each step $t$, the attack computes the loss gradient and advances by a small step size $\alpha$ (typically $\alpha = \epsilon / 4$ or $\epsilon / 10$).
  3. Projection Operator ($\Pi$): After each step, the projection operator clips the perturbed values back into the bounded $\epsilon$-ball around $x$ and clamps them to the valid input range (e.g., $[0, 1]$ or $[0, 255]$).
• Computational Complexity: Requires $K$ backpropagation passes (typically $K = 10, 20, 40$ iterations).

Carlini-Wagner (C&W) Attack

Formulated by Nicholas Carlini and David Wagner (2017), the C&W attack is an optimization-based attack designed to defeat defensive distillation and heuristic defenses. Instead of directly maximizing cross-entropy loss, C&W reframes the attack as a constrained optimization problem:

$\min_{\delta} \left( \|\delta\|_p + c \cdot f(x + \delta) \right)$

Where $f(x')$ is an objective function designed such that $f(x') \le 0$ if and only if the model misclassifies $x'$ into the target class $t$:

$f(x') = \max\left( \max_{i \neq t} Z(x')_i - Z(x')_t, -\kappa \right)$

• Here, $Z(x')$ represents the raw unnormalized logits (pre-softmax outputs), and $\kappa \ge 0$ controls the attack's confidence margin.
• To enforce box constraints ($x + \delta \in [0, 1]$) without clipping artifacts, C&W applies a change of variables: $x + \delta = \frac{1}{2}(\tanh(w) + 1)$, optimizing over unconstrained variable $w$.
• Impact: C&W finds minimal-distortion adversarial examples ($L_0, L_2, L_\infty$) and systematically broke defensive distillation by operating directly on logits rather than vanishing softmax gradients.

---

Adversarial Transferability and Physical-World Exploits

One of the most dangerous empirical properties of adversarial examples is transferability: an adversarial sample $x_{adv}$ generated on a known, white-box surrogate model $M_A$ will frequently fool an entirely different, unseen target model $M_B$, even when $M_B$:

• Uses a completely different architecture (e.g., $M_A$ is a Convolutional Neural Network like ResNet, while $M_B$ is a Vision Transformer or Random Forest).
• Was trained on a disjoint subset of training data.
• Operates in a proprietary, black-box cloud API.

Transferability occurs because deep learning models trained on similar tasks learn fundamentally similar decision boundaries, projecting data onto shared, low-dimensional manifold representations. Attack vectors aligned along high-loss dimensions of $M_A$ often align with high-loss directions of $M_B$.

Physical-World Adversarial Examples

In cybersecurity and physical infrastructure, attackers cannot inject digital pixel arrays directly into network pipelines. Instead, they must survive physical capture through cameras, microphones, or radio sensors:

• Adversarial Patches (Brown et al., 2017): Highly textured, colorful visual stickers designed to dominate a classifier's attention. Placed anywhere in the camera's field of view, an adversarial patch can force an object detector to fail to detect a person (cloaking) or misclassify a stop sign as a toaster, regardless of camera angle or distance.
• Expectation Over Transformation (EOT) (Athalye et al., 2018): Physical attacks must withstand environmental noise (rotations, scaling, lighting changes, camera lens distortions, distance). EOT optimizes the perturbation across a distribution of transformations $T$: $\max_\delta \mathbb{E}_{t \sim T}\left[ \mathcal{L}(\theta, t(x + \delta), y) \right]$ EOT allows adversaries to 3D-print physical adversarial objects (such as a toy turtle classified as a rifle from any viewing angle).
• Malware PE Evasion via Byte Padding: In endpoint security, ML-based antivirus (ML-AV) inspects Portable Executable (PE) binaries. Attackers generate adversarial byte perturbations and inject them into unused binary regions (e.g., the DOS header slack space, new non-functional PE sections, or overlay bytes appended to the end of the file). Because these regions are never executed by the Windows PE loader, the malware's malicious functionality remains intact while its global byte histogram and structural feature vectors shift into the ML-AV's "benign" classification space.

---

Worked Scenario: Evading Autonomous Vision and Traffic Sign Classification

Incident Context

A municipal transit authority deploys an autonomous shuttle fleet equipped with a deep learning vision system. The perception stack uses a convolutional neural network ($f_\theta$) to classify roadside signs. During a scheduled red team exercise, security engineers evaluate the shuttle's vulnerability to physical evasion.

Original Image (x):          Stop Sign (Ground Truth: Label 14)
Target Malicious Class (y*): Speed Limit 45 (Label 4)
Perturbation Bound:          L_infinity <= 12/255
Physical Medium:             Adversarial Sticker Overlay on Sign Face

Execution Steps

1. Surrogate Model Selection: The red team trains a surrogate ResNet-34 classifier using publicly available street sign datasets (GTSRB), matching the input dimensions ($224 \times 224 \times 3$) of the target vehicle's vision system.
2. Optimization via PGD with EOT: Using PGD, the engineers optimize a localized perturbation sticker $\delta$ over 40 iterations with a step size $\alpha = 2/255$. To survive physical conditions, they apply EOT across simulated brightness variations ($0.7 \times$ to $1.3 \times$), perspective tilts ($-15^\circ$ to $+15^\circ$), and blur kernels: $\delta^{(t+1)} = \Pi_{\mathcal{S}}\left( \delta^{(t)} - \alpha \cdot \text{sign}\left( \nabla_\delta \mathbb{E}_{t \sim T} [\mathcal{L}_{CE}(f_{surr}(t(x + \delta)), \text{Target: 45})] \right) \right)$
3. Physical Deployment: The optimized perturbation is printed using a standard commercial inkjet printer onto weather-resistant matte vinyl and affixed across the center face of a standard 30-inch octagonal Stop Sign.
4. Inference Outcome: As the shuttle approaches the intersection at 25 MPH, its camera captures the sign. The onboard vision model classifies the sign as Speed Limit 45 with 94.2% confidence. The shuttle accelerates through the intersection without braking, confirming a critical physical evasion vulnerability.

---

Exam Traps and Architectural Pitfalls

• Trap 1: Confusing Evasion with Poisoning: Exam questions frequently test whether an attack occurs at training time or inference time. Evasion attacks occur strictly at inference/test time on frozen model parameters $\theta$. If the question describes an attacker injecting corrupted samples into a training database or altering weights, it is a data poisoning or backdoor attack, not evasion.
• Trap 2: Misinterpreting $L_0$ as Bounding Magnitude: Candidates often assume $L_0$ limits how drastically a feature can change. $L_0$ measures sparsity (the count of altered coordinates), not the size of the change. Modifying a single pixel from pitch black ($0$) to pure white ($255$) has an $L_0$ norm of exactly $1$, even though the $L_\infty$ change is at maximum ($255$).
• Trap 3: The Black-Box Immunity Fallacy: Organizations frequently believe keeping their model architecture, weights, and APIs private provides sufficient protection against evasion. This is false: adversarial transferability allows adversaries to train local white-box surrogates and transfer attacks with high success rates, while score-based and decision-based black-box algorithms can extract decision boundaries without ever viewing model code.