3.6 Linux as a Virtualization Guest & Cloud Instances (102.6)

3.6 Linux as a Virtualization Guest & Cloud Instances (102.6)

Quick Summary: Modern Linux systems frequently run as guest virtual machines (VMs) or cloud instances. Understanding the distinction between Type 1 (bare-metal) and Type 2 (hosted) hypervisors, deploying guest integration tools (open-vm-tools, qemu-guest-agent, VirtIO drivers), and detecting virtualization environments using utilities like systemd-detect-virt is fundamental. Additionally, cloud instance bootstrapping via cloud-init and instance metadata services (IMDS), as well as /etc/machine-id lifecycle management, are essential LPIC-1 objectives.

---

1. Hypervisor Classifications and Architecture

A hypervisor (or Virtual Machine Monitor, VMM) creates and manages virtual machines by abstracting physical CPU, memory, storage, and networking resources.

  TYPE 1 (Bare-Metal Hypervisor)         TYPE 2 (Hosted Hypervisor)
  ┌──────────────────────────────┐       ┌──────────────────────────────┐
  │  Guest VM 1  │  Guest VM 2   │       │  Guest VM 1  │  Guest VM 2   │
  ├──────────────┴──────────────┤       ├──────────────┴──────────────┤
  │  Hypervisor (KVM/ESXi/Xen)   │       │ Hypervisor (VirtualBox/WS)   │
  ├──────────────────────────────┤       ├──────────────────────────────┤
  │    Bare-Metal Physical HW    │       │    Host OS (Linux/Windows)   │
  │     (CPU, RAM, NIC, SAN)     │       ├──────────────────────────────┤
  └──────────────────────────────┘       │    Bare-Metal Physical HW    │
                                         └──────────────────────────────┘

Hypervisor Taxonomy Matrix

---

2. Guest Integration Tools & Paravirtualization (VirtIO)

Running Linux inside a VM requires guest integration software to bridge communication between the guest OS and the host hypervisor.

Key Guest Integration Daemons

1. open-vm-tools: The open-source implementation of VMware Tools for Linux guests running on VMware ESXi / Workstation. Managed via the vmtoolsd service; handles guest clock synchronization, automated graceful shutdown, and host-guest performance telemetry.
2. qemu-guest-agent: A daemon running inside KVM/QEMU guests (qemu-ga service) communicating over a dedicated virtio-serial character device. Enables the host hypervisor to freeze filesystems for online snapshots, set user passwords, and query guest IP addresses.
3. VirtualBox Guest Additions: Kernel modules (vboxguest, vboxsf, vboxvideo) providing shared folders, seamless mouse integration, and dynamic display resizing.

VirtIO Paravirtualized Drivers

Standard full virtualization emulates legacy physical hardware (e.g., Intel e1000 NIC or IDE disk controllers), introducing high CPU translation overhead. VirtIO provides paravirtualized device drivers where the Linux guest kernel is aware it is running in a virtual machine, enabling zero-copy shared-memory ring buffers with near-native bare-metal speed.

• virtio_net: High-throughput paravirtualized network interface driver.
• virtio_blk / virtio_scsi: High-performance block storage drivers.
• virtio_balloon: Dynamic memory ballooning driver allowing the host to reclaim unused RAM from the guest without rebooting.
• virtio_console: Host-to-guest serial communication channel.

---

3. Detecting Virtualization Environments

Linux provides specialized utilities and /sys interfaces to identify whether the operating system is running on bare metal, inside a virtual machine, or within a container.

1. systemd-detect-virt

The standard systemd utility to query virtualization technology.

# Detect any virtualization (outputs technology name and returns exit code 0 if virtualized)
systemd-detect-virt

kvm

Output Values Reference

• Hypervisors: kvm, qemu, vmware, oracle (VirtualBox), microsoft (Hyper-V/WSL), xen, amazon.
• Containers: lxc, docker, podman, systemd-nspawn, openvz, wsl.
• Bare Metal: Outputs none and exits with return code 1.

Command Flags

• -v / --vm: Detects only virtual machines (ignores containers; exits with non-zero if in a container or on bare metal).
• -c / --container: Detects only container environments (ignores VMs).
• -q / --quiet: Suppresses text output; only returns exit status (0 = virtualized, non-zero = not virtualized).

2. Alternative Detection Utilities

# 1. virt-what (Heuristic shell tool)
sudo virt-what
# Output: kvm

# 2. lscpu (Inspect Hypervisor vendor field)
lscpu | grep -i hypervisor
# Output: Hypervisor vendor: KVM

# 3. dmidecode (DMI / SMBIOS hardware table inspection)
sudo dmidecode -s system-manufacturer
sudo dmidecode -s system-product-name
# Output: QEMU / VMware, Inc. / innotek GmbH / Microsoft Corporation

# 4. hostnamectl (System summary)
hostnamectl | grep Virtualization
# Output: Virtualization: kvm

# 5. Direct sysfs DMI query (No root required)
cat /sys/class/dmi/id/sys_vendor
cat /sys/class/dmi/id/product_name

---

4. Cloud Instance Provisioning & cloud-init

When deploying instances across public cloud providers (AWS EC2, Google Cloud Platform, Microsoft Azure, OpenStack), virtual machines are launched from generic base images. cloud-init is the standard multi-distribution package that initializes and customizes the cloud instance during its first boot.

  Cloud Hypervisor (AWS / GCP / Azure / OpenStack)
         │
         ▼
  Provides Link-Local Metadata Service (IMDS)
  http://169.254.169.254/latest/meta-data/
         │
         ▼
  Guest OS Boots ──> cloud-init Service Starts
         │
         ├─ 1. Fetches Hostname, Public SSH Keys, Network Config
         ├─ 2. Ingests User-Data Script (#cloud-config YAML or Bash)
         ├─ 3. Resizes Root Partition & Filesystem (growpart)
         └─ 4. Generates Host SSH Keys & Configures Default User

Instance Metadata Service (IMDS)

Cloud platforms expose an internal link-local HTTP endpoint accessible only from within the running instance at IPv4 address http://169.254.169.254.

# Query instance metadata on AWS / OpenStack
curl http://169.254.169.254/latest/meta-data/instance-id
curl http://169.254.169.254/latest/meta-data/public-ipv4
curl http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key

User-Data Configuration

Users provide initialization instructions via User-Data, which can be formatted as a standard Bash script or a #cloud-config YAML declaration.

#cloud-config
hostname: web-node-01
fqdn: web-node-01.example.com
manage_etc_hosts: true

users:
  - name: sysadmin
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: sudo, wheel
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... sysadmin@mgmt

packages:
  - nginx
  - curl

runcmd:
  - systemctl enable --now nginx
  - echo '<h1>Deployed via cloud-init</h1>' > /var/www/html/index.html

Key cloud-init File Paths

• Main Configuration: /etc/cloud/cloud.cfg and /etc/cloud/cloud.cfg.d/*.cfg
• Instance Cache Data: /var/lib/cloud/instance/
• Execution Log Files: /var/log/cloud-init.log and /var/log/cloud-init-output.log

---

5. Unique Machine Identity (/etc/machine-id)

Every systemd-based Linux operating system requires a unique 128-bit hexadecimal identifier stored in /etc/machine-id (and mirrored/symlinked at /var/lib/dbus/machine-id).

Operational Importance

• DHCP Client Identifiers: Used by systemd-networkd and NetworkManager to generate DUIDs for DHCP address leases.
• Systemd Journal Logging: Indexes local journal logs in /var/log/journal/<machine-id>/.
• Application Clustering & Telemetry: Ensures unique node identification across distributed clusters.

The Golden Image / VM Cloning Pitfall

⚠️ LPIC-1 Trap: When creating a template or "golden master" VM image to clone multiple instances, administrators must truncate or delete /etc/machine-id. If multiple cloned VMs boot with the identical machine-id, they will request the identical DHCP IP address from the network router, overwrite each other's journal logs, and cause cluster identity collisions!

# Best Practice: Sanitizing machine-id prior to templating a VM
sudo truncate -s 0 /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id

# Regenerating a new unique machine ID upon first boot of a clone
sudo systemd-machine-id-setup
# OR for D-Bus
sudo dbus-uuidgen --ensure

---

6. SSH Host Keys: The Other Half of Template Sanitisation

/etc/machine-id is the identity file most candidates remember, but the 102.6 Terms and Utilities list names SSH host keys right alongside it — and they fail in a noisier, more dangerous way when they are cloned.

What Host Keys Are

An OpenSSH server generates a persistent key pair per algorithm at first start and stores them in /etc/ssh/:

These are host keys, not user keys. They identify the server to connecting clients, which is the opposite direction from ~/.ssh/id_ed25519, which identifies the user to the server. On first connection the client records the server's public host key in ~/.ssh/known_hosts.

Why Cloning Them Is a Problem

If a golden image is captured with its host keys in place, every VM cloned from that image presents the identical host key. Two consequences follow:

1. Security. Server identity verification collapses. Any of those clones — or an attacker who extracts the private key from the published image — can impersonate any other. A man-in-the-middle against one clone is undetectable, because the key genuinely matches what the client has pinned.
2. Operations. A client that already trusts clone A and then connects to clone B at a recycled IP address receives a matching key where it expected a different one, or a mismatching one after re-provisioning, producing the familiar WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! refusal.

Sanitising a Template

# On the golden image, immediately before shutdown: remove host keys and machine identity
# rm -f /etc/ssh/ssh_host_*
# truncate -s 0 /etc/machine-id

# Debian/Ubuntu: also clear the D-Bus machine id if it is a real file rather than a symlink
# rm -f /var/lib/dbus/machine-id

# Verify nothing remains
# ls -l /etc/ssh/ssh_host_* 2>/dev/null || echo "host keys cleared"

On first boot of each clone, regeneration happens automatically:

• systemd hosts run sshd-keygen.service (or the distribution's ssh-keygen -A hook) before sshd.service starts.
• cloud-init performs the same job through its ssh module, and can additionally publish the fresh host key fingerprints to the instance console log so operators can verify them out of band.
• Manual regeneration is ssh-keygen -A, which creates one key of each supported type only where none exists.

# Force regeneration by hand
# ssh-keygen -A

# Read back the fingerprints a client will be asked to trust
$ for k in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf "$k"; done
256 SHA256:9pQ7…f1Kc root@web01 (ED25519)
3072 SHA256:2mDx…v8Ra root@web01 (RSA)

Exam Rule: Three items must be cleared before a Linux system is captured as a template or golden image: SSH host keys (/etc/ssh/ssh_host_*), the machine ID (/etc/machine-id, and the D-Bus machine id where it is a separate file), and any persistent network interface naming or DHCP client lease state. Leaving guest drivers installed is correct and expected — the paravirtualised guest drivers (VirtIO block and network modules, open-vm-tools, qemu-guest-agent) are exactly the components a clone needs, so they belong in the image. Identity belongs to the instance; drivers belong to the image.