3.4 Batch Workloads: Jobs & CronJobs

Batch Workloads: Jobs & CronJobs

While Services and Deployments run long-lived, continuous processes, batch workloads run to completion and exit. Kubernetes provides Jobs for finite batch operations (ETL pipelines, database schema migrations, video transcoding) and CronJobs for recurring, time-based scheduled tasks (nightly backups, report generation, cache warming).

---

1. Kubernetes Jobs Architecture

A Job creates one or more Pods and ensures that a specified number of them successfully terminate with exit code 0.

+-----------------------------------------------------------------------------------------+
|                                 JOB EXECUTION MODES                                     |
|                                                                                         |
|   1. NON-INDEXED JOB (Work Queue)                                                       |
|      completions: 3, parallelism: 2                                                     |
|      [Pod 1 (Worker)] ---> Exit 0                                                       |
|      [Pod 2 (Worker)] ---> Exit 0                                                       |
|      [Pod 3 (Worker)] ---> Exit 0 (Job Complete: 3/3 Succeeded)                         |
|                                                                                         |
|   2. INDEXED JOB (Static Partitioning)                                                  |
|      completions: 4, parallelism: 2, completionMode: Indexed                            |
|      [Pod Index 0: JOB_COMPLETION_INDEX=0] (Processes chunk 0-250MB)                    |
|      [Pod Index 1: JOB_COMPLETION_INDEX=1] (Processes chunk 250-500MB)                  |
|      [Pod Index 2: JOB_COMPLETION_INDEX=2] (Processes chunk 500-750MB)                  |
|      [Pod Index 3: JOB_COMPLETION_INDEX=3] (Processes chunk 750MB-1GB)                  |
+-----------------------------------------------------------------------------------------+

Core Job Specification Fields

apiVersion: batch/v1
kind: Job
metadata:
  name: db-backup-job
spec:
  completions: 3
  parallelism: 2
  backoffLimit: 4
  activeDeadlineSeconds: 1200
  ttlSecondsAfterFinished: 300
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: backup
          image: postgres:16-alpine
          command: ["pg_dump", "-h", "db-svc", "-U", "admin", "-f", "/backup/data.sql"]

Detailed Field Mechanics

1. completions: Total number of Pods that must exit with 0 for the Job to be marked complete. Defaults to 1.
2. parallelism: Maximum number of Pods allowed to run concurrently at any given moment. Defaults to 1.
3. backoffLimit: Number of retries before marking the Job as Failed. Defaults to 6. Pod retries use exponential backoff (10s, 20s, 40s, and so on), capped at 6 minutes between retries.
4. activeDeadlineSeconds: Hard maximum duration for the Job. If the Job does not finish within this time, all active Pods are terminated and the Job status is set to DeadlineExceeded.
5. ttlSecondsAfterFinished: Automatically deletes the completed or failed Job and its associated Pods after the specified number of seconds, preventing etcd clutter.
6. restartPolicy: Must be either OnFailure or Never. It cannot be Always (which is reserved for continuous services).
   • OnFailure: The container restarts inside the same Pod (retaining the node allocation).
   • Never: The container is not restarted; kubelet fails the Pod, and the Job controller creates a brand new Pod on a potentially different node.

Indexed Jobs (completionMode: Indexed)

When completionMode: Indexed is set, Kubernetes injects an environment variable JOB_COMPLETION_INDEX (ranging from 0 to completions - 1) into each Pod. This allows worker Pods to know their exact shard/partition without coordinating via an external message broker.

---

2. Kubernetes CronJobs Architecture

A CronJob creates Jobs on a repeating schedule using standard 5-field cron syntax.

+-----------------------------------------------------------------------------------------+
|                               CRON SCHEDULE EXPRESSION                                  |
|                                                                                         |
|                 ┌───────────── minute (0 - 59)                                          |
|                 │ ┌───────────── hour (0 - 23)                                          |
|                 │ │ ┌───────────── day of month (1 - 31)                                |
|                 │ │ │ ┌───────────── month (1 - 12)                                     |
|                 │ │ │ │ ┌───────────── day of week (0 - 6) (Sunday to Saturday)         |
|                 │ │ │ │ │                                                               |
|                 * * * * *                                                               |
|                                                                                         |
|   Example: "0 2 * * 1-5" -> Run at 02:00 AM every weekday (Monday through Friday)       |
+-----------------------------------------------------------------------------------------+

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 1 * * *"
  timeZone: "America/New_York"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 200
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  suspend: false
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: reporter
              image: python:3.11-slim
              command: ["python", "generate_reports.py"]

---

3. Concurrency Policies in CronJobs

The concurrencyPolicy dictates how the CronJob controller handles a new schedule trigger if the previous Job execution has not yet completed.

+-----------------------------------------------------------------------------------------+
|                           CRONJOB CONCURRENCY POLICIES                                  |
|                                                                                         |
|   1. Allow (Default):                                                                   |
|      Job 1 (01:00) ----------------------------------> Still running at 02:00           |
|      Job 2 (02:00) ----------------------------------> Runs concurrently with Job 1     |
|                                                                                         |
|   2. Forbid:                                                                            |
|      Job 1 (01:00) ----------------------------------> Still running at 02:00           |
|      Job 2 (02:00) (SKIPPED / BLOCKED to prevent resource contention or race conditions)|
|                                                                                         |
|   3. Replace:                                                                           |
|      Job 1 (01:00) -------------------------> [TERMINATED / CANCELLED at 02:00]         |
|      Job 2 (02:00) -------------------------> Starts fresh execution                    |
+-----------------------------------------------------------------------------------------+

---

4. Operational Troubleshooting and Edge Cases

startingDeadlineSeconds

If a CronJob misses its scheduled execution time (e.g., control plane outage, cluster downtime, or concurrencyPolicy: Forbid), startingDeadlineSeconds defines the maximum allowable delay (in seconds) to start the missed Job. If the delay exceeds this limit, the execution is recorded as missed.

[!CAUTION] If more than 100 execution windows are missed since the last scheduled time and no startingDeadlineSeconds is configured, the CronJob controller stops scheduling future Jobs entirely and logs an error in its controller event stream.

Imperative Management Commands

# Trigger an ad-hoc Job execution immediately from a CronJob template
kubectl create job --from=cronjob/nightly-report manual-run-001

# Pause a CronJob without deleting it
kubectl patch cronjob nightly-report -p '{"spec":{"suspend":true}}'

# Resume a suspended CronJob
kubectl patch cronjob nightly-report -p '{"spec":{"suspend":false}}'

# Check Job logs
kubectl logs job/db-backup-job