4.1 Git LFS Architecture & .gitattributes Tracking

4.1 Git LFS Architecture & .gitattributes Tracking

Quick Summary: Git's native object database is optimized for lightweight text diffs. Committing large binary files (e.g., machine learning models, media assets, compiled dependencies) causes exponential repository bloat because Git cannot delta-compress binary data effectively. Git Large File Storage (LFS) resolves this by replacing binaries in Git commits with tiny text pointer files and storing the binary payloads in an external remote blob store (such as Azure Repos LFS or GitHub LFS). This section covers the object model, the clean and smudge filter pipeline, and .gitattributes tracking; migration and file locking follow in Section 4.2.

---

The Git Object Model & The Binary Bottleneck

To understand why large binary files degrade Git repositories, one must examine how Git stores data internally within the .git/objects directory. Git is a content-addressable store built on a Directed Acyclic Graph (DAG) consisting of four core object types: blobs, trees, commits, and annotated tags.

Git Object Architecture & Binary Delta Failure:

Commit Object (Author, Committer, Tree SHA)
      │
      ▼
 Tree Object (Permissions, Filename, Blob SHA)
      │
      ├──► Text Blob: code.cs (v1: 10 KB) ──► (v2 edit: 10.2 KB) ──► Packfile Delta: +200 bytes stored
      │
      └──► Binary Blob: model.onnx (v1: 500 MB) ──► (v2 edit: 500 MB) ──► Packfile Delta: 500 MB stored (Full duplicate)

Why Delta Compression Fails for Binaries

When Git writes text files, it compresses each blob using zlib. Over time or during network transfers, Git consolidates loose objects into packfiles (.pack files accompanied by .idx index files). Within packfiles, Git applies delta compression (directed acyclic graph delta encoding):

1. For source code files, Git analyzes two versions of a file, recognizes identical lines, and stores only the modified lines as a compact diff against the base object.
2. For binary assets—such as compressed archives (.zip, .tar.gz), video/audio files (.mp4, .wav), image assets (.psd, .png), compiled binaries (.dll, .so), and machine learning weights (.onnx, .pt, .safetensors)—the underlying bytes are already compressed or encrypted.
3. Modifying even a single metadata tag, weight parameter, or pixel changes the entropy of the entire file. Consequently, Git's delta compression algorithm fails to locate recurring sliding-window sequences.
4. Git is forced to store nearly the entire binary file anew for every single commit where that file is touched.

The Operational Consequences of Binary Bloat

When large binaries are committed directly to standard Git repositories:

• Permanent Historical Drag: Because Git is distributed, running git clone downloads the entire commit history. If a developer commits a 200 MB test dataset, deletes it in the very next commit, and pushes to Azure Repos, every single developer and CI/CD build agent who ever clones that repository thereafter will still download that 200 MB blob forever.
• Cloning Timeouts & Disk Exhaustion: Ephemeral CI/CD build agents (such as Microsoft-hosted Azure Pipelines agents or GitHub-hosted runners) must clone the repository from scratch on every pipeline run. Multi-gigabyte packfiles cause pipeline timeouts, consume expensive bandwidth, and exhaust local disk space.
• Packfile Limits & Memory Pressure: Running git gc or git pack-objects on repositories with massive binaries exhausts available virtual memory, causing out-of-memory (OOM) fatal crashes on developer workstations and build agents.

---

Architecture of Git Large File Storage (LFS)

Git Large File Storage (LFS) is an open-source extension developed by GitHub, Microsoft, and community contributors that overcomes the binary bottleneck by decoupling file metadata from file payload storage.

The Core Design Principle

Instead of storing the actual binary payload inside the Git tree object, Git LFS:

1. Intercepts the binary file during staging.
2. Computes its cryptographic SHA-256 hash.
3. Transfers the actual binary payload to a dedicated remote HTTP/HTTPS blob store (integrated natively into Azure Repos, GitHub, or an external cloud storage provider).
4. Commits a lightweight pointer file (typically under 150 bytes) into the Git commit tree in place of the binary.

Git LFS Working Directory vs. Repository Storage:

Local Working Tree                    Git Index & Repository History             Remote LFS Blob Storage
┌────────────────────────┐            ┌────────────────────────────┐             ┌────────────────────────┐
│ weights.onnx (450 MB)  │ ──clean──► │ Pointer File (130 bytes):  │ ──push────► │ Raw 450 MB Payload     │
│ (Actual binary file)   │            │ - version: v1              │             │ (Stored in Azure Repos │
│                        │ ◄─smudge── │ - oid: sha256:7f3a8b...    │ ◄─fetch─── │  LFS Blob Storage)     │
└────────────────────────┘            │ - size: 471859200 bytes    │             └────────────────────────┘
                                      └────────────────────────────┘

The LFS Pointer File Anatomy

An LFS pointer file is a plain text file formatted with key-value pairs matching the official Git LFS specification:

version https://git-lfs.github.com/spec/v1
oid sha256:7f3a8b92c4e1d5a76041283e5891d4829375b140837d36489a2956f89104b2a1
size 471859200

• version: Defines the LFS specification schema URL.
• oid: The Object Identifier, strictly formatted as sha256: followed by the 64-character hexadecimal SHA-256 hash of the complete binary content.
• size: The exact physical file size in bytes.

Because only this 130-byte text pointer is stored in the Git commit tree, branching, merging, diffing, and cloning operations remain exceptionally fast. Committing 50 iterations of a 500 MB model consumes only 50 text pointers in the Git packfile (less than 10 KB total Git history), while the 50 distinct binary payloads reside safely in scalable cloud blob storage.

The Git Filter Pipeline: Clean and Smudge Filters

Git LFS achieves complete transparency for developers through Git's configurable filter driver mechanism. When you run git lfs install, Git registers global filter configurations in your local ~/.gitconfig:

[filter "lfs"]
    clean = git-lfs clean -- %f
    smudge = git-lfs smudge -- %f
    process = git-lfs filter-process
    required = true

Here is the exact lifecycle of a file tracked by Git LFS:

1. The Clean Filter (git add / Staging):

   • The developer saves a 200 MB design file: design/homepage.psd.
   • The developer runs git add design/homepage.psd.
   • Git checks .gitattributes and routes the file through the clean filter (git-lfs clean).
   • The clean filter hashes the binary stream, writes the full 200 MB binary into the local object cache at .git/lfs/objects/, and generates the 130-byte text pointer.
   • Git writes the text pointer to the Git staging index (.git/index). The heavy binary is never written into a standard Git blob.

2. The Pre-Push Hook (git push):

   • When the developer runs git push origin main, Git invokes the .git/hooks/pre-push hook installed by Git LFS.
   • Git LFS scans the commits being pushed, identifies any LFS pointers, and initiates an HTTPS API call to the remote Git LFS endpoint (e.g., https://dev.azure.com/{org}/{project}/_git/{repo}/info/lfs/objects/batch).
   • Git LFS uploads the unpushed binary payloads to the remote blob store.
   • Once the LFS uploads succeed, Git proceeds to push the Git commit and tree objects (containing the pointers) to the remote repository.

3. The Smudge Filter (git checkout / Cloning):

   • Another developer runs git pull or git checkout main.
   • Git unpacks the tree object and passes the pointer file through the smudge filter (git-lfs smudge).
   • The smudge filter reads the SHA-256 OID from the pointer file.
   • It checks the local .git/lfs/objects/ cache. If the binary is present locally, it copies it into the working tree.
   • If the binary is missing locally, Git LFS downloads the specific binary payload from the remote LFS store and replaces the pointer file with the hydrated, full-size binary in the working directory.

[!NOTE] LFS Batch API and Optimization: Modern Git LFS uses the filter-process long-running protocol rather than invoking separate smudge and clean sub-processes for every single file. This allows Git LFS to batch HTTP requests, negotiate parallel chunked downloads, and dramatically accelerate checkouts.

---

Configuring and Tracking Files via .gitattributes

Git LFS determines which files to intercept based on the root .gitattributes file. If a file pattern is not tracked in .gitattributes, Git will treat it as a standard Git blob.

Step 1: Initialize Git LFS in the Environment

Before tracking files, initialize Git LFS hooks on your machine:

# Configure system/user Git configuration with LFS filters
git lfs install

Step 2: Track Specific Extensions or Paths

Use git lfs track to add patterns to .gitattributes:

# Track Photoshop files anywhere in the repository
git lfs track "*.psd"

# Track machine learning models under the models directory
git lfs track "models/*.onnx"

# Track compressed training datasets
git lfs track "data/**/*.tar.gz"

This generates or updates the .gitattributes file in the working directory:

*.psd filter=lfs diff=lfs merge=lfs -text
models/*.onnx filter=lfs diff=lfs merge=lfs -text
data/**/*.tar.gz filter=lfs diff=lfs merge=lfs -text

Step 3: The Golden Rule of .gitattributes

[!IMPORTANT] AZ-400 Exam Rule: You MUST commit .gitattributes to the repository immediately after configuring it. If a developer runs git lfs track "*.onnx" but fails to commit .gitattributes, other team members will pull the repo without those filter definitions. When those team members commit .onnx files, Git will commit raw 500 MB blobs directly into standard Git history, completely defeating your LFS strategy!

git add .gitattributes
git commit -m "chore: configure Git LFS tracking for PSD, ONNX, and archive assets"
git push origin main