14.2 Data Deduplication Architecture, Usage Types & Management
Key Takeaways
- Data Deduplication is a post-processing engine: files are written normally and only later sliced into variable 32 KB to 128 KB chunks, compressed, and stored once in the chunk store.
- Optimized files become reparse points pointing into the System Volume Information chunk store, so the volume reports far more logical data than physical capacity.
- The Default usage type targets general file shares with a 3-day minimum file age; Hyper-V tunes for VDI; Backup optimizes large sequential virtual backup files immediately.
- Deleting data does not free space until the Garbage Collection job runs and unreferenced chunks are reclaimed, which is why free space appears unchanged right after a large delete.
- Deduplication is supported on NTFS data volumes only, not on the boot or system volume and not on ReFS.
Last updated: August 2026
Data Deduplication Architecture, Usage Types & Management
1. Data Deduplication Architecture & Chunk Store
Data Deduplication in Windows Server is a post-processing, variable-chunk storage optimization engine. It identifies redundant data blocks across files on a volume, stores a single unique instance of each chunk in a compressed repository, and replaces redundant file content with pointers.
+-----------------------------------------------------------------------------------+
| DATA DEDUPLICATION CHUNKING PIPELINE |
| |
| ORIGINAL FILES ON VOLUME (e.g., D:\Data) |
| +--------------------------+ +--------------------------+ |
| | File A: [Ch1][Ch2][Ch3] | | File B: [Ch2][Ch3][Ch4] | |
| +--------------------------+ +--------------------------+ |
| | | |
| v v |
| [DEDUPLICATION ENGINE] --> Variable Chunking (32 KB - 128 KB, Avg 80 KB) |
| --> SHA-256 Hashing & Redundancy Analysis |
| | |
| v |
| CHUNK STORE (Hidden in System Volume Information\Dedup\ChunkStore) |
| +--------------------------------------------------------+ |
| | [Unique Chunk 1] [Unique Chunk 2] [Unique Chunk 3] [4] | (Compressed) |
| +--------------------------------------------------------+ |
| ^ |
| | |
| REPARSE POINTS (Stub Files on D:\Data) | |
| +--------------------------+ +--------------------------+ |
| | File A: [Ptr1][Ptr2][Ptr3| | File B: [Ptr2][Ptr3][Ptr4| |
| +--------------------------+ +--------------------------+ |
+-----------------------------------------------------------------------------------+
How Data Deduplication Operates
- Post-Processing Engine: Unlike real-time deduplication that consumes CPU cycles during active write streams, Windows Server Deduplication runs as a scheduled background job, preventing write latency during peak production hours.
- Variable-Sized Chunking: Uses Rabin fingerprinting algorithms to divide file streams into variable-sized chunks between 32 KB and 128 KB (averaging approximately 80 KB). Variable chunking ensures that inserting data in the middle of a file only changes adjacent chunks rather than invalidating the entire downstream block structure.
- Hashing & Compression: Computes a cryptographically secure SHA-256 hash for each chunk. Unique chunks are compressed using the XPRESS algorithm and committed to the Chunk Store.
- Reparse Points (Stub Files): The original file in the directory structure is stripped of its data payloads and converted into a Reparse Point. When an application requests the file, the deduplication file system filter driver (
dedup.sys) transparently intercepts the I/O, fetches the appropriate chunks from the Chunk Store, decompresses them in memory, and returns the stream to the application without user awareness.
2. Usage Types & Job Types
When enabling Data Deduplication on a volume, an administrator must select the appropriate Usage Type to match the underlying workload pattern.
Usage Types Comparison
| Usage Type | Target Workload | Default Minimum File Age | Optimization Mechanics |
|---|---|---|---|
| Default | General Purpose File Servers, user home folders, project shares | 3 Days (Files modified < 3 days are skipped) | Standard background optimization; skips open/locked files |
| Hyper-V | Virtual Desktop Infrastructure (VDI) running open .vhdx files | 3 Days (same age gate as Default) | Optimizes in-use and partial files, so open VHD/VHDX files are still processed; leverages VDI image commonality |
| Backup | Virtualized backup applications (Microsoft DPM, Veeam, Commvault) | 0 Days — the only usage type that optimizes a file the moment it is written | Priority optimization; optimizes in-use files but not partial files. Tuned for large sequential backup streams |
[!IMPORTANT] Hyper-V Usage Type Constraint: In Windows Server, the
HyperVusage type is supported only for VDI (Virtual Desktop Infrastructure) deployments or virtualized backup servers. Deduplication is NOT supported for general Hyper-V compute hosts running production line-of-business server virtual machines (such as SQL Server or Active Directory Domain Controllers) due to random I/O overhead.
Deduplication Job Types
Data Deduplication executes four distinct maintenance jobs:
+-----------------------------------------------------------------------------------+
| DATA DEDUPLICATION JOB TYPES |
| |
| JOB TYPE PURPOSE & BEHAVIOR |
| ------------------+-------------------------------------------------------------|
| Optimization | Scans candidate files, chunks, hashes, compresses unique |
| (Optimization) | data into the Chunk Store, and installs reparse stubs. |
| ------------------+-------------------------------------------------------------|
| Garbage Collection| Identifies orphaned chunks in the Chunk Store that are no |
| (GarbageCollection| longer referenced by any reparse points (after file deletes)|
| | and physically frees disk space. |
| ------------------+-------------------------------------------------------------|
| Integrity Scrubbing| Validates CRC/hash integrity of all chunks in the Chunk |
| (Scrubbing) | Store. If corruption is found, repairs chunks using mirrors |
| ------------------+-------------------------------------------------------------|
| Unoptimization | Undoes deduplication; fully hydrates all files from the |
| (Unoptimization) | Chunk Store back into standard raw files. |
+-----------------------------------------------------------------------------------+
3. Managing Data Deduplication with PowerShell
# 1. Install the Data Deduplication Role Feature
Install-WindowsFeature -Name FS-Data-Deduplication -IncludeManagementTools
# 2. Enable Deduplication on a General Purpose File Share (Volume D:) with 5-day age
Enable-DedupVolume -Volume 'D:' -UsageType Default
Set-DedupVolume -Volume 'D:' -MinimumFileAgeDays 5 -ExcludeExtension 'iso','zip','mp4'
# 3. Enable Deduplication on a Dedicated Backup Target (Volume E:)
Enable-DedupVolume -Volume 'E:' -UsageType Backup
Set-DedupVolume -Volume 'E:' -MinimumFileAgeDays 0
# 4. Manually trigger an on-demand Optimization job with high priority
Start-DedupJob -Volume 'D:' -Type Optimization -Priority High -Memory 50
# 5. Execute Garbage Collection to reclaim disk space after large file deletions
Start-DedupJob -Volume 'D:' -Type GarbageCollection -Priority High
# 6. Execute an Integrity Scrubbing job to verify chunk store integrity
# Default run checks critical metadata only; add -Full to validate all data
Start-DedupJob -Volume 'D:' -Type Scrubbing
# 7. Query Deduplication Status and Savings
Get-DedupStatus -Volume 'D:' | Format-List
Get-DedupVolume -Volume 'D:' | Format-Table Volume, FreeSpace, UsedSpace, SavedSpace, OptimizedFilesCount
Test Your Knowledge
An administrator configures a Windows Server file server hosting a dedicated target volume for nightly enterprise virtual machine backups generated by Veeam Backup & Replication. The administrator wants Data Deduplication to process new backup files immediately without waiting several days for file aging. Which PowerShell command should be executed?
A
B
C
D
Test Your Knowledge
A storage administrator deletes several terabytes of obsolete archived files from a 20 TB NTFS volume where Data Deduplication is enabled. However, File Explorer and Disk Management show that the available free disk space on the volume did not increase after the file deletion. What action must be performed to reclaim the free space?
A
B
C
D